All he got angry at me for was taking Clem to the St. Johns farm (didn't know they were cannibals at the time...) and saving Doug over Carley (I chose Doug because even though it was obvious that only one would survive from a 'this is a game perspective', Doug seemed to be at the greatest risk and Carley only had a zombie grab her foot - happened to Lee dozens of times).
Have to say that was a pretty lousy reason to stalk us. I mean he didn't exactly seem rational but considering I didn't take the supplies from the car he said he wasn't going to go after me until he found those things out. Those two things don't seem to justify the change of mind.
I bought Lilley with us on the RV but I'm almost certain he didn't even bring Lilley up.
I guess the decisions that you make affect how crazy and irrational this guy seems rather than whether he likes you or not.