Originally I let go of Ben and told Kenny "I need him..." when trying to gather a party to go look for Clem. He gave me the whole spiel about how I haven't been there for him at times and stayed with the boat even when I've sided with him over 98% of the time throughout the entire game. I ended up going with Omid and Christa (or whatever her name is).
I decided to play the last two chapters of episode 4 over and saved Ben instead, which should have (and did) piss Kenny off even more. However, during the confrontation about the bite/Clem, I decided to say "Clementine is my family..." and after Kenny gave his whole spiel again he sympathized with me and decided to come instead. He did, however, have a gripe when I asked Ben to come but I told him to deal with it and reminded Ben of how Clem stood up for him, so the kid decided to come along as well. In the end, I got the whole team to come with me when I went back.
I believe that it really comes down to your final choice of words to Kenny during that last conversation. Unless, I suspect, you've been a dick to Kenny throughout the episodes, but who knows.
I do, however, wonder what might happen to the boat now that I left no one to guard it.