I think Ben already proves you're wrong in Ep. 4, seriously, abandoning Clem was the least of his wrongdoings in it. Like I said before, the guy just downright f-ing scares me, even when he tries to help somebody dies. And no, I don't think alot of people are gullible enough to believe a band of murderers when they say they're keeping his friend like Ben did in Ep. 3 or removing the hatchet from the door that was keeping the walkers out of the part of the building Lee's group was in which resulted in the walkers killing Brie in Ep. 4. Nobody is expecting him or anybody else to adapt perfectly to a ZA in the space of 3 months, but Ben's a proven walking disaster.
Maybe I'm just too old to view Ben from a more sympathetic point of view, but now with Ep. 4 I'm just wondering what Ben has to do to finally alienate his defenders.