Adventure games should definitely be point and click. There's really no reason not to use this system, they don't contain action or anything requiring very precise controls.
Okay, for the sake of discussion, let's say the capable team over here came up with some good reasons that a particular game had to be direct control.
What is it you're afraid would be lost? Or put another way, what is it about point & click that makes it the "right" way for an adventure game to be played? (Or have you just been burned by poorly implemented direct control in the past? Which games?)
Also, for those who don't like it - have you ever tried using a gamepad for an adventure game? Did you find that this changed the experience, or did you still not like it?