I think it all depends on how you implement it.

For instance...anyone here ever played Titanic: Adventure out of Time (and yes, you can laugh at me for actually having played that game--but in my defense, I hated it)? There's a segment not long before the ship hits the iceberg where you have to fight your way past a bad guy in your basic fistfight. Let me tell you, I hated that part. It was completely counterintuitive, there was no instructions or anything and no warm-up/foreshadowing that this would happen, and the bad guy moved like somebody'd lit his feet on fire, while my character didn't, no matter how fast I clicked the mouse.

That action sequence was a big part of why I ended up literally throwing the CD out the window. Drove me nuts.

Now, then we have the Old Mine Road sequence in Full Throttle. That, I loved. The manual actually gave clear directions as to how to switch weapons/throw punches/etc. If you lost, there weren't negative consequences (in Titanic, IIRC, you lost a bunch of valuable time by being unconscious). And it was just easy for me to coordinate things--timing and how you manuever your bike are a big factor in winning those sequences, and that, I could handle. Randomly throwing punches and hoping I took out a guy built like a brick blockhouse, as in Titanic? Not so much.