The below may comprise largely of opinion:

Yes, the puzzles are what makes up most of the gameplay, but they're not the things that make it an adventure game.

I play a shooting game because I feel an urge to blast stuff, and I play a RTS game because I feel an urge to use my brain to blast stuff. I play an adventure game because I feel the urge to lose myself in a story for a while. If it's puzzles that provide the mechanism for that, fine, but if not then at long as it provides me with a story a character focused experience that I've come to expect from adventures, I don't see what I'm missing.

The fact is, as gameplay, adventure puzzles historically aren't even that fun. Maybe two thirds or more of puzzles in adventure games are either easy enough to get right away or so hard that most people use a walkthrough to get past it. The remaining third are subtle ones that you might wonder about for a little while and then actually work out. If the puzzles were what really mattered I would have given up on adventures long, long ago. The FUN part about solving puzzles is being able to progress that little bit further in the story.