ReDi, you're thinking of the gambling club puzzle on Phatt Island. It's got similarities but works from a very different angle. "If this is three (shows five fingers), what's this (shows two fingers)?" I won't say the answer just to bug the people who never played Monkey Island.
Also, Cyphox, that sounds like a really boring way to do it. The pattern was a lot easier to figure out if you had that helpful sheet from the under-renovation speakeasy.