I'm not saying that we don't use our fair share of memory, but task manager is not a reliable way to judge memory usage. Here is a pretty good article on what the task manager does (and doesn't) represent.
An interesting test is to simply minimize the game then look in the task manager. Suddenly we're not using so much memory! Instant patch!