As far as I know, one of the biggest 'wins' for 64bit is in addressing, for anything above a 32 bit address space (more than 4gb).
You have to have pae support on a 32bit proc (and enable it in the os) to hit more than 4gb, which does some type of mapping from 64 bit onto a 32 bit address space. Memory addresses have to be translated, and not a straightforward mapping. Memory access becomes a bit slower. Add to this that the mapping can be different between processes. So if you have processes sharing memory (ie for ipc), you get even more overhead as it has to be translated, then translated again, to be referenced by each site of a shared memory block.
more about pae: http://en.wikipedia.org/wiki/Physical_Address_Extension
64 bit has some downsides. I think the hybrid x86_64 chips have weird pipelines to the cpu to support both 32bit and 64bit mode..where it does some weird split into two 32bit words, and uses part of each chunk to tell which mode it is in.. thus reducing you by addressible memory by some factor. Not sure about this one..not really that much of a hardware guy.
Also, the word size is larger, so there is some overhead. Memory pointers take up more space, etc. So random app X might use a bit more ram on 64bit than on 32bit. *shrug*
I am sure some of the above is off, and I am sure the more hardware centric folks in the audience will point out my inaccuracies.
oh. also.. http://en.wikipedia.org/wiki/64-bit