Data that isn't used often is a good candidate for being paged out, in order to make more physical memory available.
But unfortunately, physical memory isn't the only thing using that address range.
It is more accurate than my physical memory, as you yourself noticed.
In practice, the only advantage of using 64-bit Windows is that you can install more physical memory.
"I have to have the physical memory to hold all of that."
When this happens the page needs to be taken from disk and put back into physical memory.
This way, the amount of physical memory allocated for the process does not increase until data is written.
The older 36-bit computers were limited to a similar amount of physical memory as well.
Generally, making more physical memory available also reduces page faults.
Running programs' code and data may be scattered over several areas of physical memory or even placed on a disk until needed.