A Primer on Windows' Memory Management

In 1986, Intel released the 386 processor, which offered support for a new instruction set (IA32), that was an extension of the original x86 instruction set. Among the most notable features in IA32 was an increase in the amount of memory the CPU could address, moving from 20bit addressing(1MB) to 32bit addressing(4GB)(ed: this is not including the convoluted mess that was segmented addressing). All x86 CPUs released since then have supported the same instruction set, including the same 4GB limit. Only recently have 64bit x86 CPUs been released, and while they support more than 4GB of memory, still are limited to 4GB when operating in 32bit mode.

As we have mentioned in previous articles, most modern system running in 32bit x86 mode have trouble seeing and using more than roughly 3GB of memory. This is because part of the total 4GB of memory space (not the physical memory) is reserved for various functions, such as computer components transferring data between each other using memory-mapped input-output(MMIO). The textbook example of this is the CPU transferring data to the memory of a video card, where a chunk of the address space equal to the size of the memory of the video card is reserved by the video card, and any data sent to those addresses actually ends up going to the video card. This design has many technical merits, but it makes the consumed memory addresses unavailable for use with physical memory.

Things only get more complex as we start including the operating system (in this case Windows) in to the equation. The above is actually handled by a combination of Windows and the BIOS, meanwhile Windows also needs some address space so that programs can communicate with the Windows kernel, for storing buffers, for storing memory tables, etc; all of which means we have lost even more address space. All of the above besides preventing us from addressing 4GB of physical memory are also the cause of the actual 2GB barrier that is the problem.

Quickly, there is one more pre-requisite piece of information: virtual address space. For a 32bit Windows application(Win32), each application has a full 4GB worth of private addressing space that it can use, which again is 32bits and a result of how a 32bit processor works. How the above exactly works is beyond the scope of this article, but it's sufficient to say that at some point virtual addresses get translated in to other addresses for mapping data between the application and physical memory or the swap file. The important thing to take from this is that each application has its own 4GB virtual address space, regardless of the hardware the computer contains or what applications are running. Now we may begin to understand the 2GB barrier.

Due to design reasons outside the scope of this article, Windows takes a pre-determined portion of each application's virtual address space and reserves it for itself. Windows uses its portion of the virtual address range for all of the address needs listed above. At the end of the day, and what really matters, is that in designing Windows Microsoft opted to split up the virtual address space of an application in half; 2GB goes to Windows (kernel space) and 2GB goes to the application (user space). Under normal circumstances this 2GB of space is all a 32bit application has to work with, this is the 2GB barrier and as we'll see is the cause of the problems with Supreme Commander.

It is also worth noting at this point that virtual address space is not directly correlated to physical memory size. Windows and any applications running under it may use up to all of their allocated virtual address space, with Windows simply swapping data between the hard drive and physical memory if the amount of physical memory needed is in excess of what's available - this is virtual memory, not to be confused with virtual address space. The only meaningful relation between virtual address space and physical memory is that the amount of physical memory an application can use can never be greater than its virtual address space. The user space must take up a larger portion of the virtual address space if an application is to use more than 2GB of physical memory.

Index Removing the 2GB Barrier
POST A COMMENT

69 Comments

View All Comments

  • nullpointerus - Thursday, July 12, 2007 - link

    64bit versions of Windows(i.e. XP and Vista) do not suffer from the traditional 2GB barrier, as all the kernel mode addressing is usually moved to well above the confines of the limited 32bit addressing area. As such, these versions of Windows don't need to have their space allocations adjusted for an application to gain access to more addressing space, bypassing the instability and any possible performance problems that occurs as a result of making this adjustment.

    I used 64-bit Vista for a while but just couldn't deal with the BSOD's and other wierd behavior that would happen with 4 GB RAM installed and the BIOS memory hole option set. For example, the nVidia SATA drivers (3 updates on Windows Update alone) for my board would BSOD on every boot when the BIOS memory hole option was enabled, and the PVR-150's driver would randomly corrupt itself and turn into a green/garbled slideshow:

    http://www.hauppauge.co.uk/board/showthread.php?t=...">http://www.hauppauge.co.uk/board/showthread.php?t=...

    I wish the 64-bit driver developers would get off their butts and fix this stuff.
    Reply
  • gersson - Thursday, July 12, 2007 - link

    Large Address Aware Status

    it says, "STALKER: Oblivion Lost"
    Reply
  • Ryan Smith - Thursday, July 12, 2007 - link

    I'm not sure I see what the problem is. Could you go in to more detail? Reply
  • gersson - Thursday, July 12, 2007 - link

    You meant to place 'Stalker' cos there is already an entry for oblivion and 'lost' (I assume you mean 'lost planet')
    Read like this:

    Observe first line:

    STALKER: Oblivion Lost Yes
    Lost Planet No
    Company of Heroes Yes
    Supreme Commander No
    Enemy Territory: Quake Wars Yes
    Battlefield 2142 No
    Call of Juarez DX10 BM No
    World of Warcraft No
    The Elder Scrolls IV: Oblivion No
    Photoshop CS3 Yes
    Dreamweaver CS3 No
    Reply
  • DigitalFreak - Thursday, July 12, 2007 - link

    No. Stalker had the working title of Stalker: Oblivion Lost... Reply
  • gersson - Thursday, July 12, 2007 - link

    hmm ok, didn't know that -- but you can see how I had arrived @ my conclusion :-P Reply
  • The Boston Dangler - Thursday, July 12, 2007 - link

    the game was released as STALKER: Shadow of Chernobyl. no biggie Reply
  • Ryan Smith - Thursday, July 12, 2007 - link

    Fixed. I keep forgetting they changed the name. Reply
  • grant2 - Thursday, July 12, 2007 - link

    So this game will crash reproducably when using a large map, with many players, and many units.

    And GPG released it like this? Either their QA department is rather incompetent, or management were buffoons to ignore such an obvious error.

    Very disapointing!
    Reply
  • EODetroit - Thursday, July 12, 2007 - link

    I've been hoping Anandtech would take memory issues more seriously for a while now, glad you're getting your teeth into it for real now!

    "Unfortunately Windows Vista reverted to a non-accelerated desktop at this point, preventing us from capturing a screenshot of the exact memory readouts or the error message."

    Use a camera!
    Reply

Log in

Don't have an account? Sign up now