A Messy Transition: Practical Problems With 32bit Addressing In Windows
by Ryan Smith on July 12, 2007 12:00 PM EST- Posted in
- Software
Except in a few cases where 64-bit code is clearly faster, the primary purpose for Vista x64's existence is to resolve the problems of 32-bit addressing space, and we're just not at the point yet where even most enthusiasts are pushing that limit. Once applications begin to push the 2GB addressing space limitation of Win32 (something we expect to hit very soon with games) or total systems need more than 4GB of RAM, then Vista x64 in its current incarnation would be a good choice.
For some time now we have been mentioning the potential problems that are likely to result from the switchover from 32bit(x86) Windows to 64bit(x64) Windows. Due to a multitude of issues, including Windows' memory management, the basic design of the PC architecture, and consumer support issues, there is no easy path for mass migration from 32bit Windows to 64bit Windows. As a result we have been expecting problems as consumers begin to make the messy transition.
We published the above mentioned guide on February 1st, expecting the fall/winter 2007 games to be the ones to push the 2GB addressing space limitation of Windows, and it turns out we were wrong. It turns out that two weeks after we published the above article, THQ published Supreme Commander, a RTS with a massive appetite for resources. It can be simultaneously GPU limited and CPU limited, which is why it's a standard benchmark here for our performance articles, it's also memory limited in more than one way: it's hitting the 2GB barrier of 32bit Windows.
An artifact of the design of 32bit processors and the 32bit API for Windows, the 2GB barrier is a cap on how much addressing space (related to but not equivalent to memory usage) a single application can use. This isn't a bug but rather the result of how hardware and software was created so many years ago, and while everyone has known this barrier will inevitably be hit, as we'll see there are several reasons why it can't simply be moved or bypassed. Meanwhile hitting it involves affected applications crashing for what can appear to be no good reason, and understanding why the 2GB barrier exists and what can be done will be important for resolving those crashes.
On a personal note, I am a semi-casual player of real time strategy(RTS) games and I've been playing Supreme Commander lately. This is a different kind of article, it's a record and the result of my own efforts to resolve why I was having crashing issues with Supreme Commander. With no intended disrespect towards THQ or the game's developers (Gas Powered Games) we could have not possibly asked for a better example of the 2GB barrier in action. It's exactly the experience we believe many people will have as they hit the 2GB barrier, mainly those power users who use large monolithic applications such as games or multimedia tools. This is an article on what the problem with the 2GB barrier is, what kind of experiences a user may expect when hitting it, and what can be done to fix it.
But before we get too far ahead of ourselves, let's discuss memory management in Windows. Understanding the problem with Supreme Commander requires understanding what the 2GB barrier is, why it's there, and what makes it so problematic.
Facebook
Twitter
RSS
69 Comments
View All Comments
titan7 - Thursday, July 12, 2007 - link
There is nothing you can do. If you ship a game you could test it and ensure it doesn't run out of memory (e.g. GameCube has just 24megs of system memory and the last Zelda looked great. Quite a ways away from 2048megabytes PC games run into!), but what about a mod?When the application starts just allocate 512 megabytes or whatever you feel is reasonable. When new throws an exception free that memory, display a warning you're low on memory and need to upgrade to Vista64, and continue. When it new fails again one microsecond later you're screwed so display a message to the end user along the lines of "I told you so!" End users really like that type of thing. ;)
You could get a bit fancier by replacing all units with simple cubes or something, but all that does is delay in the inevitable a bit longer.
ncage - Thursday, July 12, 2007 - link
1) First step is to detect which OS you are using at startup. Is it 64 bit os or not?2) You SHOULD never code your application around a 2GB memory limit. It is very bad coding practice. Going through thorough examples in a short post like this isn't very practicle
3) Some higher level langauge/constructs abstract this away from you. For example if you are using .Net CLR you don't really have to worry about this unless maybe your doing some crazy pinvoke stuff which in most cases you shouldn't be doing anyways. Of course if your doing VB6 your never going to get around it anyways because vb6 is only 32 bit.
4) If you are doing C++ or assembly with windows then you can use the GlobalMemoryStatus() Function to Effectively see how much available address space you have.
Ryan Smith - Thursday, July 12, 2007 - link
The key to any of this is monitoring how much of the virtual address pool is in use; there should be an API call to ask Windows this. The easiest thing to do would be to give a warning at 1.9GB or so and then either do nothing, trigger a crash early, or attempt to reduce detail or flush space to stay below the 2GB barrier. The warning is the easy part, the hard part is preventing the crash, and I don't honestly believe anyone can or will be preventing crashing.yacoub - Thursday, July 12, 2007 - link
I just wish we had a better solution than Vista. Sure we can use 64bit XP but that's only going to last how much longer with full patch support from MS?If Vista wasn't such a pile and didn't perform worse in games when using equivalent hardware as the same system running XP, it wouldn't be such an unappealing alternative.
And even so, when running 4GB of RAM, how much over a LargeAddressAware flagged game with the 3GB boot.ini switch are you really gaining by using 64bit OS? Not much, really. We first need motherboards that are happy running 8GB of RAM, RAM cheap enough to buy 8GB for a reasonable price (which is not too hard with DDR2 2GB DIMMs right now), and do so at full performance/speed settings.
Really it's not just a move to a 64bit OS, it's also a move to 8GB of RAM.
OR it's simply having developers who code their games to work properly within 3GB of addressable RAM.
instant - Saturday, July 21, 2007 - link
How many more patches do you need for XP 64bit anyway?As long as the games work for it, why care about microsoft updating "security" hotfixes or not.
Tegeril - Thursday, July 12, 2007 - link
Articles like this make me very glad that I opted to go with 64bit Vista. All my hardware is supported at this point with stable drivers (we can argue the Creative X-Fi, but it works fine). I'm just amazed that people saw this coming and yet we have games that just die because of the problem. 64 bit isn't as bad as people make it out to be regarding compatibility :D - except iTunes and Quicktime :(halfeatenfish - Thursday, July 12, 2007 - link
How do the *nix variants deal with this same issue? Do they even have it? Can someone shed some light, especially in terms of OS X... Leopard, if anyone knows anything there. But Tiger info is just as good.titan7 - Thursday, July 12, 2007 - link
They all do the same thing for 32bit. Bu generally speaking *nix has been 64bit for years (decades) so if it is a problem just run the 64bit version of everything. And being more cross platform their code tends to have less hacks like you get in Windows apps that assume there is an extra bit available on every pointer. A single bit! Bah, we're talking about billions of bytes and elite programmers are trying to squeeze every last bit out of their application at the expense of future compatibility. LAME.The Boston Dangler - Thursday, July 12, 2007 - link
OSX is a 64-bit system, *nix ymmvMadBoris - Thursday, July 12, 2007 - link
Cool article Ryan. Good to see these issues getting more global attention.Since 32 bit seems it is here to stay for a lot longer than we want it to, and with software bloat continuing, this will hopefully continue to put pressure on driver devs to write better drivers that can handle >2GB addresses without issue. So that people can use the /3Gb switch without concern. I personally have never had problems with /3GB with any of my hardware/drivers but certainly 'less mainstream' drivers may not be handled with the care that they should be.
I like the breakdown of games/apps that support the LargeAddressAware flag, maybe this list can grow for future articles covering more apps/games. I also enjoyed your testing on the "potential" penalty of less kernel space, something I never took the time to do on my own.
Imagine my suprise today when making my rounds to my favorite hardware site. ;)