A Case Study: Supreme Commander

As we stated earlier, the inspiration behind this article is Supreme Commander, which was crashing during gameplay due to what we discovered to be was the game exhausting its user space. Without taking extreme and convoluted measures, most applications can do little besides (gracefully) crash when they've exhausted their user space. Server applications, which on the whole tend to be the king of memory hogs in the first place, tend to have some sort of compensation mechanism, but for more generalized applications like games this is not the case.

In this case, Supreme Commander was crashing late in to big games on a system that otherwise is completely stable. No amount of testing could come up with a problem outside of Supreme Commander, and most telltale of all was that it was only a problem with large maps with large number of players with high quality settings; turning anything down made the crashing stop. Although not clearly a memory error it greatly reduces the list of possible problems to a few things. The entire process, we suspect, will be similar for everyone hitting the 2GB barrier: the affected application is crashing under heavy load. Thankfully the problem is possible to debug, albeit with moderate difficulty.

With all of that said, I did not come up with the diagnosis or the solution myself and while the problem in retrospect is quite obvious, at the time I did not even take the 2GB barrier in to consideration. As such, credit for the diagnosis and solution belongs to MadBoris of the Gas Powered Games forums who identified Supreme Commander as hitting the 2GB barrier and publishing a solution for it, which we will cover here.

Getting to the root of the problem requires additional tools beyond what comes with Windows XP or Vista. The task manger, the logical tool for this task, doesn't track the information we need to know. At best, the task manager can list how much data is in the various memory pools altogether, however this is not the metric we're looking for. What we actually need to know is how much of the application's share of the virtual address space has been allocated (the fundamental problem after all is when we run out of unused virtual address space to allocate) which due to a multitude of reasons can be much greater than the amount of memory in use by an application. For finding this we'll need Sysinternals' Process Explorer.

The above screenshot was taken as this paragraph was written, showcasing several different memory values for the running processes. The WS Private (Working Set Private) column lists how much physical memory the application is using, and this is the same column listed by the Windows Task Manager for memory usage. The Private Bytes column is the total amount of memory the application is using. Last but not least however is the Virtual Size column, which finally lists the amount of virtual address space allocated, the metric we're after.

We'll quickly focus on Microsoft Word as an example of the discrepancy between physical memory usage and private size. While Word is only using 17MB of physical memory, its virtual size is 170MB, 10 times the physical memory usage and 9 times the total memory usage. This is more or less a worse case scenario but it also points out just how misleading any memory usage - physical or total - is when trying to diagnose this kind of problem. Applications or games with high memory usage tend to not have nearly as a severe spread, but as we can see it's possible for an application to hit the 2GB barrier well before it needs 2GB of physical memory.

Getting to the meat of things however, the process readout for Supreme Commander basically says everything that needs to be said. In order to prove an exaggerated point, we started a 6 player game with 5 AI units on a maximum size map (81km x 81km) and set the game speed to maximum with maximum quality settings, while patching Supreme Commander and adjusting our system configuration to break the 2GB barrier. This actually isn't very playable for a variety of reasons (we'll overtax the CPU quickly due to all the AI units, slowing the game down dramatically) but represents not only what can happen in bad situations, but also is something that happens even in more manageable situations on smaller maps later in to the game.

Just at the start of the game, our virtual size was already in excess of 1.6GB, dangerously close to the 2GB barrier. 16 minutes in to the game, we have already hit a virtual size of 2.2GB, meaning had we not broken the 2GB barrier the game would have already crashed. At this point our total memory usage (private bytes) is only at 1.7GB, with 1.4GB of that in physical memory, nearly maximizing the RAM usage of our 2GB system. Our spread between memory usage and virtual size is .5GB, showcasing how deceptive memory usage is in identifying 2GB barrier issues.

Finally at 22 minutes in to the game, the game crashes as the virtual size has reached the 2.6GB barrier we have reconfigured this system for. Perhaps the most troubling thing at this point is that Supreme Commander is not aware that it ran out of user space in its virtual address pool, as we are kicked out of the game with a generic error message. Unfortunately Windows Vista reverted to a non-accelerated desktop at this point, preventing us from capturing a screenshot of the exact memory readouts or the error message.

Removing the 2GB Barrier A Case Study, Cont
POST A COMMENT

69 Comments

View All Comments

  • miahallen - Saturday, July 14, 2007 - link

    What I found extremely interesting is that you had problems modifying your boot.ini file for a lsrger than 2.6GB app space. I have an almost identical configuration, and have modded almost all my games headers, and my boot.ini is set to 3GB app space. When I modded the boot.ini, I was unaware of the possible problems, but since it didn't cause any, I've been perfectly happy with it.

    A8N-SLI Delux
    Opty 165 @ 2.5GHz
    2GB DDR400
    8800GTX (768MB VRAM)
    Vista x86

    I've modded the headers for:
    STALKER
    C&C3
    CoH
    Dark Messiah M&M
    DiRT
    FSX
    Silent Hunter 4
    TDU

    Thanks Ryan, for the great article!
    Reply
  • MadBoris - Sunday, July 15, 2007 - link

    quote:

    I've modded the headers for:
    STALKER
    C&C3
    CoH
    Dark Messiah M&M
    DiRT
    FSX
    Silent Hunter 4
    TDU


    This should be made clear to people.
    If an application is not Large Address Aware there is usually very good reason the developers did not include it that way. There maybe things you break in the game or cause stability problems by adding it.

    So people should not be just adding this to all their games, especially considering many games don't come close to the 2GB ceiling in the first place, so their is no benefit only potential negatives.
    Reply
  • ChristopherO - Monday, July 16, 2007 - link

    This is true... One shouldn't go setting the binary flag because they *can*.

    I will however confirm that C&C3 and FSX run into the 2GB barrier. FSX is fine once it is patched.

    However, C&C3 is so poorly written that it can easily pass 2GB and run into the revised 3GB barrier on any multiplayer map with the maximum number of AI and/or opponents. Detail settings as low as the "medium" preset do nothing to alleviate this problem.

    Unfortunately C&C apparently has absolutely no code to purge and re-use memory as I've been in huge games, on the downward slope in terms of remaning opponents/units, and the app still hits a revised 3GB memory space and crashes.

    As a developer myself, memory management is *not* that complicated. It takes some forethought, but the EA people apparently never bothered. This is a huge letdown since I've been a C&C fan since the first game came out my first year of college.
    Reply
  • miahallen - Saturday, July 14, 2007 - link

    I have to add one more thing about my system, may be important.....most of my games I play at 2048x1536, as opposed to you test rig at 1600x1200. That and you were using Vista Ultimate, I'm using Home Premium. Do you think this is what is effecting the results? If so, Home Premium (or even Home Basic) is the x86 OS of choice for gamers, not Ultimate! Reply
  • miahallen - Sunday, July 15, 2007 - link

    I was hoping you would comment on this Ryan, do you think the difference in our boot.ini mod was due to the version of Vista we run? Any thoughts? Reply
  • sheh - Thursday, July 12, 2007 - link

    ...and disturbingly it doesn't set off any sort of multiplayer cheat detection in the game in spite of the fact that we have modified the executable in a very visible way.

    The game probably checks the code (and maybe data) section(s) in memory, and not the actual EXE file (makes sense considering you can use memory patchers). The header might not be important for cheat prevention.
    Reply
  • MadBoris - Thursday, July 12, 2007 - link

    quote:

    ...and disturbingly it doesn't set off any sort of multiplayer cheat detection in the game in spite of the fact that we have modified the executable in a very visible way.

    Yeah, I remember thinking that the first day I did it, actually in those days it was Securom protected which I was actually more suprised about bypassing. But seriously the exe header should not be something that cheat protection should look for. I can only say glad it didn't or crashing would be a constant problem. ;)
    Reply
  • atlr - Thursday, July 12, 2007 - link

    I thoroughly enjoyed this article. Good job. Reply
  • atlr - Thursday, July 12, 2007 - link

    A lot of good comments have helped edit and tune this article too. (meh, not this one though) Yayyy, community. Reply
  • JetBlack69 - Thursday, July 12, 2007 - link

    From a programming perspective, what is a programmer to do? I assume this can happen in a program when "new" returns NULL because the program is out of memory, but what can a program do gracefully?

    I assume it could just display an error message, but if it were during a game, how could it handle it gracefully and not crash or give an error message? Would it lower the texture detail or remove unneeded objects on the screen?
    Reply

Log in

Don't have an account? Sign up now