Benchmarking Software: an Analysis of Far Cry 2 Settings under AMD and NVIDIA

Before we get started, let's take a look at our test setup:

Platform: ASUS Rampage II Extreme, Core i7-965, 6GB DDR3-1333, Intel SSD
AMD Driver: Final 8.10 hotfix
NVIDIA Driver: 180.44

Our first goal in getting our testing rolling was to find out what to test and to understand the settings in the game better. We spent time playing the game at different quality levels with different hardware like we generally do. But because we wanted to take advantage of the benchmark tool, we decided to collect a bit of data on different settings with one card from AMD and one card from NVIDIA. We look at three different quality levels under two different DX APIs with two different AA settings across five different resolutions. For those keeping count, that's 60 tests per card or 120 tests total for this section.

The result isn't as much like our usual hardware focused tests, as this provides us with more of an analysis of the game itself. We get a better perspective on how the game responds in different situations with different hardware on different platforms without the need to test every piece of hardware out there. Our hope was that this page could help people who are running a particular setup see generally how performance might change if they tweaked one of the variables. Of course, you can't predict specific performance with this, as there isn't enough data for interpolation purposes, but knowing the general trend and what sort of changes make the largest differences can still be useful.

This test is run with our custom timedemo rather than any of the built in benchmarks.

The cards we chose are the highest end NVIDIA and AMD single GPU solutions (the GeForce GTX 280 and the Radeon HD 4870 1GB). While not everyone will have these cards, we were able to test the broadest range of playable data with them. We'll start our analysis with the NVIDIA hardware in DX9 and DX10.

Now take a deep breath because these graphs can be a little tricky. Each graph is only 6 resolution scaling lines, but you'll want to approach them by looking at two groups of three: blue diamonds, red squares, green triangles are no antialiasing, while purple X, blue *, orange circles are 4xAA.

Under DX9 and NVIDIA hardware, High quality performs significantly higher than Very High quality both with and without AA. Moving from Very High quality to High quality gives at best a 47% increase in performance while the worst case is 27% with 4xAA and 37% without. Performance increases in this case generally trend downward as resolution increases. We also see that High quality 4xAA outperforms Very High quality with no AA. While there is a crossover point, Very High quality with 4xAA also performs very similarly to Ultra High quality with no AA.

Moving to DX10 under NVIDIA hardware, High quality performance takes a dive while the rest of the numbers stay relatively stable. This basic indication here is that DX9 won't gain you much performance (and will sometimes drop your performance a bit) unless you are looking at High quality mode at which case it could be very worth it to run DX9. As a further consequence, the performance benefit of dropping down to High quality in DX10 mode makes it essentailly useless. High quality with 4xAA looses the advantage over Very High quality with no AA. Very High quality or better is the way to go under DX10, and DX9 should only be paired with High quality mode or lower.

The analysis of the AMD data is very similar to what we see with NVIDIA. We see the same big performance advantage of High quality DX9 with DX10 actually increasing performance at the higher quality levels (the exception is at 2560x1600 where performance drops off more sharply than the GTX 280). The major difference here is in the fact that moving from Ultra High quality to Very High quality gives you a much larger performance increase under AMD than NVIDIA. This means that Very High 4xAA has a larger advantage over Ultra High with no AA (except at 2560x1600), and that it is more worth it to drop back to a lower quality setting to gain performance on AMD hardware. We still recommend Ultra High quality though, unless 4xAA is something you just can't live with out (in that case, Very High quality plus 4xAA is probably the way to go).

The comparison we haven't made yet is NVIDIA versus AMD. These tests show that under DX10 AMD Radeon HD 4870 1GB is either higher performing than or performing on par with the NVIDIA GeForce GTX 280 (except at ultra high resolutions with 4xAA). This is very impressive due to the $100 price advantage (the GeForce GTX 280 comes in at 33% more expensive than the Radeon HD 4870 1GB at $400). If you've got a 2560x1600 monitor and want to run Ultra High quality with 4xAA, that's the only case where the GeForce GTX 280 is worth it, though you'll be pushing the playability limit there and SLI with two cheaper cards might be a better way to go.

Going forward, we will be looking at DX10 with Ultra High quality settings and will generally favor testing without AA as we feel that Ultra High quality is a better use of resources than 4xAA. For multi-GPU and high end testing, we will still try to include 4xAA numbers though. This custom timedemo will also be the test we stick with rather than the built in RanchSmall demo.

Index Testing with RanchSmall
Comments Locked

78 Comments

View All Comments

  • Brunnis - Saturday, November 22, 2008 - link

    I just wanted to point out that my HD4870 1GB stutters to the extreme with this game. I'm running an E8400 @ 3.8GHz, 8GB RAM and Vista 64-bit. It's however worth pointing out that there is no stuttering in DX9, only DX10. I've tried Cats 8.10, the second hotfix and now Cats 8.11. All have exactly the same problem, with horrible stuttering in DX10. The graphs from the benchmark tool clearly illustrate the issue with frequent spikes in them.

    I also have a friend with a HD4870 1GB and he has experienced the exact same problems. So, the HD4870 is definitely affected by AMD's shoddy drivers.
  • DerekWilson - Monday, November 24, 2008 - link

    the "stuttering" in the benchmark is part of the benchmark -- it just plays back with stutters in it. it's like that no nv, amd -- all GPUs. the only stuttering that is actually stuttering is what you experience in the game.

    the fps versus frame graphs on my 4870 1GB are smooth, while other cards show spikes.
  • Slash3 - Saturday, November 22, 2008 - link

    What resolution and graphic settings are you using? Is there a particular setting or gameplay situation that makes it worse?
  • Giacomo - Saturday, November 22, 2008 - link

    The stuttering issue, with my HIS HD3870 ICEQ3 (running all the latest Catalysts, on Vista x64), shows up at any resolution and quality settings under DX10. The problem shows up when you move around in the scenario: if you just stand, and look around, no stuttering; if you walk, a little stuttering sometimes; if you run, it gets worse; if you drive a car, it's horrible.

    Giacomo
  • RagingDragon - Saturday, November 22, 2008 - link

    Sounds like it's loading textures (or some other data) from main memory into video memory. For some reason they aren't being preloaded, or are being bumped from video RAM then have to be reloaded. That would also explain why the 512MB have worse issues than 1GB cards.
  • Slash3 - Saturday, November 22, 2008 - link

    Would it be possible to make available the custom demo created for this benchmark? A page describing the test setup for the benchmark systems would also be greatly appreciated. Thanks!
  • Willardjuice - Saturday, November 22, 2008 - link

    I don't have any issues running the game with my R700 using the second 8.10 hotfixes.
  • 4ccmusic - Saturday, November 22, 2008 - link

    Did anyone say this was the biggest let down of this year. If I wanted a game like GTA, I would get GTA.
  • toyota - Saturday, November 22, 2008 - link

    I have only played about two hours and I think I have had enough. Its a failure in my book.
  • Amuro - Saturday, November 22, 2008 - link

    I have a GTX 280 tri SLI system, so the game runs very good. However, I stopped playing it after just an hour of play. The graphics are a joke, not even half as good as Crysis and Warhead, and not to mention the entire game world has an ugly orange/yellow mixed with brown tint to it.

Log in

Don't have an account? Sign up now