Introduction

Here at Anandtech, performance testing is an important aspect in how we review our hardware. When we are looking at graphics cards, game benchmarks give us a good idea of a card's performance capabilities. We are always interested in new ways to test graphics hardware, and while we often use game benchmarks, there are other tools out there that can be useful for testing computer hardware. One of these programs is 3DMark, a popular benchmarking tool developed by Futuremark, and today marks the release of their latest version of this program, 3DMark06.



There is a kind of interconnectivity between hardware and software wherein each tends to affect and be affected by the other, specifically regarding gaming technology advancements. There are times where certain games or software come about, which test the limits or surpass the capabilities of the graphics hardware at that time. Currently though, we are seeing the opposite situation, where incredibly powerful graphics cards and gaming setups (i.e. SLI, Crossfire) surpass the system requirements of even the most demanding games with the highest settings enabled.

While perhaps frustrating for some people, scenarios like these are are generally good news for the end user, as it creates opportunities for major advancements to occur in the technology of games or game hardware. Right now, the hardware that is available is prompting advancements in game development, and we can't easily predict what types of games we might see in the near or semi-near future. Tools like 3DMark are useful because they give us the ability to test hardware in very different and precise ways that current games cannot.

That being said, 3DMark is essentially a benchmarking tool, and there are limits to its usefulness. We'll talk more about this and run some benchmarks across several graphics boards to give us an idea of how this program stresses our cards. We'll also be talking about some new features in this edition of 3DMark, which look impressive, particularly the High Dynamic Range and Shader Model 3.0 additions.



3DMark06

3DMark is a program that has been around for a while now and Futuremark has just released the newest edition. 3DMark06 has some new features, but it's essentially the same reliable tool that it has been in the past. The benchmarking demos have been updated graphically and look very impressive, and interestingly, there is a playable game included in the program this time around.

This version of 3DMark adds some graphical enhancements to the three demos from 3DMark05, and adds a new demo of an impressively rendered arctic outpost. The other three demos are: "proxycon", a futuristic shoot-out scene; "firefly", a night scene depicting two fireflies in a forest; and "canyonflight", which shows an airship (very reminiscent of the one in The Mummy Returns) encountering a huge sea serpent.







The images above are comparisons between 3DMark05 and 3DMark06, and show the improved SM 2.0, HDR and SM 3.0 enhancements to the demos. The High Dynamic Range additions are particularly impressive, and the new shadow effects in each of the demos look very nice as well.

Of course, Futuremark does a great deal of research when deciding how to implement a feature. Unfortunately, no one can predict what all other game developers will end up doing (let alone the way in which they will go about doing it). The HDR implementation, for example, is based on a ground-up approach with full FP16 render targets. This allows them to render reflections and refraction of HDR light sources with bloom, lens flare, and all those great HDR effects that we've come to know and love. Tone-mapping is applied at the end as a post processing step to render the floating point HDR framebuffer data out to an integer display. While all of this is fine, there are some issues with the approach. First, unless some form of supersample AA is used, only ATI's hardware can perform multisample FSAA on an FP16 render target. For this reason, many game developers have opted to avoid such an approach. Also, while both NVIDIA and ATI hardware can do floating point blends, only NVIDIA hardware can perform hardware filtering on floating point render targets.

On top of that, one of the most effective real world HDR implementations that we've seen so far has relied on a dynamic exposure rather than floating point precision (Valve's Source HDR). There is some overhead involved, but the effect is quite good, while allowing full filtering and FSAA on all hardware without the need for custom shader programs to reinvent the wheel. Arguably, 3DMark06 might show a picture of performance on current hardware after game developers no longer care about making all the features work across the board on this generation of GPU, but this is a bit of a stretch and its likely that much more will have changed by that point.

The game that is included is another addition to 3DMark06, but is so poor that it almost couldn't be called an actual game. It is basically a robot shooter game set in a rocky landscape, which admittedly is well-rendered, where you have to shoot little flying robots that zip around and are frustratingly hard to hit. The movement and controls are incredibly frustrating and the game is so boring and confusing that it doesn't really warrant any playing time at all, and it seems that it was only included as some kind of afterthought or proof of concept.

One thing that we want to touch on is the fact that there is some controversy over 3DMark, specifically whether or not it is best suited for testing performance between different types of graphics cards. We at Anandtech don't typically use 3DMark in our graphics card performance tests because we feel that it is not the best measure of real-world performance. While it does give an accurate depiction of the capabilities of a given card, it stresses the cards in ways that no games really do right now, in an attempt to predict what future games may implement. Because of the fact that video cards are ultimately for playing games, it can be argued that a consumer would have a much better idea of what card to buy for their gaming setup by seeing game benchmark results over 3DMark's results. This is our philosophy, and for comparing graphics hardware, we will rely on real world tests over synthetic benchmarks.

However, all this aside, 3DMark06 is a remarkable program in its own right. Feature analysis, stress testing, and image quality comparisons are all useful applications of 3DMark06. For quite some time, we have used 3DMark in system level tests as well. But that's enough on the software. Let's look at the kind of performance that we see with it.



Performance Tests

We ran several tests with the latest version of 3DMark to see what kind of results we would get across a range of NVIDIA and ATI cards. There are a number of different tests that 3DMark can perform to stress different aspects of the system, but the three tests that we will be focusing on here are Shader Model 2.0 graphics, HDR/Shader Model 3.0 graphics, and the overall score. Here is what we tested:

NVIDIA 6600 GT
NVIDIA 6800 GS
NVIDIA 7800 GT
NVIDIA 7800 GTX
NVIDIA 7800 GTX 512
ATI X800 GTO
ATI X1300 Pro
ATI X1600 XT
ATI X1800 XL
ATI X1800 XT

MSI K8N Neo4 Platinum/SLI motherboard
AMD Athlon 64 FX-55 Processor
2x512MB OCZ 2-2-2-6 1T DDR400 RAM
Seagate 7200.7 120 GB Hard Drive
OCZ 600 W PowerStream Power Supply

The 3DMark06 benchmark is very straightforward. There are different settings that can be changed such as resolution and Anti-Aliasing, and clicking "Run 3DMark" will run the demos back-to-back and display the results when finished. For these tests, we chose the default resolution of 1280X1024 (no AA or AF).







(Note that in the HDR/SM3.0 graph, the X800 GTO is not included as it is incapable of rendering these effects.)

We can see by the graphs that 3DMark's graphics tests favor NVIDIA hardware over ATI, which is a further illustration of why 3DMark isn't really the best program for performance testing between cards. We know that different games tend to do better or worse on different hardware depending on the way the game was made, so a tool like 3DMark won't necessarily determine which card would be best suited for a specific game. This is an important fact for users in the market for a new card to realize before they spend their hard-earned money.

3DMark06 is a very stressful benchmark and as could be expected, some of the less powerful cards ran the demos very slowly. Because the graphics are so intensive, the effects were quite stunning, especially on the high end cards like the 7800 GTX (512) and the X1800 XT. During the firefly scenes, the warm glow of the fireflies in the night created intricate, hypnotic reflections of the foliage, and the brilliant white of the snow and sun in the arctic outpost demo at the end were striking. But at the end of our testing, we are mostly left with shallow beauty rather than a deep, meaningful connection. The overall 3DMark scores don't really give us much more information than we already have; other than simply letting us know what hardware runs 3DMark better.



Final Words

As we've said before, 3DMark is a somewhat specialized benchmarking tool with the ability to measure a lot of different aspects of a gaming system, and while it may not be best suited for testing performance between cards, there are many other things this program is useful for.

3DMark06 is quite useful for doing more focused comparisons with hardware components. For instance, comparing 3DMark results between a card and an overclocked version of itself can give a good idea of how a given card's clock speeds scale. Another use would be for testing drivers and to determine what kind of improvements certain features may have had between updates. We typically use looped game benchmarks when testing the stability of a graphics card while overclocking, but 3DMark would make a good tool for this as well. With 3DMark's demos, any graphical tearing or visual anomalies would very easily be seen. By taking screen shots, 3DMark would also be a great tool for comparing image quality in Anti-Aliasing or filtering for example.

There are likely many other uses for this program, which we can't mention here, and there is no doubt that 3DMark will remain a popular benchmarking program. Our uses for this program mostly involve more specific feature comparisons rather than those between the performance of different cards. Again, real-world tests show how 3DMark test scores don't really reflect actual performance in a game, particularly when you consider that different games will always favor different graphics hardware.

The bottom line is that a graphics card was made for playing games. Futuremark has developed a nice tool with excellent graphical elements in this latest version of 3DMark, which hopefully game makers will aspire to achieve in future games. Regardless of how you use it, 3DMark06 shows off some very impressive graphics and is a definite improvement over 3DMark05 both in visual quality and in the types of performance tests used. We certainly won't be focusing on 3DMark scores in future graphics card comparisons, but we may see some of the feature set tests or image quality comparisons pop up down the road.