Futuremark: of Benchmarks and Gaming
by Jarred Walton on January 30, 2008 2:00 AM EST- Posted in
- Software
What's in a benchmark?
At their core, modern 3D games consist of a graphics engine designed to spit out polygons onto your screen, and the internal workings of these engines can vary wildly from game to game. To discount the 3DMark graphics engines as "synthetic" is missing the mark; they are no more synthetic than any other gaming engine. The problem of course is that while other gaming engines are actually used in — at the very least — one game, 3DMark exists as its own entity.
What do performance benchmarks using The Witcher, Unreal Tournament 3, Crysis, or any other title really tell us? They tell us how specific hardware happens to perform in that title; no more and no less. Sure, gaming engines are often used in multiple titles, but once the developers are done modding and tweaking things, there's no guarantee that performance of the engine in one title will be the same as what we find in another title. Consider Gears of War, Bioshock, and Unreal Tournament 3 as an example; generally speaking, performance is similar but rarely identical.
 |
 |
 |
Things are further complicated when you consider that a benchmark of one specific area of a game/game engine doesn't tell you much about how hardware might perform in other areas. Take Oblivion as an example: performance in towns, dungeons, outdoor areas, underwater, and the gates can and does differ; if you want a truly comprehensive look at Oblivion performance, you would need to benchmark representative selections of all those environments. That's precisely what we did when Oblivion first launched, and we found that some while the GPU was clearly the bottleneck in outdoor environments, other areas demanded more of the CPU.
But is that really necessary? Most people are really only concerned with worst-case performance; you might be able to get 80 FPS or more inside a town, but if you drop to 15 FPS in outdoor environments it is unlikely you will be happy with the overall experience. We can substitute any other title in place of Oblivion — Crysis for example has some seriously taxing levels towards the end of the game, where frame rates might be less than one fourth of what you experience at the beginning of the game. The net result is that any benchmark is only a snapshot of performance.
When we look at performance in a variety of benchmarks, we do so in order that we might gain an overall impression of how hardware actually performs. History helps explain why we do this. Which is the faster architecture: Pentium 4 or Athlon XP? The answer: it depends on the application. Which GPU architecture is better: G70 or R500? Again, it depends what game you're playing. It makes our job easy when we get hardware that is clearly faster in virtually every situation, but that rarely happens. Driver and compiler optimizations can even help turn a mutt into a purebred — sort of like DNA tampering.
Facebook
Twitter
RSS
18 Comments
View All Comments
perzy - Wednesday, January 30, 2008 - link
A question for Anandtech:Why dont you say the truth: ATI/AMD cards s**k at OpenGL compared to Nvidia. ATI/AMD has an edge in DirectX instead.
It not just speed but IQ also in OpenGL.
I have found out this the hard way, through my wallet(had to buy a nvidia card because my ATIcard was so bad in UT99) and no thanks to any hardware site.
Are you so firmly corrupted that you cant print that?
Honestly I dont understand. I'm probably an idiot.
MrKaz - Wednesday, January 30, 2008 - link
Living in a rock lately?The ATI/AMD has now surpassed the Nvidia team in OpenGL.
Some thought would be impossible.
http://www.3dprofessor.org/Current%20Reviews.htm">http://www.3dprofessor.org/Current%20Reviews.htm
Also the ATI/AMD has made major steps in Linux that I have doubts that Nvidia is still the leader there too.
The problem is that websites like this one don’t update the tests/results, so everyone still thinks Nvidia is the ultimate king.
Hey AMD kicked Intel balls for 6 years and lots of people saw Intel products has the “better” during that time without any "knowledge".
I bet people like you still things ATI Opengl drivers still suck based on 1999 tests. Nice way to go out and live 2008.
greylica - Wednesday, January 30, 2008 - link
Open device drivers is the most important step to ensure a new openGL market will rise again. AMD still didn't give a totally free device driver, but the manual of certain cards to developers, as far as I know.I am waiting to see, but the results of the V8600 are impressive.
But the Consumer Radeon still s**ks a lot in OpenGL device drivers in Windows and in Linux. I am a Blender/Linux user.
Most of Nvidia cards ( even if it's not a quadro card ) is very good in OpenGL/Linux environment.
In the past I have problems with device drivers "optimized" for games in directX, and in the past there was an card that simply doesn't have the OpenGL client device drivers too.
On the other hand, the mayor problem of AMD/ATI is advertising here in Brazil. Intel is on some channels once a week. I ask:
Where is AMD products ?
Proteusza - Wednesday, January 30, 2008 - link
UT99? You mean 9 years ago, when 3D was still quite new? Um... Do you know how long 9 years is in computing? Please dont tell me you still dont buy ATI because of that!Also remember that, at the time, Glide was huge. So UT probably didnt run very well on anything except 3dfx cards. have a look at modern benchmarks with games like Quake IV and Doom 3. As far as I remember, there isnt much of a difference.
Another thing - how many modern games use OpenGL? I can only think of the 2 examples above.
teldar - Wednesday, January 30, 2008 - link
That and as far as IQ, is that supposed to mean AI? AI is based on other game code other than visual, and the GPU doesn't have anything to do with AI. That's all CPU based. So how is it AMD's fault if there was something wrong with the AI on your computer?T
chrnochime - Wednesday, January 30, 2008 - link
I think he meant I[/B}mage Q[/B}uality.crimson117 - Wednesday, January 30, 2008 - link
wonder if that will fix the bolding...
crimson117 - Wednesday, January 30, 2008 - link
maybe two will do it?