Designing Our Next GPU Test Suite

240 Comments

View All Comments

• Brandonr87 - Sunday, March 14, 2010 - link

  I think Synthetic benchmarks are important, mainly because they can't test EVERY game, so for some people looking at what they can do on a standard scale may be helpful.
  
  As for games to test, I think there should be a wide breadth of games, but usually I just look at what games get similar frames to the ones I play, and go from there.
  
  I'd vote for Modern Warfare 2, and Dawn of Discovery (aka Anno 1404) but I know Anno probably won't make the list.

• floobit - Sunday, March 14, 2010 - link

  I develop HPC applications for chemistry. I'd like a benchmark that does many matrix operations using very large (10k x 10k) matrices. Another useful application would 4 threads of iterated floating point calculations, a la monte carlo or an MD simulation. In all of these scenarios the floating point variables would all need to be double precision.

• BitJunkie - Sunday, March 14, 2010 - link

  It would be really great if you could devise some kind of normalised performance rating that indicates performance relative to some reference hardware and can also be normalised for changes in the test suite.
  
  Something along the lines of:
  
  [(ave_rpg_perf / ref_rpg_perf) + (ave_fps_perf /ref_rpg_perf)]/test_suite_factor
  
  Everytime you devise a new test suite you run the old test suite and new test suite on the same range of hardware and take a statistical look at what the delta observe in the performance rating, then scale back the new results to be normalised with the old test suite. I might be missing some subtleties in this - but I just gave it 2 mins thought, sure you smart fellas can devise something a bit more robust and future proof...
  
  You'd then get a sense of absolute performance from direct FPS measurements and a sense of relative performance between generatiosn of hardware and test suites.
  
  I guess it would be the ATmark and ultimately not synthetic. You could insert other items into the equation such as render times and even weight these based on your perceptions of what people value.
  
  Would love to see somethign like this: you could circulate to key hardware vendors to gain their buy-in and then keep it confidential to avoid sniping from the side lines about your choice of weights and the validity of the formulation.
  
  Just an idea.
  
  

• samspqr - Sunday, March 14, 2010 - link

  I agree that a summary rating would be really useful, but I know it opens an old debate that was never truly closed: the results (of what card is faster than which other one) will depend on the fine details:
  
  * simple average vs. geometric average: I'd use geometric, because it's least affected by a big difference in one particular game, but I'm not totally sure it's the best option, and it definitely will affect the comparisons
  
  * the reference card used: in the old days, it would make one maker look better than the other dependin on whether you used a reference that was good for DX or OGL; now I guess it won't matter so much, but I'd pick an ATI one, as they are the ones dominating the market; and pick a relatively high-end one, otherwise you risk having to redesign your rating system too often
  
  * the weight of each resolution: does 25x16 really matter? I'd only use 19x12, but 12x10 and 16x10 are still popular...
  
  * the weight of each setting: again, I'd love to use AA+AF always, but for the review of a mid-to-low-end card it makes no sense
  
  * the weight of each game: you may think one is more important than others (in the old days that would be quake3)
  
  still, I'd love to see anandtech take a stand on each of these issues, and publish that ATmark in their reviews

• NetShroud - Sunday, March 14, 2010 - link

  I would like to see Team Fortress 2 and Mirror's Edge.

• DigitalFreak - Sunday, March 14, 2010 - link

  Source engine games are useless for benchmarking. Over 60fps is easily achievable even on crappy hardware.

• Hrel - Sunday, March 14, 2010 - link

  Bioshock 2 - Because I play it, I love it and it's new.
  Mass Effect 2 - Same reasons.
  3D Mark - because it's silly to not have an objective measurement to weigh the subjective games against.
  Then just keep up with the latest CryEngine game, not because I like the games, I don't enjoy them at all, but because it stresses systems.

• JonnyDough - Wednesday, March 17, 2010 - link

  Bioshock and Mass Effect 2 aren't really new anymore. But then again, neither are any of the games I play. I have yet to find a real good reason to upgrade my system and its based on fairly old tech - although it is waning a bit. Good games take a good bit of time to play and there are so many out there that are old now and will run on my current system.

• superccs - Sunday, March 14, 2010 - link

  1. performance/$
  2. silent and cool running heat sink designs availability
  3. special features that allow for greater usability, GPU video encoding, physics, or 3D type stuff
  4. Real in game detail, color, overall prettiness factor differences
  
  Things you can get rid of: SLI and Xfire setups, no one actually does that.
  
  Great hardware reviews, as always.
  
  :D

• jordanclock - Sunday, March 14, 2010 - link

  I know the issue of repeatable test runs makes it difficult, but I recommend a couple of MMOs to be benchmarked. For instance, Champions Online seems to do pretty well at putting stress on even the most up-to-date systems.
  
  Maybe you could talk to the MMO developers about setting up a sandbox server just for running benchmarks? It would be very nice to see how a certain video card performs beyond the tutorial zones or main cities.