Designing Our Next GPU Test Suite

240 Comments

View All Comments

• T2k - Thursday, March 18, 2010 - link

  [quote]
  I'm going to start off the comments here with one condition: no synthetic benchmarks. Our editorial policy continues to be that we only want to use real games, as synthetic benchmarks just encourage AMD and NVIDIA to focus on optimizing for something people can't play. So please don't bother asking for 3DMark. [/quote]
  
  EXCELLENT!
  
  MAY I SUGGEST A SIMPLE RULE?
  For any benchmark to be considered it must be based on an unmodified engine of a published (NA or EU or Asia) GAME.
  
  That's it.
  
  This means the end of 3DMark, Unigine Heaven, SiSoft Sandra etc in this benchmarking suite.

• slickr - Wednesday, March 17, 2010 - link

  ok.
  In the RTS scene add: Starcraft 2(beta for now) and Napoleon Total War. These 2 titles, along with the current DOW2 are good enough.
  In the Action/FPS scene add: Uncharted 2, Assasins Creed 2, Aliens vs Predator and Metro 2033. Keep only Crysis.
  In the RPG scene add: Dragon Age Origins, Mass Effect 2 and Demon's Souls.
  In the driving/racing scene add: GTA4, NFS: Shift and Dirt 2.
  In the MMO/hybrid scene add: Star Trek, Sims 3, X3.

• fepple - Tuesday, March 16, 2010 - link

  I'd be interested to see a one off article on synthetics. Not really the high level 3dmarks I'm hoping there are some real low level synthetic benchmarks around you could use to test specific parts of each card? Could then extend that to look at how each of those components go together to draw the finished scenes in games, with potential to explain the difference between certain games and then what bottlenecks are restricting each one. I realise this would be a load of work but your pretty much only people that could do it :)

• Ryan Smith - Wednesday, March 17, 2010 - link

  For a major architecture article (e.g. Evergreen launch, GF100 launch, etc) we'll sometimes use tools like that to look at the capabilities of the architecture. But this isn't something we would do on a per-card level like we do with our GPU benchmark suite.

• blasterrr - Tuesday, March 16, 2010 - link

  i would like to see minimum framerates and i would like to see most of the tests in a resolution and a quality setting that is playable.
  So in most of the tests minimum framerates should not in the 25-30 fps range and avg frames should be in the 50-60 fps range.
  If minimum framerates are always in the 30 fps range the avg. fps do not have to be as high.
  
  And typical demanding gameplay situations should be tested. It s not sufficient for a computer to run a game at 60 fps in most situations but also in critical gameplay situations where you have more load on the cpu and gpu, where fluid gameplay is even more important than in a scenario, where you don t have to react in a fast and fluid way a computer has to be able produce certain minimum framerates.
  In this scenarios it is often the case that some graphics cards in the charts would switch places.

• TheHolyLancer - Monday, March 15, 2010 - link

  bring in wow (no joke) on 800*600 -> 1920*1200 and then crysis/warhead/2 on 1680*1050 -> 2560*2048 to cover the top end and low end with varying settings at each res (at least 3 AA/AF combos for each res), this will act as the "synthetic" benchmark of low gfx demand and high gfx demand. If wow is undesirable as a bench due to the connectivity to server issue or lack of benchmarking tool (maybe use some of that tools that created them funny videos with? them machinma deals?), replace with CS 1.6 or freelancer or other low gfx demanding game.
  
  then bring in some 1920*1200 goodness with max AA / AF with L4D2 for source, bioshock 2/ME 2 for unreal, MW2 , CIV IV or V, C&C 3/4/RA3, SC II, Sup Com II.
  
  the above OR if you want to be less hardcore with your benchies, the above games but under 1680*1050 with mid AA / AF instead of 1920*1200 under max AA/ AF

• Antryg - Monday, March 15, 2010 - link

  Test each game at 2 resolutions & 2 quality-settings: that shows you how each card scales/doesn't-scale with both quality & resolution!
  Anything less than 2x2 and you end up blind to the actual curve of performance for that card.
  Anything more than 4 and you are filling-in the curve, instead of diversifying one's dataset with more games/apps.
  
  Bonus rule: make 3/4 of the games be the "problematic" ones ( like that GTS or whatever it was called, racing-game someone mentioned, that doesn't work right on ANY cards... ) so that the mindless-enthusiasm we are biologically prone to can be torpedoed directly with some facts of life...
  
  Cheers!

• jive - Monday, March 15, 2010 - link

  I find it really odd that in this particular case the testing methods of Anandtech differ from the objective and scientific approach you normally take.
  
  What point is test against some game titles when your argument of not to include 3DMark is not to include title for which the GPU vendors optimise their drivers for? Last time I checked the release notes of GPU driver it consist a huge list of specific titles where the driver brings in some improvement. What is that if not optimizing for single title? I'm not saying you should include 3DMark but the decision to omit synthetic benchmarks is done using exceptionally poor argument which is almost utterly without any ground.
  
  What I want to see from GPU review is the efficency of the HW/SW implementation for some standard API. If you test against a 'real' applications you end up testing the application's implementation of the certain API and/or the driver optimisations for the application under test.
  
  If your site really wants to differentiate from the mass of HW sites do what you have done in server and storage benches, create your own. Use only the standard compliant ways to use the API and run with different CPU setups to reveal the effects of the platform.
  
  If not, you're blending into the gray mass of uninteresting reviewers of batch run game benchies ...

• Targon - Monday, March 15, 2010 - link

  The big problem(which you touch on), is that the drivers are tuned to specific games/applications for performance. As a result, since not everyone plays the same games, it is sometimes a good test to use BOTH titles that the drivers are tuned to do well with, and those that are not.
  
  I am talking here about games like The Witcher, which did not see enough advertising here in the USA, but would be good to test performance on. Do you think AMD or NVIDIA tuned their drivers for that game? Other games that require a lot of GPU power but do not have specific driver optimizations from AMD or NVIDIA are good to test with as well.
  
  I am in the minority it seems, in that I am not a fan of first person shooter, or World War II combat games. As a result, when I find a game, as often as not it hasn't gotten much attention when it comes to checking the GPU performance with games _I_ play.
  
  So, Synthetic Benchmarks have their place in a comparison, but looking for lesser known titles to use in testing would also say a lot.
  

• ellarpc - Monday, March 15, 2010 - link

  I want to start off by saying I've been an Anandtech reader for most of a decade. I disagree with your stand on 3dmark. It really is a lot of fun to run. I would like to see synthetic benchmarks added. Just because games are different than 3dmark doesn't mean end users don't want to know the potential bragging rights a video card has to offer compared to the others. I love to play games but I also love to overclock to the max and run synthetics. When buying a video card I generally go for the one that scores higher because I enjoy running benchmarks almost as much as playing some games. I can spend hours tweaking just to lift another 1000 points. I go to overclockersclub to get my information on the futurmark bench results and I'd rather not have to stray from Anand. Anandtech is really the only place that doesn't run them. Your stand on that really is narrow minded. I may not play 3dmark but I use it alot for fun. I hope AMD and NVIDIA continue to optimize for it just as they do for games.
  
  on another note:
  I would also like to see BFBC2 GPU showdown. A list of what will play it what won't.
  Thanks anand. Love you guys alot.