Designing Our Next GPU Test Suite

240 Comments

View All Comments

• Nickel020 - Sunday, March 14, 2010 - link

  I would love to see more information on how exactly you benchmark. While I generally trust Anandtech to to use meaningful scenarios that somewhat reflect real game performance, you only provide very little info on how you get your data. Or when you do, it's often hard to find as you only mentioned in one article.
  Looking at what other sites use as benchmarks I know that they're using a lot of timedemos as benchmarks that have about as much meaning as 3DMark, as they do not reflect the actual game performance.
  
  If you say that you don't want to use synthetic benchmarks as they're "meaningless" then you should explain how you benchmark and why your numbers really do reflect actual in-game performance. More transparency is the way to go here imho.
  
  Maybe you could do a "How we test" article that explains your test methdology.
  
  Also, it's not always possible to choose a "worst case" scenario as a benchmark, so it would be nice if you said how many FPS are actually necessary for stutter-free gameplay. This is for example necessary to give your WoW numbers meaning as right now we don't how a system that gets say 80 FPS in your test performs under non-repeatable raid circumstances.
  The goal of benchmarks should be to assess relative performance and absolute performance (i.e. how much performance do I need to play X at details Y?), and your articles often lack a bit in the absolute performance part. It's relatively easy to answer "What GPU should I buy for $200?" after reading your reviews, but you can't really be sure about how much you need to spend to get the performance you want.

• arnavvdesai - Sunday, March 14, 2010 - link

  I am not sure if you have seen the project out of Microsoft labs called Pivot. Its a way of seeing your data in different ways. If all of your data was compiled together it would be very beneficial to folks. For instance just adding your normal GPU reviews along with your chipsets processor etc. reviews would allow someone to combine all of them together in meaningful ways.
  For instance, it could allow someone to build a computer of the various parts you review and see how it would perform under the various scenarios you test against. This would mean someone could get a combination of various parts(some old, some new) and see where exactly how their computers would perform. I might sound like a shill but I seriously suggest you look at this technology. Its free to download atm I think. I am sure if you google Pivot+Microsoft Labs
  you would be able to check it out for yourself. I know this has got nothing to do with a new Test bed just a general suggestion on data presentation. I think you should add Metro 2033 when it comes out as it has Dx11 and supposed to be its own engine. Also Cry Engine 3 will be out and you should add that as soon as its available.

• Nighteye2 - Sunday, March 14, 2010 - link

  I think starcraft 2 would be a good addition, as many people will be playing it.
  
  Also, I would like to see some synthetic benchmarks included - not their overall scores, but their scores on certain functions. It would help explain differences in performance between cards across games. It helps to know why a card performs the way it does in certain games, and from that you can better predict performance in newer games.
  
  Also, I'd like to see scores for medium graphic settings - that people buying mainstream cards will likely be playing at. Some like all settings at high, but with low AA and AF and a resolution of 1280*720 - which looks excellent on a 1920*1080 monitor. But requires a lot less graphic horsepower.
  

• wxmanunr - Sunday, March 14, 2010 - link

  I'd love to see CPU Scaling results using several CPUs. A bargain one, mid-range, premium, and no holds barred.

• Paulman - Monday, March 15, 2010 - link

  +1

• GrizzledYoungMan - Sunday, March 14, 2010 - link

  I was recently burned pretty badly with my purchase of an ATI Radeon 5770, as part of a new Windows 7 build.
  
  To make a long story short, ATI's 5000 series is actually MANY TIMES SLOWER than most pre-2004 GPUs in 2D applications that rely heavily on 2D drawing, like Photoshop, AutoCAD and Illustrator. In short, programs I and many others use to make a living.
  
  Evidently a fix is available, but ATI isn't going to build it into their drivers till Catalyst 10.4 or 10.5.... nearly a full year after the release of the 5000 series. Utterly unacceptable.
  
  I think that part of the reason for this gimpy performance is the excessive focus on framerates and 3D performance in enthusiast sites. The ATI 5000 series was designed to win benchmarks, and it does. But some of us need to also us our PCs to work. And in that case, your options are to either buy a $1000 professional GPU that basically uses last years mid-range chip... or nothing.

• dubyadubya - Sunday, March 14, 2010 - link

  +1 2d performance has suffered for years. My old 9800 pro would wipe the floor with any newer GPU's I have tested. ATI 3xxx series and up are so slow in 2d you can see it. Nvidia cards were much better until the 1xx.xx and higher drivers came out. Nvidia borked something and its still borked to this day. Now we have Windows 7 with its new graphical subsystem. Sure it uses less system memory but 2d hardware acceleration is nearly a thing of the past. Some good reading here. http://www.passmark.com/support/performancetest/2d...">http://www.passmark.com/support/performancetest/2d...
  The trouble with testing 2d performance is a lack of a real world way of doing it. I use and support Passmark Performance test even though its synthetic. It would be nice to see 2d results on all cards tested. 90%+ of PC's time is spent in 2d mode so 2d is important! http://www.passmark.com/products/pt.htm">http://www.passmark.com/products/pt.htm

• GrizzledYoungMan - Sunday, March 14, 2010 - link

  Exactly - it's where we spend 90% of our time on PCs, and yet
  
  To give all you users an idea of what we're talking about here: if I try to select a bunch of objects in Illustrator, or move a complex shape in AutoCAD LT, my system slows to a halt as redrawing these elements takes forever. With my ancient 7800GTX, this worked beautifully - smooth and crisp and responsive.
  
  Nuts, I just made myself nostalgic for that 7800GTX. Single slot cooling, silent operation, drivers that WORKED, 4xFSAA at 1600x1200. It was like they had found the Terminator's head or something.
  
  But I digress. Point is: it might not even be necessary to design a test suite that accurately benches 2D performance, because that's not really what we're after. Rather, we just want a smooth user experience. So a single paragraph describing the subjective experience of those kinds of operations in Illustrator and AutoCAD relative to a GPU that is known to do it well (the 9800 series isn't a bad choice) might be good enough.

• GrizzledYoungMan - Sunday, March 14, 2010 - link

  EDIT: But some of us need to also us our PCs to work. = But some of us also need to use our PCs to work.
  
  Evidently a hangover and righteous indignation is a bad combination.

• superxero044 - Sunday, March 14, 2010 - link

  its one of the best looking games around