Designing Our Next GPU Test Suite
by Ryan Smith on March 14, 2010 12:00 AM EST- Posted in
- Ryan's Ramblings
In keeping with our desire to refresh our GPU test suite periodically, we’re going to be redoing our GPU test suite to rotate in some more modern games, along with rotating in some DirectX11 games capable of taking advantage of this generation of GPU’s full capabilities. And while we already have a pretty solid idea of what we’re going to run, we wanted to throw out this question anyhow and see what responses we get.
What games would you like to see in our next GPU test suite, and why?
What we’d like to see is whether our choices line up with what our readers would like to see. We can’t promise that we’ll act on any specific responses, but we have our eyes and ears open to well-reasoned suggestions. So let us know what you think by commenting below.
Facebook
Twitter
RSS
240 Comments
View All Comments
mmatis - Monday, March 15, 2010 - link
Pong:http://www.play.vg/games/130-Flash%20Pong.html">http://www.play.vg/games/130-Flash%20Pong.html
-or-
http://www.falstad.com/pong/">http://www.falstad.com/pong/
???
}:-]
Froboz - Monday, March 15, 2010 - link
I find that I often need to adjust the settings so that the game plays well during intense scenes where the system is taxed. Measuring average FPS doesn't give the best indication of what resolutions and quality settings can actually be used for regular gameplay.It would be nice to use some of the more intense scenes in games and to report the minimum frame rate or 5th percentile of frame rates.
Rogerdodge1 - Monday, March 15, 2010 - link
ive seen a few good suggestions but had some of my own....most people agree EE cpus are no good for most of us. i think you need 3 tiers for your reviews.
sub $100 cards get sub $100 cpus, maybe an athlon x2 at 2.7-3.2ghz. cards from $100-$250 get either a phenom2 945 or maybe a C2Q 8400. cards over $250 get something like an i7 870.
resolutions scale with the cards as well.
sub $100 cars are more likely to be used with lower resolutions like 1280*1024, 1440*900 and maybe 1680*1050
$100-$250 cards will probably have at least a1680*1050 screen with them and possibly 1920*1080 or 1200.
cards over $250 will usually have 1920*1080 or higher so use that, 1920*1200 and 2560*1600, or go with dual or triple 1080p displays instead of 2560*1600.
when doing crossfire or sli you should definitely use 2560*1600 or dual/triple 1080p monitor setups(maybe even dual/triple 2560*1600 if you have the money to piss away, or can get them as freebies :P )
as for the games?
there are several game engines out there and i think you should try to find the 2 most demanding (graphically of course) for each of the top 5 modern engines. if performance is very close between the top 2 then only use 1 in you benchmarking suite, but if performance is significantly different use both. also include 1 title each from the most popular 5 older engines. that gives you a max of 15 games, and more likely only 10. also if you include a chart showing what other games use the same engines as the ones you tested,then people can get some idea where those would fall performance-wise without having to test 30 or 40 games every time you review a card. if a certain game is particularly popular but uses a less widely used engine, some consideration should be given to including it(maybe it could take the place of a popular but not graphically intense engine like the source engine?), but we all understand that you can only do so much in the time you have.
many people seem to want gpgpu testing on these cards and i agree that having that information would be nice, but there really aren't many programs that use the gpu that are hardware agnostic, so.... if you can find some useful ones that are hardware agnostic(openCL, direct compute, whatever), great, let us know what they are. if not, oh-well.
as far as 2d performance goes, i don't believe that we need it for every card you test. maybe 1 mid-range card from each series?
Thank You for all your hard work keeping us up to date on things that matter in the world of computers, and i look forward to seeing what you guys decide to go with.
Tech on Napkins - Monday, March 15, 2010 - link
1. Use a poll to find the median platform being purchased by your readers. If no one has a Core i7-980, it's not giving the buyer a true sense of what to expect if you use it as your video card test station. If you find a game IS cpu limited, mention it, which brings me to point...2. Have Battlefield: Bad Company 2 as a benchmark. I see it as an example of things to come, where physics takes a much stronger role in the experience. It pins my overclocked e8500 to the near 100% on both cores. While generally smooth, the occasional chaotic scene can get a bit choppy, which brings me to point...
3. Minimum FPS is key. It was my CPU that was most likely the bottleneck in those scenes above, but I've seen video cards that benchmarked just fine, but they had severe performance issues when you looked at minimum FPS. Is it running out of RAM? Out of RAM bandwidth? CPU bottleneck?
Wixman666 - Monday, March 15, 2010 - link
A lot of the people here that read your articles (including myself) are system builders.Run benchmarks on a couple of mid-range CPUs, preferably overclocked. Such as an AMD 550/555 BE OC to 3.6/3.8 and then an I5 750 OC to 3.6. Something easily attainable by anyone using a stock cooler with minimal effort.
Software wise, I'd like to see a World of Warcraft benchmark. Why? Because 12 million people play, and I sell more computers to WoW players than any other single segment.
I know someone is going to say WoW will run on a toothpick and a rubber band, but I'd still like to see it included. 12 million people is a big reader base.
Not just a running around in the middle of nowhere benchmark, I'm talking a 25 man ICC raid measure.
MrSpadge - Monday, March 15, 2010 - link
I'd like to see the GPU power consumption measured directly as XBit-Labs does it.It might also be interesting to check the power consumption under full load at auto fan speed (stock) and at maximum fan speed (for 24/7 GP-GPU crunchers). For GF100 rumors say that half of its power consumption is due to leakage, which increases at higher temperatures. For GT200 I've already seen load differences of ~20W just due to fan speed.
Ananke - Monday, March 15, 2010 - link
Hello,I played Crysis when it came out. Since then I had GTX260 892MB, HD4850 1GB, HD4570 1GB, GTX285 OC, and currently HD5850. Crysis is beautiful game, and even if it sounds retarded, for my own needs I always play with my new card Crysis first - just to have the perception and be able to compare with previous card that I've own.
I play on 1920*1280 (because the monitor native resolution) with all settings on highest. This test worked for me and allowed me to have a trend.
My idea is simple - use a graphically sophisticated game on several resolutions 720p and above with average eye candy first, and maximum settings at second. Get the popular display resolutions from Steam usage statistics if you like. Use a game that heavily tax the GPU, maybe a DirectX 11 one (if available). The point is not to give me a benchmark in different classes of games, but ONE or few games on many cards - so I have consistency and ability to compare.
IMPORTANT - please add same GPGPU applications. Does the cards have any computational value besides the gaming. If, for example, two cards have similar performance but one is accelerating handily a bunch of useful applications - this is a value for some users. Bench that, mention which software can use GPU for advantage, so users later comparing prices can draw their own purchase decisions.
I personally, like to transcode with HD5850 an h.264 to PSP or phone video a 4.37 GB file in 17 minutes! This has value for me above the GTX285. I also find value in a card running at 42 C temp at load versus another working at 80 C at load and burning all my PC internals.
Ananke - Monday, March 15, 2010 - link
my bad, the third card was HD4870, not 4570isaac12345 - Monday, March 15, 2010 - link
It would good to have different configurations at different price points. Not everyone has an extremely high end system and it would be good to get an idea of how the GPUs fare with different components at different price points. You could have 3 or 4 configs like midrange, lowrange, high end and no bottleneck config.Ardemus - Monday, March 15, 2010 - link
My main request isn't a particular game. I want some particular stats.1) Obtain a moving average of FPS/Second
2) Find %time over 10, 20, 30, and 60 FPS
3) Find average and standard deviation of the duration of sequential time spent under each rate.
Data for one game would look something like this, and would make a nice pair of graphs in aggregate.
GameX 94%>10fps 82%>20fps 75%>30fps 12%>60fps
+ 3s 11s 2m 14m
Ave spans .1s<10fps 12s<20fps 6m<30fps 8m<60fps
- 0s 11s 5m 7m