Why Average Frames Per Second Can Be Misleading

Timedemos make up the core backbone of our video card benchmarks. Timedemos are simply pre-recorded scenes that are re-rendered as the program plays them back. The example that we use here is the timedemo "radar" from the game Wolfenstien: Enemy Territory. Depending on the conditions of the testbed, radar might render more or less frames than another machine, but both machines will always take the same amount of time to play back. The program generally reports the number of frames rendered and we divide by the number of seconds that the timedemo lasted. This calculates our average frames per second.

Plotting more than just the average FPS is not a new concept. AMD, Intel, NVIDIA and ATI have all expressed their thoughts to us about plotting box-and-whisker or line graphs of our timedemos rather than just an average FPS. Of course, CPU manufacturers do not like timedemos at all, but that is an argument that we will leave for a different day. Let us claim during our timedemo that we record the instantaneous frames per second several times. "Instantaneous" may be a bit of a misnomer if you do not have an explicit math background, but for argument's sake, let's say that we are calculating how many frames we render in exactly one second and plotting those FPS over the course of a minute. The graph may look something like this.

Sample Videocard Capture

This line graph demonstrates a 60-second clip from a timedemo with 45 average frames per second. When we look at an average FPS, we expect the timedemo's instantaneous frames per second to look something like this. In reality, the timedemo may look something more like this extreme example below.

Another Sample Videocard Capture

Both graphs have an average ~45 frames per second. In fact, the lower graph actually averages to slightly higher. We make written notes of issues like this in our Windows GPU analyses, but a line graph puts it into a format that is less subjective.

Index Some Test Cases
POST A COMMENT

23 Comments

View All Comments

  • - Saturday, October 24, 2009 - link

    http://www.goph3r.com/mh">http://www.goph3r.com/mh
    (air jordan, air max, shox tn, rift, puma, dunk sb, adidas) nike jordan shoes 1-24 $32
    lv, coach, chane bag $35
    COOGI(jeans, tshirts, hoody, jacket) $30
    christian audigier(jeans, tshirts, hoody) $13
    edhardy(shoes, tshirts, jeans, caps, watche, handbag) $25
    Armani(jeans, tshirts,) $24
    AF(jeans, coat, hoody, sweater, tshirts)Abercrombie & Fitch $31
    http://www.goph3r.com/mh">http://www.goph3r.com/mh
    Reply
  • quanta - Friday, October 01, 2004 - link

    If you are going to compare image quality, what is the basis of comparison? That means that a reference (software) renderer is needed. But even then, choosing right technique means ones need to draw extremely detailed scenes similar to CAD programs, but such renditions will be too slow. And of course, there's the issue of how to objectively define the 'right' way vs. 'wrong' way of optimization in ways that most can agree, and can be done by computers.

    In any case, we can't afford yet another GPU benchmark that is easily exploited (in bad ways, that is).
    Reply
  • tygrus - Tuesday, September 28, 2004 - link

    Nice start.
    I get frustrated by companies that delay support for <5% of the market because of small numbers when its the lack of support that is causing the low numbers. If ATI improved the drivers for linux then more people would use their existing ATI cards under linux and more importantly more people will buy ATI for their linux system instead of Nvidia. It's the issue of "what came first, the chicken or the egg", when their attitude should be more like "build it and they will come".
    Reply
  • TrogdorJW - Tuesday, September 28, 2004 - link

    Just a quick question here, as I see a potential performance issue with the "hacked" OpenGL/SDL framerate utility. You mention that the software will automatically capture screen shots periodically. That ought to increase the demand on the HDD and potentially the rest of the system for brief instants when the screen captures are done. Hopefully there will be multiple benchmarking runs done, one with the screen captures and one without. Although, I suppose we have to see what sort of difference the screen captures actually make in performance first. :) Reply
  • raylpc - Saturday, September 25, 2004 - link

    I agree with fic. I'm going to buy a new AMD64 system and I will get Nvidia just because their linux support is way superior than ATi's, although ATi does make better cards. Reply
  • aaime - Saturday, September 25, 2004 - link

    I hope you can add some 2D benchmarks as well, since, for example, the 2D performance of the ATI drivers really suck! A 2D perf. benchmark should involve also use of Render, text antialiasing and such to be representative.
    Anyway, thank you, it's nice to see some attention to the Linux world from you :-)
    Reply
  • fic - Friday, September 24, 2004 - link

    Probably the big reason that only 4% of owners of ATI cards run linux is that their support sucks. This is a self fulfilling thing. ATI doesn't support linux therefore if you are going to run linux you don't use ATI. I image that a huge percentage of linux users use Nvidia because they have decent drivers. Reply
  • Brule - Friday, September 24, 2004 - link

    Great article. That's why they call it computer "science". Looking forward to the results on a personal linux-using level as well. Reply
  • javalino - Friday, September 24, 2004 - link

    Well. I hope this type of benchmark didnt effect the continuos windows benchmarks. As the ATI said this week, 4% of owers of ati cards run linux, so, not much people want to see linux benchmarks, unless they are better than windows ones.
    Reply
  • Illissius - Friday, September 24, 2004 - link

    Amazing. I've been looking for a tool like this for Linux for ages. Gimme now! Gimme now! Reply

Log in

Don't have an account? Sign up now