Final Words

Bringing things to a close, when AMD first came to us about this, we had mixed feelings about what they were proposing. Typically, manufacturers only take issue with benchmarking methodologies when they have some deficiency on their part. Surprisingly enough, AMD's stance was equal parts recognition of where they had issues and offering their input on the right way to do things.

Meanwhile, we’ve wanted to do an article like this for some time now; specifically a dissection of FRAPS so that we could better explain why we have not been using FRAPS for investigating stuttering. To our surprise, we found AMD repeating many of the positions we held, and more importantly they were offering up further facts and additional data that we didn’t have access to that would help support our own position. So despite whatever AMD’s intentions were, this worked out well for us.

Ultimately AMD’s message has been one of information, explanation, and admission of oversight. AMD has been clear with us from the start that the primary reason they even ended up in this situation is because they weren’t doing sufficient competitive analysis, and that they have revised their driver development process so that they now do this analysis to prevent future problems. The fact that NVIDIA seemed to have figured all of this out much earlier was a point of frustration for AMD. The company likely left non-negligable amounts of performance on the table over the years, which could've definitely helped in close races.

At the same time they’ve been hard at work on fixing existing stuttering problems, with many of those fixes being delivered while fixes for more DX10+ games are right around the corner. 

At the same time however AMD’s message is not just one about stuttering, but also one about benchmark and analysis methods with FRAPS. FRAPS, despite its limitations, has clearly exposed problems with AMD’s drivers that resulted in stuttering that AMD needed to fix. Meanwhile measuring frame intervals with FRAPS has become an increasingly common technique in reviews, only to really become popular at the same time as when AMD has finally fixed many of these issues.

AMD’s concern – and one that NVIDIA has shared with them in the past – is that measuring the rendering pipeline at the beginning of the pipeline like FRAPS goes about it does not accurately represent what the end user is seeing, due to the various buffers in the Pipeline and how the Present mechanism works. While FRAPS was good enough to pick up on the major stuttering issues in AMD’s drivers, as these issues get resolved it’s far too coarse a tool to pick up on finer issues, and in fact what FRAPS is now seeing is decoupled from what the user is seeing due to the presence of the context queue and other buffers. All of these, for the record, are points we agreed with AMD on, even before our meeting.

The end result of all of this is that change is in the air. Just as how quickly as Scott Wasson and others changed the nature of GPU reviewing by using FRAPS to measure frametimes, things must change again for GPU reviewers. If FRAPS is no longer an adequate tool to measure stuttering and frame intervals – as both AMD and NVIDIA insist – then new methods and new tools must be created to measure those factors at the end of the rendering pipeline, where the results would match what the end user is seeing. Though on the subject of tools, AMD for their part is favoring double-blind trials as the ultimate method of detecting stuttering. They’re fundamentally right since the perceptibility of stuttering depends on the person, but admittedly this is also the least objective/qualitative way of evaluating stuttering.

In any case, just as how change is in the air for GPU reviews, AMD has had to engage in their own changes too. They have changed how they develop their drivers, how they do competitive analysis, and how they look at stuttering and frame intervals in games. And they’re not done. They’re already working on changing how they do frame pacing for multi-GPU setups, and come July we’re going to have the chance to see the results of AMD’s latest efforts there.

For us this is what we hope to be the start of our own changes. There are tools in development that meet our criteria for better measuring frame intervals, and hopefully in the not too distant future we’ll be able to discuss those tools to a much greater degree, and to use those tools to go about measuring frame intervals in the manner we’ve always wanted to. But that is a story for another day, so until then you’ll have to stay tuned to find out.

AMD & Multi-GPU Stuttering: A Work In Progress
POST A COMMENT

103 Comments

View All Comments

  • eezip - Tuesday, March 26, 2013 - link

    First one! Wow, I'm lame. Thanks Ryan - keep 'em coming! Reply
  • B3an - Tuesday, March 26, 2013 - link

    Yes very lame. You should sit down and think what you're doing with your life and what kind of sad person you are. Reply
  • xaml - Tuesday, March 26, 2013 - link

    It reads as if you didn't, so why do you point a finger. Reply
  • stickmansam - Tuesday, March 26, 2013 - link

    I agree with the double blinding idea. Techreport had some videos on the skyrim stuttering and I showed it my bro with the card names covered and he actually preferred the AMD card. Personally I though both of them were playable since the 240fps video exaggerated any stuttering issues. If you can't tell the difference in a 60hz or 120hz video/monitor there is no difference.

    It would be nice if someone would develop an tool to measure the frames as they are being displayed, like as they are actually being viewed.
    Reply
  • krumme - Tuesday, March 26, 2013 - link

    The benefit of blindtest is twofold:
    It removes all the complexity involved in testing, and get to the point where it matters.
    Secondly we get an oppinion as to what the benefits of the game have, going to higher quality settings.
    Anand for much good, have the same staff, and we will get to know Ryan preferences in just a few rounds of testing.
    Then he can have a nice assistant changing the cards for him :)
    Reply
  • DanNeely - Tuesday, March 26, 2013 - link

    While not blind, HardOCP's maximum playable settings testing is done to capture this. They report min/avg/max/graphed FPS; but at whatever combination of settings gave the most eye candy while still being fast and smooth enough to be enjoyable. At times this has resulted in observations that "while the raw FPS numbers imply that turning on X should be doable the gameplay results indicated otherwise" (generally due to stuttering problems). Reply
  • Havor - Tuesday, March 26, 2013 - link

    I always liked HardOCP's maximum playable settings approach.

    But now i think that Ryan Shrout from PC Perspective is doing the best testing there is, by actually capturing all the frames with a capture card.

    So no testing @ the start as FRAPS dose or some ware in the middle, no realworld frame output, better then that you cant do.

    http://www.pcper.com/reviews/Graphics-Cards/Frame-...
    http://www.pcper.com/reviews/Graphics-Cards/Frame-...
    http://www.pcper.com/reviews/Graphics-Cards/Frame-...

    Its a real interesting read, and i hope they will start doing testing real soon, as hard numbers are hard to come by, as there is still no perfect way of testing frame times.
    Reply
  • Sabresiberian - Tuesday, March 26, 2013 - link

    "Playable" and "optimal" are different things; for the most part no one is suggesting the games and cards that have more problems with stuttering are "unplayable".

    And, some people don't notice what bugs the fire out of others. Stuttering is one of those things. I think this has a lot to do with the fact that these problems have existed for quite awhile and people just got used to them, so kind of automatically ignore them.

    So, I agree, if you don't notice it then it's not important. But if you do, then it is. :) I noticed this phenomenon years ago, and am very excited to see numbers that people can show quantifying the situation so that it can be discussed on more than a seat-of-the-pants level.
    Reply
  • Soulwager - Tuesday, March 26, 2013 - link

    The problem with double blinding is that some people notice more than others. If you're used to high end equipment on a 120hz monitor, you'll notice a hell of a lot more problems than dude off the street that normally plays on his laptop. Reply
  • medi01 - Wednesday, March 27, 2013 - link

    Last time I've checked on toms, AMD's GPUs were better in this regard.

    Looks like yet another article to "compensate" for 7790.
    Reply

Log in

Don't have an account? Sign up now