The Test

For the purposes of our testing we’ll be looking at the 6 games we’ve adopted for use with FCAT due to their proven reliability. These are Total War: Shogun 2, HItman: Absolution, Sleeping Dogs, Battlefield 3, Bioshock Infinite, and Crysis 3. All of our results unless otherwise noted are using Catalyst 13.8b1 for the AMD cards, and NVIDIA’s 326.19 beta drivers for the GeForce cards.

Our metric of choice for measuring frame times and frame pacing is a metric we’re calling Delta Percentages. With delta percentages we’re collecting the deltas (differences) between frame times, averaging that out, and then dividing delta average by the average frame time of the entire run. The end result of this process is that we can measure whether sequential frames are rendering in roughly the same amount of time, while controlling for performance differences by looking at the data relative to the average frame time (rather than as absolute time). This gives us the average frame-to-frame time difference as a percentage.

Bioshock Infinite - Delta Percentages - 2560x1440 - Ultra Quality + DDoF

In general, a properly behaving single-GPU card should have a delta average of under 3%, with the specific value depending in part on how variable the workload is throughout any given game benchmark. 3% may sound small, but since we’re talking about an average it means it’s weighed against the entire run, as the higher the percentage the more unevenly frames are arriving. For a multi-GPU setup we’d ideally like to see the delta percentages be equal to our single-GPU setups, but this is for the most part unreasonable. There is no hard number for what is or isn’t right here, but based on play testing we’d say 15%-20% is a reasonable threshold for acceptable variance, with anything under 10% being very good for a multi-GPU setup.

Finally, in our testing we did encounter an issue with Catalyst 13.8 that required we make some slight adjustments to FCAT to compensate for this bug, so we need to make note of this. For reasons we can’t sufficiently explain at this time but has been confirmed by AMD, in some cases in Crossfire mode AMD’s latest drivers are periodically drawing small slices of old frame buffers at the top of the screen. The gameplay impact is minimal-to-nonexistent, but this problem throws off FCAT badly.

To quickly demonstrate the problem, below we have two consecutive frames from one of our Battlefield 3 runs. The correct FCAT color order here is dark blue, green, light blue, and olive. The frames corresponding to dark blue and green occur on frame one, and light blue and olive on frame two. Yet looking at frame two, we see a small 6 pixel high stripe of dark blue at the very top of the image. At this point the dark blue frame should have already been discarded, as the cards have moved on to the green and later light blue frames. Instead we’re getting a very small slice of a frame that is essentially 2 frames old.

The gameplay impact from this is trivial to none; the issue never exceeds a 6 pixel slice, only occurs at the top of the frame (which is generally skybox territory), and is periodic to the point where it occurs at most a few times per minute. And based on our experience this primarily occurs when a buffer swap should be occurring during or right after the start of a new refresh cycle, which is why it’s so periodic.

However the larger issue is that FCAT detects this as a frame drop, believing that over a dozen frames have been dropped. This isn’t actually possible of course – the context queue isn’t large enough to hold that many frames – and analysis shows that it’s actually part of the old frame as we’ve explained earlier. As such we’ve had to modify FCAT to ignore this issue so that it doesn’t find these slices and count them as dropped frames. The issue is real enough (this isn’t a capture error) and AMD will be fixing it, but it’s not evidence of a dropped frame as the stock implementation of FCAT would assume.

Ultimately our best guess here is that AMD is somehow mistiming their buffer swaps, as the 2 frame old aspect of this correlates nicely to the fact that the dark blue and light blue frames would both be generated by the same GPU in a two-GPU setup.

CPU: Intel Core i7-3960X @ 4.3GHz
Motherboard: EVGA X79 SLI
Power Supply: Antec True Power Quattro 1200
Hard Disk: Samsung 470 (256GB)
Memory: G.Skill Ripjaws DDR3-1867 4 x 4GB (8-10-9-26)
Case: Thermaltake Spedo Advance
Monitor: Samsung 305T
Video Cards: AMD Radeon HD 6990
AMD Radeon HD 7970GE
AMD Radeon HD 7990
NVIDIA GeForce GTX 590
NVIDIA GeForce GTX 680
NVIDIA GeForce GTX 690
Video Drivers: NVIDIA ForceWare 326.19
AMD Catalyst 13.5 Beta 2
AMD Catalyst 13.6 Beta 2
AMD Catalyst 13.8 Beta 1
OS: Windows 8 Pro

 

Catalyst 13.8 Beta 1: The First Multi-GPU Frame Pacing Driver Catalyst 13.8 Results in Summary
POST A COMMENT

102 Comments

View All Comments

  • boot318 - Thursday, August 01, 2013 - link

    I've read a couple people got "black screened" when they did this update on one GPU. I'm not saying that will happen, but you better prepare for it if you do. Reply
  • Bob Todd - Thursday, August 01, 2013 - link

    I may have missed this when I skimmed through the results, but have you heard anything about rough estimates from AMD about a frame pacing release supporting Eyefinity (e.g. Q4, H1 2014, etc.)? I know it's still a tiny percentage of users, but there are relatively cheap 1080p IPS panels now so building a nice looking 5760x1080 setup is pretty affordable these days. After playing games this way, it's something I wish I had done earlier, and I'm eager to see a frame pacing driver supporting this setup. Reply
  • Ryan Smith - Thursday, August 01, 2013 - link

    Sorry, AMD didn't give us an ETA on that one. Let me see if I can still get one out of them. Reply
  • DanNeely - Thursday, August 01, 2013 - link

    HardOCP says DX9 and Eyefinity support should be available in a driver update later this month.

    http://www.hardocp.com/article/2013/08/01/amd_cata...
    Reply
  • DeviousOrange - Thursday, August 01, 2013 - link

    I am hoping this will also improve Dual Graphics, will give it a test over the weekend. Reply
  • Homeles - Thursday, August 01, 2013 - link

    Well I'll be damned. They did it. Not quite as good as Nvidia, but at this point, the difference isn't really one worth mentioning. Reply
  • xdrol - Thursday, August 01, 2013 - link

    The link is bad for the driver, please remove "-auth" from the URL. Reply
  • chizow - Thursday, August 01, 2013 - link

    Like watching a baby crawl. Good first step for AMD, but still a long way to go.

    AMD and their fans can thank the press (mainly TechReport and HardOCP, sorry Derek, you guys were way late to the party and still not fully onboard with FCAT measurements) and Nvidia fans for making such a big stink of this. Lord knows AMD and their fans were too busy looking the other way to address it, anyways.

    Hopefully AMD and their fans take something away from this: if you want to improve your product, don't try to sweep it under the rug, address it, own it, and demand a fix for it.
    Reply
  • chizow - Thursday, August 01, 2013 - link

    Sorry my above post should reference the author Ryan, not Derek (was thinking of your predecessor), when referring to AT not being at the forefront of this runtframe/microstutter issue.

    Also, I feel the accolades given to TechReport, while not completely undeserving, should also be given to PCPer's Ryan Shrout and some of the German publications like PCGamesHardware. While TechReport did start the ball rolling with some new ways to measure frame latency/microstutter, Ryan Shrout really harped on the runtframe issue until Nvidia worked with him in unveiling FCAT. Also, the German sites have been hammering AMD for years about their much worst microstuttering in CF, largely ignored by the NA press/blogs. And finally Kyle at HardOCP has said for years SLI felt smoother than CF with some Pepsi challenge type user testing, but not so much hard evidence as presented here as well as other sites.

    Finally Ryan, are these new metrics you've done an excellent job of formulating going to make it into future benchmarks? Or are you going to just assume the issue has been fixed going forward? I would love to take AMD's word on it but as we've seen from both vendors in the past, driver regression is commonplace unless constantly revisited by users, reviewers, and the vendors alike.
    Reply
  • Ryan Smith - Friday, August 02, 2013 - link

    "Finally Ryan, are these new metrics you've done an excellent job of formulating going to make it into future benchmarks?"

    They'll be in future articles in a limited form, similar to how we handled the GTX 780 launch. It takes a lot of additional work to put this data together, which isn't always time we have available. Especially if it becomes doing hours of extra work to collect data just to say "yep, still no stuttering."
    Reply

Log in

Don't have an account? Sign up now