Our First FCAT & The Test

First announced back at the end of March, FCAT has been something of a bewildering experience for us. NVIDIA has actually done a great job on the software, but between picky games, flaky DVI cables, and dead SSDs (we killed an Intel enterprise grade SSD 910 with FCAT) things have not gone quite to plan, pushing back our intended use of FCAT more than once. In any case, with most of the kinks worked out we’re ready to start integrating it into our major GPU reviews.

For the time being we’re putting FCAT on beta status, as we intend to try out a few different methods of presenting data to find something that’s meaningful, useful, and legible. To that end we’d love to get your feedback in our comments section so that we can further iterate on our presentation and data collection.

We’ve decided to go with two metrics for our first run with FCAT. The first metric is rather simple: 95th percentile frametimes. For years we’ve done minimum framerates (when practical), which are similar in concept, so this allows us to collect similar stats at the end of the rendering pipeline while hopefully avoiding some of the quirkiness that comes from looking at minimum framerates within games themselves. The 95th percentile frametime is quite simply the amount of time it takes to render the slowest 5% of frames. If a game or video card is introducing significant one-off stuttering by taking too long to render some frames, this will show us.

This is primarily meant to capture single-GPU issues, but in practice with AMD having fixed the bulk of their single-GPU issues months ago, we don’t actually expect much. None the less it’s a good way of showing that nothing interesting is happening in those situations.

Our second metric is primarily focused on multi-GPU setups, and is an attempt to quantize the wild frametime variations seen at times with multi-GPU setups, which show up as telltale zigzag lines in frametime graphs.

In this metric, which for the moment we’re calling Delta Percentages, we’re collecting the deltas (differences) between frametimes, averaging that out, and then running the delta average against the average frametime of the entire run. The end result of this process is that we can measure whether sequential frames are rendering in roughly the same amount of time, while controlling for performance differences by looking at the data relative to the average frametime (rather than as absolute time).

In general, a properly behaving single-GPU card should have a delta average of under 3%, with the specific value depending in part on how variable the workload is throughout any given game benchmark. 3% may sound small, but since we’re talking about an average it means it’s weighed against the entire run. The higher the percentage the more unevenly frames are arriving, and exceeding 3% is about where we expect players with good eyes to start noticing a difference. Alternatively in a perfectly frame metered situation, such as v-sync enabled with a setup that can always hit 60fps, then this would be a flat 0%, representing the pinnacle of smoothness.

Moving on, we’ll be running FCAT against 6 of our 10 games for the time being: Sleeping Dogs, Hitman: Absolution, Total War: Shogun 2, Battlefield 3, Bioshock, and Crysis 3. The rest of our games are either highly inconsistent or generally fussy, introducing too much variance into our FCAT results.

Finally, due to the amount of additional time it takes to put together FCAT results, we’re going to primarily publish FCAT results with major product launches and major driver updates. Due to how frame metering works, the only time frame consistency significantly changes is either with the introduction of new architectures/GPUs, or with the introduction of significant driver changes, so those are the scenarios we’ll be focusing on.

The Test

NVIDIA’s launch drivers for the GTX 780 are 320.18, drivers that are essentially identical to the public 320.14 drivers released last week.

CPU: Intel Core i7-3960X @ 4.3GHz
Motherboard: EVGA X79 SLI
Power Supply: Antec True Power Quattro 1200
Hard Disk: Samsung 470 (256GB)
Memory: G.Skill Ripjaws DDR3-1867 4 x 4GB (8-10-9-26)
Case: Thermaltake Spedo Advance
Monitor: Samsung 305T
Video Cards: AMD Radeon HD 7970 GHz Edition
AMD Radeon HD 7990
NVIDIA GeForce GTX 580
NVIDIA GeForce GTX 680
NVIDIA GeForce GTX 690
NVIDIA GeForce GTX 780
NVIDIA GeForce GTX Titan
Video Drivers: NVIDIA ForceWare 320.14
NVIDIA ForceWare 320.18
AMD Catalyst 13.5 Beta 2
OS: Windows 8 Pro

 

Software, Cont: ShadowPlay and "Reason Flags" DiRT: Showdown
Comments Locked

155 Comments

View All Comments

  • SymphonyX7 - Thursday, May 23, 2013 - link

    *mildly/narrowly trailing the GTX 680
  • chizow - Thursday, May 23, 2013 - link

    AMD released some significant driver updates in ~Oct 2012, branded "Never Settle" drivers that did boost GCN performance significantly, ~10-20% in some cases where they were clearly deficient relative to Nvidia parts. It was enough to make up the difference in a lot of cases or extend the lead to where the GE is generally faster than the 680.

    On the flipside, some of AMD's performance claims, particularly with CF have come under fire due to concerns about microstutter and frame latency, ie. the ongoing runtframe saga.
  • Vayra - Thursday, May 23, 2013 - link

    Drivers possibly?
  • kallogan - Thursday, May 23, 2013 - link

    High end overpriced gpu again ! Next !
  • wumpus - Thursday, May 23, 2013 - link

    Except that the 780 is nothing more than a Titan with even more cuda performance disabled. Presumably, they are expecting to get Titan sales to people interested in GPU computing, if only for geeklust/boasting.
  • wumpus - Thursday, May 23, 2013 - link

    My above comment was supposed to be a reply. Ignore/delete if possible.
  • ifrit39 - Thursday, May 23, 2013 - link

    Shadow Play is the most interesting news here. It costs a not-insignificant amount of money to buy a decent capture card that will record HD video. This is a great alternative as it requires no extra hardware and has little CPU/GPU overhead. Anything that ends up on the net will be compressed by youtube or other service anyway. I can't wait to remove fraps and install shadow play.
  • ahamling27 - Saturday, May 25, 2013 - link

    Fraps isn't the best, but they somehow have the market cornered. Look up Bandicam, I use it exclusively and I get great captures at a fraction the size. Plus they aren't cut up into 4 gig files. It has at least 15x more customization like putting watermarks in your capture or if you do like to segment your files you can have it do that at any size or time length you want. Plus you can record two sound sources at once, like your game and mic, or your game and whatever voice chat software you use.

    Anyway, I probably sound like I work for them now, but I can assure you I don't. This Shadow Play feature is definitely piquing my interest. If it's implemented wisely, it might just shut all the other software solutions down.
  • garadante - Thursday, May 23, 2013 - link

    There were two things that instantly made me dislike this card, much as I've liked Nvidia in the past: completely disabling the compute performance down to 600 series levels which was the reason I was more forgiving towards AMD in the 600/7000 series generation, and that they've priced this card at $650. If I remember correctly, the 680 was priced at $500-550 at launch, and that itself was hard to stomach as it was and still is widely believed GK104 was meant to be their mid-range chip. This 780 is more like what I imagined the 680 having been and if it launched at that price point, I'd be more forgiving.

    As it is... I'm very much rooting for AMD. I hope with these new hires, of which Anandtech even has an article of their new dream team or some such, that AMD can become competitive. Hopefully the experience developers get with their kind-of-funky architecture with the new consoles, however underwhelming they are, brings software on the PC both better multithreaded programming and performance, and better programming and performance to take advantage of AMD's module scheme. Intel and Nvidia both need some competition so we can get this computer hardware industry a bit less stagnated and better for the consumer.
  • EJS1980 - Tuesday, May 28, 2013 - link

    The 680 was $500 at launch, and was the main reason why AMD received so much flak for their 7970 pricing. At the time it launched, the 680 blew the 7970 away in terms of gaming performance, which was thee reason AMD had to respond with across the board price drops on the 7950/70, even though it took them a few months.

Log in

Don't have an account? Sign up now