The 2018 GPU Benchmark Suite & the Test

Another year marks another update to our GPU benchmark suite. This time, however, is more in line with a maintenance update than it is a complete overhaul. Although we've done some extended compute and deep learning benchmarking in the past year, and even some HDR gaming impressions, our compute and synthetic lineup remains largely the same. But before getting into the details, let's start with the bulk of benchmarking, and the biggest reason for these cards anyhow: games.

Joining the 2018 game list is Far Cry 5, Wolfenstein II, Final Fantasy XV and Middle-earth: Shadow of War. We are also bringing in F1 2018 and Total War: Warhammer II. Returning from last year is Battlefield 1, Ashes of the Singularity: Escalation, and Grand Theft Auto V. All-in-all, these games span multiple genres, differing graphics workloads, and contemporary APIs, with a nod towards modern and relatively intensive games.

AnandTech GPU Bench 2018 Game List
Game Genre Release Date API(s)
Battlefield 1 FPS Oct. 2016 DX11
(DX12)
Far Cry 5 FPS Mar. 2018 DX11
Ashes of the Singularity: Escalation RTS Mar. 2016 DX12
(DX11, Vulkan)
Wolfenstein II: The New Colossus FPS Oct. 2017 Vulkan
Final Fantasy XV: Windows Edition JRPG Mar. 2018 DX11
Grand Theft Auto V Action/Open world Apr. 2015 DX11
Middle-earth: Shadow of War Action/RPG Sep. 2017 DX11
F1 2018 Racing Aug. 2018 DX11
Total War: Warhammer II RTS Sep. 2017 DX11
(DX12)

That said, Ashes as a DX12 trailblazer may not be as hot and fresh as it once was, especially considering that the pace of DX12 and Vulkan adoption in new games has waned. The circumstances are worth an investigation on their own, but the learning curve required in modern low-level API and the subsequent return may not be convincing right now. As a more general remark, most developers and publishers tend not to advertise or document DX12 support as much as they used to, nor is it clearly labelled in game specifications as many times DX11 is the unmentioned default.

Particularly for NVIDIA and GeForce RTX, pushing DXR and raytracing means pushing DX12, of which DXR is a component. The API has a backstop in the form of Xbox consoles and Windows 10, and if multi-GPU is to make a comeback, whether that's via compatible workloads (VR), flexible usage (ray tracing workload topologies), or just the plain old inevitability of Moore's Law. So this is less likely to be the slow end of DX12.

In terms of data collection, measurements were gathered either using built-in benchmark tools or with AMD's open-source Open Capture and Analytics Tool (OCAT), which is itself powered by Intel's PresentMon. 99th percentiles were obtained or calculated in a similar fashion, as OCAT natively obtains 99th percentiles. In general, we prefer 99th percentiles over minimums, as they more accurately represent the gaming experience and filter out any artificial outliers.

We've also swapped out Blenchmark, which seems to have been abandoned in terms of updates, in favor of a BMW render from the Blender Institute Cycles Benchmark, and a more recent one from a Cycles benchmark developer on Blenderartists.org. There were concerns with Blenchmark's small tile size, which is not very applicable to GPUs, and in terms of usability we also ran into some GPU detection errors which were linked to inaccurate Blenchmark Python code.

Otherwise, we are also keeping an eye on a few trends and upcoming developments:

  • MLPerf machine learning benchmark suite
  • Blender Benchmark
  • Futuremark's 3DMark DirectX Raytracing benchmark
  • DXR and Vulkan raytracing extension support in games

Another point is that we do not have a permanent HDR monitor for our testbed, which would be necessary to incorporate HDR game testing in the near future; 5 games in our list actually support HDR. And as we look at technologies that enhance or alter image quality (e.g. HDR, Turing's DLSS), we will want to find a better way of comparing differences. This is particularly tricky with HDR as screenshots are inapplicable and even taking accurate photographs will most likely be viewed on an SDR screen. With DLSS, there is a built-in reference quality based on 64x supersampling, which in deep learning terms is the 'ground truth'; an intuitive solution would be to use a neural network based method of analyzing quality differences, but that is likely beyond our scope.

The following tech demos and test applications were provided via NVIDIA:

  • Star Wars 'Reflections' Demo (includes real time ray tracing and DLSS support)
  • Final Fantasy XV Official Benchmark (includes DLSS support)
  • Asteroids Demo (features mesh shading and variable LOD)
  • Epic Infiltrator Demo (features DLSS)

The Testbed

Because NVIDIA is not productizing any other reference-quality GeForce RTX 2080 Ti and 2080 card besides the Founders Editions, which are non-reference by specifications, we've gone ahead and emulated the true reference specifications with a 90MHz downclock and lowering the TDP by roughly 10W. This is to keep comparisons standardized and apples-to-apples, as we always look at reference-to-reference results.

In a classic case of Murphy's Law, our usual PSU started malfunctioning around the time of the review, but given the time constraints we couldn't do a 1:1 replacement in time. As it is a digital PSU, we were beginning to use it for PCIe power readings to augment system measurements, but for now we will have to stick power draw at the wall. For the time being, we've swapped it out with another high-quality and high-wattage PSU.

CPU: Intel Core i7-7820X @ 4.3GHz
Motherboard: Gigabyte X299 AORUS Gaming 7 (F9g)
Power Supply: Corsair AX860i
EVGA 1000 G3
Hard Disk: OCZ Toshiba RD400 (1TB)
Memory: G.Skill TridentZ DDR4-3200 4 x 8GB (16-18-18-38)
Case: NZXT Phantom 630 Windowed Edition
Monitor: LG 27UD68P-B
Video Cards: AMD Radeon RX Vega 64 (Air Cooled)
NVIDIA GeForce RTX 2080 Ti Founders Edition
NVIDIA GeForce RTX 2080 Founders Edition

NVIDIA GeForce GTX 1080 Ti Founders Edition
NVIDIA GeForce GTX 1080 Founders Edition
NVIDIA GeForce GTX 980 Ti
NVIDIA GeForce GTX 980
Video Drivers: NVIDIA Release 411.51 Press
AMD Radeon Software Adrenalin Edition 18.9.1
OS: Windows 10 Pro (April 2018 Update)
Spectre/Meltdown Mitigations Yes, both
Meet The GeForce RTX 2080 Ti & RTX 2080 Founders Editions Cards Battlefield 1
Comments Locked

337 Comments

View All Comments

  • godrilla - Wednesday, September 19, 2018 - link

    Conclusions*
  • BurntMyBacon - Thursday, September 20, 2018 - link

    I'm going to take a (admittedly small) leap of faith and suggest that nVidia most likely is not intentionally limiting performance of Turing cards. Given the amount of hardware dedicated to tasks that don't benefit rasterization, it just doesn't seem like could have left that much performance on the table. It is much more likely that they've simply got prices set high with the intent of dropping them once high end pascal inventory clears out. Of course, after the mining push, they've seen how much the market is willing to bear. They may be trying to establish a new pricing structure that gives the extra profits to them rather than retailers.
  • mapesdhs - Wednesday, September 26, 2018 - link

    Give the amount of deceitful PR being used for this launch, I don't think your leap is justified.
  • tamalero - Wednesday, September 19, 2018 - link

    I honestly believe the endgame of Nvidia is simple. They want to increase their margin, and the only way to to that is to sell the WHOLE full chips, tensor and all to gamers. While still charging top notch to Pros.

    This would lead Nvidia to make LESS variants, saving costs in having to design multiple versions when they cant scale down or cut.
  • PopinFRESH007 - Wednesday, September 19, 2018 - link

    your argument is invalidated by the evidence of this product launch. All three cards are on different chips.
  • eva02langley - Thursday, September 20, 2018 - link

    You are kidding me? This is exactly this. They made an all around chip to tackle pros, gamers and compute. Vega has the same issue. It was aimed at being an iGPU to a dGPU. It does extremely well at low input, but as a dGPU.

    They save cost and standardize their manufacturing process. It is nothing else.
  • Bensam123 - Wednesday, September 19, 2018 - link

    Going to go a weird direction with this. I believe cards are going to start diverging from one another in terms of what gamers are looking for. Hardcore gamers that are after the professional scene and absolute performance always turn graphics down, they drive high refresh rate monitor, with low response times, and high frame rates to absolutely limit the amount of variance (spiking) that is present in their gaming experience.

    Nvidia went for absolute fidelity where they believe the best gaming experience will come from picture perfect representations of an environment, such as with ray tracing. I see ray tracing as a gamer and I go 'Welp that's something I'll turn off'. Hardware review websites are only looking at gaming from a cinematic standpoint, where best gaming will always have everything maxed out running at 8k. Cards do perform differently under different resolutions and especially with different amounts of eye candy turned on. I really think Anand should look into testing cards at 1080p on lowest settings with everything uncapped - Not as the only datapoint, but as another one. Professional gamers or anyone who takes gaming seriously will be running that route.

    Which runs into another scenario, where card makers are going to diverge. AMDs upcomming 7nm version of Vega for instance may continue down Vegas current path, which means they'll be focusing on current day performance (although they mentioned concentrating more on compute we can assume the two will intersect). That means while a 2080ti might be faster running 4k@ultra, especially with rays if that ever takes off, it may lose out completely at 1080p@low (but not eyecancer, such as turning down resolution or textures).

    For testing at absolute bleeding speeds, that 1% that is removed in 99% percentile testing really starts to matter. Mainly because the spikes, the micro-stutters, the extra long hiccups get you killed and that pisses off gamers that aim for the pinnacle. Those might seem like outliers, but if they happen frequently-infrequently enough, they are part of a distribution and shouldn't be removed. When aiming for bleeding speeds, they actually matter a lot more.

    So thus is born the esports gaming card and the cinematic gaming card. Please test both.
  • PopinFRESH007 - Wednesday, September 19, 2018 - link

    so they should include the horizontal line of a completely CPU bound test? Also I'm not understanding the statistical suggestions, they make no sense. Using the 99th percentile is very high already and the minuscule amount of outliers being dropped are often not due to the GPU. As long as they are using the same metric for all tests in the data set it is irrelevant.
  • Bensam123 - Thursday, September 20, 2018 - link

    Not all games are CPU bound, furthermore it wouldn't be completely horizontal, that would imply zero outliers, which never happens. In the case of that instead of looking at 99% frame time you would instead focus on the other part 1% frame time or all the stuttering and microstutters. You can have a confidence in tail ends of a distribution if there is enough data points.

    Also having played tons of games on low settings, you are 100% incorrect about it being a flat line. Go play something like Overwatch or Fortnite on low, you don't automagically end up at CPU cap.
  • V900 - Thursday, September 20, 2018 - link

    Despite all the (AMD) fanboy rage about higher prices, here’s what will happen:

    Early adopters and anyone who wants the fastest card on the market, will get an RTX 2080/2070, Nvidia is going to make a tidy profit, and in 6-12 months prices will have dropped and cheaper Turing cards will hit the market.

    That’s when people with a graphics card budget smaller than 600$ will get one. (AMD fanboys will keep raging though, prob about something else that’s also Nvidia related.)

    That’s how it always works out when a new generation of graphics hit the market.

    But everyone who’s salty about “only” getting a 40% faster card for a higher price won’t enjoy the rest of the decade.

    There won’t be anymore GPUs that deliver a 70-80% performance increase. Not from AMD and not from Nvidia.

    We’re hitting the limits of Moore’s law now, so from here on out, a performance increase of 30% or less on a new GPU will be the standard.

Log in

Don't have an account? Sign up now