The 2018 GPU Benchmark Suite & the Test

Another year marks another update to our GPU benchmark suite. This time, however, is more in line with a maintenance update than it is a complete overhaul. Although we've done some extended compute and deep learning benchmarking in the past year, and even some HDR gaming impressions, our compute and synthetic lineup remains largely the same. But before getting into the details, let's start with the bulk of benchmarking, and the biggest reason for these cards anyhow: games.

Joining the 2018 game list is Far Cry 5, Wolfenstein II, Final Fantasy XV and Middle-earth: Shadow of War. We are also bringing in F1 2018 and Total War: Warhammer II. Returning from last year is Battlefield 1, Ashes of the Singularity: Escalation, and Grand Theft Auto V. All-in-all, these games span multiple genres, differing graphics workloads, and contemporary APIs, with a nod towards modern and relatively intensive games.

AnandTech GPU Bench 2018 Game List
Game Genre Release Date API(s)
Battlefield 1 FPS Oct. 2016 DX11
(DX12)
Far Cry 5 FPS Mar. 2018 DX11
Ashes of the Singularity: Escalation RTS Mar. 2016 DX12
(DX11, Vulkan)
Wolfenstein II: The New Colossus FPS Oct. 2017 Vulkan
Final Fantasy XV: Windows Edition JRPG Mar. 2018 DX11
Grand Theft Auto V Action/Open world Apr. 2015 DX11
Middle-earth: Shadow of War Action/RPG Sep. 2017 DX11
F1 2018 Racing Aug. 2018 DX11
Total War: Warhammer II RTS Sep. 2017 DX11
(DX12)

That said, Ashes as a DX12 trailblazer may not be as hot and fresh as it once was, especially considering that the pace of DX12 and Vulkan adoption in new games has waned. The circumstances are worth an investigation on their own, but the learning curve required in modern low-level API and the subsequent return may not be convincing right now. As a more general remark, most developers and publishers tend not to advertise or document DX12 support as much as they used to, nor is it clearly labelled in game specifications as many times DX11 is the unmentioned default.

Particularly for NVIDIA and GeForce RTX, pushing DXR and raytracing means pushing DX12, of which DXR is a component. The API has a backstop in the form of Xbox consoles and Windows 10, and if multi-GPU is to make a comeback, whether that's via compatible workloads (VR), flexible usage (ray tracing workload topologies), or just the plain old inevitability of Moore's Law. So this is less likely to be the slow end of DX12.

In terms of data collection, measurements were gathered either using built-in benchmark tools or with AMD's open-source Open Capture and Analytics Tool (OCAT), which is itself powered by Intel's PresentMon. 99th percentiles were obtained or calculated in a similar fashion, as OCAT natively obtains 99th percentiles. In general, we prefer 99th percentiles over minimums, as they more accurately represent the gaming experience and filter out any artificial outliers.

We've also swapped out Blenchmark, which seems to have been abandoned in terms of updates, in favor of a BMW render from the Blender Institute Cycles Benchmark, and a more recent one from a Cycles benchmark developer on Blenderartists.org. There were concerns with Blenchmark's small tile size, which is not very applicable to GPUs, and in terms of usability we also ran into some GPU detection errors which were linked to inaccurate Blenchmark Python code.

Otherwise, we are also keeping an eye on a few trends and upcoming developments:

  • MLPerf machine learning benchmark suite
  • Blender Benchmark
  • Futuremark's 3DMark DirectX Raytracing benchmark
  • DXR and Vulkan raytracing extension support in games

Another point is that we do not have a permanent HDR monitor for our testbed, which would be necessary to incorporate HDR game testing in the near future; 5 games in our list actually support HDR. And as we look at technologies that enhance or alter image quality (e.g. HDR, Turing's DLSS), we will want to find a better way of comparing differences. This is particularly tricky with HDR as screenshots are inapplicable and even taking accurate photographs will most likely be viewed on an SDR screen. With DLSS, there is a built-in reference quality based on 64x supersampling, which in deep learning terms is the 'ground truth'; an intuitive solution would be to use a neural network based method of analyzing quality differences, but that is likely beyond our scope.

The following tech demos and test applications were provided via NVIDIA:

  • Star Wars 'Reflections' Demo (includes real time ray tracing and DLSS support)
  • Final Fantasy XV Official Benchmark (includes DLSS support)
  • Asteroids Demo (features mesh shading and variable LOD)
  • Epic Infiltrator Demo (features DLSS)

The Testbed

Because NVIDIA is not productizing any other reference-quality GeForce RTX 2080 Ti and 2080 card besides the Founders Editions, which are non-reference by specifications, we've gone ahead and emulated the true reference specifications with a 90MHz downclock and lowering the TDP by roughly 10W. This is to keep comparisons standardized and apples-to-apples, as we always look at reference-to-reference results.

In a classic case of Murphy's Law, our usual PSU started malfunctioning around the time of the review, but given the time constraints we couldn't do a 1:1 replacement in time. As it is a digital PSU, we were beginning to use it for PCIe power readings to augment system measurements, but for now we will have to stick power draw at the wall. For the time being, we've swapped it out with another high-quality and high-wattage PSU.

CPU: Intel Core i7-7820X @ 4.3GHz
Motherboard: Gigabyte X299 AORUS Gaming 7 (F9g)
Power Supply: Corsair AX860i
EVGA 1000 G3
Hard Disk: OCZ Toshiba RD400 (1TB)
Memory: G.Skill TridentZ DDR4-3200 4 x 8GB (16-18-18-38)
Case: NZXT Phantom 630 Windowed Edition
Monitor: LG 27UD68P-B
Video Cards: AMD Radeon RX Vega 64 (Air Cooled)
NVIDIA GeForce RTX 2080 Ti Founders Edition
NVIDIA GeForce RTX 2080 Founders Edition

NVIDIA GeForce GTX 1080 Ti Founders Edition
NVIDIA GeForce GTX 1080 Founders Edition
NVIDIA GeForce GTX 980 Ti
NVIDIA GeForce GTX 980
Video Drivers: NVIDIA Release 411.51 Press
AMD Radeon Software Adrenalin Edition 18.9.1
OS: Windows 10 Pro (April 2018 Update)
Spectre/Meltdown Mitigations Yes, both
Meet The GeForce RTX 2080 Ti & RTX 2080 Founders Editions Cards Battlefield 1
Comments Locked

337 Comments

View All Comments

  • darkos - Friday, September 21, 2018 - link

    Please add flight simulation testing to your list of applications. eg: X-Plane, Prepar3d.
  • Vinny DePaul - Friday, September 21, 2018 - link

    I am still rocking 980 with fps over 60 and everything turned up to max. I guess I will wait.
  • mapesdhs - Thursday, September 27, 2018 - link

    That's a perfect summary of why tom's looney article was so bad. If your current hw is doing just fine for the games you're playing atm, then upgrading makes no sense. It's rather cynical of NVIDIA, and some tech sites, to basically create a need and then push people into thinking they're idiots if they don't upgrade, while hiding behind very poor price/performance dynamics.
  • DARK_BG - Friday, September 21, 2018 - link

    As someone working in the game industry for already 8+ years I can tell you only one thing.No body will rush to implement proprietary features!
    The ones that have demos or are about soon to have the features implemented are the ones Nvidia reached to not vise versa.
    This days making a game is no different than making any other product on this planet.It is corporate bussines which means you want maximum profit which translates in maximum user coverage which translates in maximum platforfm coverage PC (Windows , Mac , Linux), Consoles and mobile.
    There is just no basis to anyhow compare Vulkan to anything proprietary.Vulkan is coming with the promise that what i make will look and feel the same way visually across multple platforms without requiring too much husstle on the development side.Even when you use a flexible engine as UE4 it is not that easy to have the same stuff working across multiple platforms and changes and further development for materials and meshes are required to have the stuff atleast to look indetical.
    So i can hardly imagine that while you are bogged down with tons of bugs and trying to deliver your product across multiple platforms you will add yourself one more pain in the ass as nVidia Ray Tracing that will have doubfull income effect on your title given the small user reach.
    I can give you Wargaming and Valve games as an example of old engines that are making tons of money.
    So while nVidia is trying to ripoff people with that amount of money I'm wondering how to optimise one level so it could run fast and look cool on 6 years old midrange hardware.
  • eddman - Friday, September 21, 2018 - link

    *off-topic*

    Although it is true that proprietary features do not always take off in a meaningful way (the GPU-accelerated mode of physx as an example), it doesn't mean an open standard would always be the popular choice.

    Take big budget games from large publishers. These games, in the large majority of cases, are only available on three platforms, PS, xbox and windows, because these are the platforms that have the hardware to support such games and also have the largest audience.

    IINM, vulkan is not available on X1 and PS4. If a AAA game dev was to use vulkan on windows, they'd still need to code the game for directx on X1 and GNM or GNMX on PS4, meaning they'd have to support three APIs.

    If they go with directx on windows, then two platforms will already be covered and they'd only need to do additional coding for PS4 support.

    On the other hand, vulkan does make sense for devs of smaller games where they want to cover as many platforms as possible, specially for mobile games, where vulkan covers windows, linux, mac, android and I think ios and even switch.
  • eva02langley - Friday, September 21, 2018 - link

    Yes and no, for example Freesync is finally getting support from TV makers. Finally, I can get a 4k big screeen with HDR and... freesync.

    Open source is the way to go over proprietary technology.
  • eddman - Friday, September 21, 2018 - link

    That's why I wrote it's not always the case. Sometimes it works, sometimes not. It all has to do with the standard having industry support. Being open source does not automatically mean it'd catch on, unfortunately.
  • noone2 - Friday, September 21, 2018 - link

    Someone will and it will be great and then it will catch on. Or maybe it won't, but that's how things happen. Someone takes a risk and it pans out. Either contribute to it or don't, play safe or take a chance.

    Nvidia has obviously been making some pretty good decisions over the years and has turned out some amazing products. Sometimes they've been wrong, but more often than not they are right. If they were wrong more than not, they'd be out of business, not a $150B company.

    If you don't ever take a risk doing something new or cutting edge, you'll disappear. This is true for all technology.
  • noone2 - Friday, September 21, 2018 - link

    Oh, and remember, at some point you don't have to care about 6 year old hardware. Look at consoles. At some point the studio just stops making a game for the last-gen, even though the new gen doesn't have the same size install base yet. Or a non-franchise game just shows up for next-gen and that's it. They never even bother to attempt to make it on older stuff.
  • eva02langley - Friday, September 21, 2018 - link

    Thanks for your comment, this was my argument all along.

    The only way to force a new feature is by shear numbers. Basically, if RTX was something available on new consoles, then that would make a business stand point sense, however AMD is owning consoles and might for a long term.

    AMD should force multi-GPU via Infinity Fabric through consoles. This would work because devs would have access to additional power on the die via proper coding... and this delivered to 100% of the user base.

    If this is only developed for less than 1% of the PC user base, this will fail miserably and nobody would add support unless sponsored by Nvidia themselves.

    Financial analysts are seeing it and downgrades are coming.

Log in

Don't have an account? Sign up now