Our New Testing Suite for 2019 and 2020

Spectre and Meltdown Hardened

In order to keep up to date with our testing, we have to update our software every so often to stay relevant. In our updates we typically implement the latest operating system, the latest patches, the latest software revisions, the newest graphics drivers, as well as add new tests or remove old ones. As regular readers will know, our CPU testing revolves an automated test suite, and depending on how the newest software works, the suite either needs to change, be updated, have tests removed, or be rewritten completely. Last time we did a full re-write, it took the best part of a month, including regression testing (testing older processors).

One of the key elements of our testing update for 2018 (and 2019) is the fact that our scripts and systems are designed to be hardened for Spectre and Meltdown. This means making sure that all of our BIOSes are updated with the latest microcode, and all the steps are in place with our operating system with updates. In this case we are using Windows 10 x64 Enterprise 1709 with April security updates which enforces Smeltdown (our combined name) mitigations. Uses might ask why we are not running Windows 10 x64 RS4, the latest major update – this is due to some new features which are giving uneven results. Rather than spend a few weeks learning to disable them, we’re going ahead with RS3 which has been widely used.

Our previous benchmark suite was split into several segments depending on how the test is usually perceived. Our new test suite follows similar lines, and we run the tests based on:

  • Power
  • Memory
  • Office
  • System
  • Render
  • Encoding
  • Web
  • Legacy
  • Integrated Gaming
  • CPU Gaming

Depending on the focus of the review, the order of these benchmarks might change, or some left out of the main review. All of our data will reside in our benchmark database, Bench, for which there is a new ‘CPU 2019’ section for all of our new tests.

Within each section, we will have the following tests:

Power

Our power tests consist of running a substantial workload for every thread in the system, and then probing the power registers on the chip to find out details such as core power, package power, DRAM power, IO power, and per-core power. This all depends on how much information is given by the manufacturer of the chip: sometimes a lot, sometimes not at all.

We are currently running POV-Ray as our main test for Power, as it seems to hit deep into the system and is very consistent. In order to limit the number of cores for power, we use an affinity mask driven from the command line.

Memory

These tests involve disabling all turbo modes in the system, forcing it to run at base frequency, and them implementing both a memory latency checker (Intel’s Memory Latency Checker works equally well for both platforms) and AIDA64 to probe cache bandwidth.

Office

  • Chromium Compile: Windows VC++ Compile of Chrome 56 (same as 2017)
  • PCMark10: Primary data will be the overview results – subtest results will be in Bench
  • 3DMark Physics: We test every physics sub-test for Bench, and report the major ones (new)
  • GeekBench4: By request (new)
  • SYSmark 2018: Recently released by BAPCo, currently automating it into our suite (new, when feasible)

System

  • Application Load: Time to load GIMP 2.10.4 (new)
  • FCAT: Time to process a 90 second ROTR 1440p recording (same as 2017)
  • 3D Particle Movement: Particle distribution test (same as 2017) – we also have AVX2 and AVX512 versions of this, which may be added later
  • Dolphin 5.0: Console emulation test (same as 2017)
  • DigiCortex: Sea Slug Brain simulation (same as 2017)
  • y-Cruncher v0.7.6: Pi calculation with optimized instruction sets for new CPUs (new)
  • Agisoft Photoscan 1.3.3: 2D image to 3D modelling tool (updated)

Render

  • Corona 1.3: Performance renderer for 3dsMax, Cinema4D (same as 2017)
  • Blender 2.79b: Render of bmw27 on CPU (updated to 2.79b)
  • LuxMark v3.1 C++ and OpenCL: Test of different rendering code paths (same as 2017)
  • POV-Ray 3.7.1: Built-in benchmark (updated)
  • CineBench R15: Older Cinema4D test, will likely remain in Bench (same as 2017)

Encoding

  • 7-zip 1805: Built-in benchmark (updated to v1805)
  • WinRAR 5.60b3: Compression test of directory with video and web files (updated to 5.60b3)
  • AES Encryption: In-memory AES performance. Slightly older test. (same as 2017)
  • Handbrake 1.1.0: Logitech C920 1080p60 input file, transcoded into three formats for streaming/storage:
    • 720p60, x264, 6000 kbps CBR, Fast, High Profile
    • 1080p60, x264, 3500 kbps CBR, Faster, Main Profile
    • 1080p60, HEVC, 3500 kbps VBR, Fast, 2-Pass Main Profile

Web

  • WebXPRT3: The latest WebXPRT test (updated)
  • WebXPRT15: Similar to 3, but slightly older. (same as 2017)
  • Speedometer2: Javascript Framework test (new)
  • Google Octane 2.0: Depreciated but popular web test (same as 2017)
  • Mozilla Kraken 1.1: Depreciated but popular web test (same as 2017)

Legacy (same as 2017)

  • 3DPM v1: Older version of 3DPM, very naïve code
  • x264 HD 3.0: Older transcode benchmark
  • Cinebench R11.5 and R10: Representative of different coding methodologies

Integrated and CPU Gaming

We have recently automated around a dozen games at four different performance levels. A good number of games will have frame time data, however due to automation complications, some will not. The idea is that we get a good overview of a number of different genres and engines for testing. So far we have the following games automated:

AnandTech CPU Gaming 2019 Game List
Game Genre Release Date API IGP Low Med High
World of Tanks enCore Driving / Action Feb
2018
DX11 768p
Minimum
1080p
Medium
1080p
Ultra
4K
Ultra
Final Fantasy XV JRPG Mar
2018
DX11 720p
Standard
1080p
Standard
4K
Standard
8K
Standard
Shadow of War Action / RPG Sep
2017
DX11 720p
Ultra
1080p
Ultra
4K
High
8K
High
F1 2018 Racing Aug
2018
DX11 720p
Low
1080p
Med
4K
High
4K
Ultra
Civilization VI RTS Oct
2016
DX12 1080p
Ultra
4K
Ultra
8K
Ultra
16K
Low
Ashes: Classic RTS Mar
2016
DX12 720p
Standard
1080p
Standard
1440p
Standard
4K
Standard
Strange Brigade* FPS Aug
2018
DX12
Vulkan
720p
Low
1080p
Medium
1440p
High
4K
Ultra
Shadow of the Tomb Raider Action Sep
2018
DX12 720p
Low
1080p
Medium
1440p
High
4K
Highest
Grand Theft Auto V Open World Apr
2015
DX11 720p
Low
1080p
High
1440p
Very High
4K
Ultra
Far Cry 5 FPS Mar
2018
DX11 720p
Low
1080p
Normal
1440p
High
4K
Ultra
*Strange Brigade is run in DX12 and Vulkan modes

For our CPU Gaming tests, we will be running on an NVIDIA GTX 1080. For the CPU benchmarks, we use an RX460 as we now have several units for concurrent testing.

In previous years we tested multiple GPUs on a small number of games – this time around, due to a Twitter poll I did which turned out exactly 50:50, we are doing it the other way around: more games, fewer GPUs.

Scale Up vs Scale Out: Benefits of Automation

One comment we get every now and again is that automation isn’t the best way of testing – there’s a higher barrier to entry, and it limits the tests that can be done. From our perspective, despite taking a little while to program properly (and get it right), automation means we can do several things:

  1. Guarantee consistent breaks between tests for cooldown to occur, rather than variable cooldown times based on ‘if I’m looking at the screen’
  2. It allows us to simultaneously test several systems at once. I currently run five systems in my office (limited by the number of 4K monitors, and space) which means we can process more hardware at the same time
  3. We can leave tests to run overnight, very useful for a deadline
  4. With a good enough script, tests can be added very easily

Our benchmark suite collates all the results and spits out data as the tests are running to a central storage platform, which I can probe mid-run to update data as it comes through. This also acts as a mental check in case any of the data might be abnormal.

We do have one major limitation, and that rests on the side of our gaming tests. We are running multiple tests through one Steam account, some of which (like GTA) are online only. As Steam only lets one system play on an account at once, our gaming script probes Steam’s own APIs to determine if we are ‘online’ or not, and to run offline tests until the account is free to be logged in on that system. Depending on the number of games we test that absolutely require online mode, it can be a bit of a bottleneck.

Benchmark Suite Updates

As always, we do take requests. It helps us understand the workloads that everyone is running and plan accordingly.

A side note on software packages: we have had requests for tests on software such as ANSYS, or other professional grade software. The downside of testing this software is licensing and scale. Most of these companies do not particularly care about us running tests, and state it’s not part of their goals. Others, like Agisoft, are more than willing to help. If you are involved in these software packages, the best way to see us benchmark them is to reach out. We have special versions of software for some of our tests, and if we can get something that works, and relevant to the audience, then we shouldn’t have too much difficulty adding it to the suite.

Test Bed and Setup CPU Performance: System Tests
POST A COMMENT

207 Comments

View All Comments

  • Beaver M. - Wednesday, May 22, 2019 - link

    After so many decades being wrong you guys still claim CPU power doesnt matter much in games.
    Youre wrong. Again. Common bottleneck today in games is the CPU, especially because the GPU advancement has been very slow.
    Reply
  • Spunjji - Wednesday, May 22, 2019 - link

    GPU advancement slowing down *makes the CPU less relevant, not more*. The CPU is only relevant to performance when it can't meet the bare minimum requirements to serve the GPU fast enough. If the GPU is your limit, no amount of CPU power increase will help. Reply
  • LoneWolf15 - Friday, May 17, 2019 - link

    Is it abysmal because of the CPU though, or because of the software?

    Lots of software isn't written to take advantage of more than four cores tops, aside from the heavy hitters, and to an extent, we've hit a celing with clock speeds for awhile, with 5GHz being (not exactly, but a fair representation of) the ceiling.
    AMD has caught up in a big way, and for server apps and rendering, it's an awesome value and a great CPU. Even with that, it still doesn't match up with a 9700K in games, all other things being equal, unless a game is dependent on GPU alone.
    I think most mainstream software isn't optimized beyond a certain point for any of our current great CPUs, largely because until recently, CPU development and growth has stagnated. I'm really hoping real competition drives improved software.
    Note also that it hasn't been like the 90s in some time, where we were doubling CPU performance every 16 months. Some of that is because there's too many limitations to achieving that doubling, both software and hardware.

    I'm finding considerable speed boosts over my i7-4790K that was running at 4.4GHz (going to an i9-9900K running constantly at 4.7GHz on all cores) in regular apps and gaming (at 1900x1200 with two GTX 1070 cards in SLI), and I got a deal on the CPU, so I'm perfectly happy with my first mainboard/CPU upgrade in five years (my first board was a 386DX back in `93).
    Reply
  • peevee - Tuesday, May 14, 2019 - link

    Same here. i7-2600k from may 2011, with the same OCZ Vertex 3.
    8 years, twice the cores, not even twice the performance in real world. Just essentially overclocked to the max from the factory.

    Remember when real life performance more than doubled every 2 years? On the same 1 core, in all apps, not just heavily multithreaded? Good thing AMD at least forced Intel go from 4 to 6 to 8 in 2 years. Now they need to double their memory controllers, it's the same 128 bits since what, Pentium Pro?
    Reply
  • Mr Perfect - Friday, May 10, 2019 - link

    Same here. Over the years I've stuffed it full of RAM and SSD and been pleased with the performance. I'm thinking it's time for it to go though.

    In 2016 I put a 1060 in the machine and was mildly disappointed in the random framerate drops in games (at 1200p). Assuming it was the GPU's fault, I upgraded further in 2018 to a 1070 Ti some bitcoin miner was selling for cheap when the market crashed. The average framerates went up, but all of the lows are just as low as they ever where. So either Fallout 4 runs like absolute garbage in certain areas, or the CPU was choking up both GPUs.

    When something that isn't PCIe 3 comes out I suppose I can try again and see.
    Reply
  • ImOnMy116 - Friday, May 10, 2019 - link

    For whatever it's worth, in my experience Fallout 4 (and Skyrim/Skyrim SE/maybe all Bethesda titles) are poorly optimized. It seems their engine is highly dependent on IPC, but even in spite of running an overclocked 6700K/1080 Ti, I get frame drops in certain parts of the map. I think it's likely at least partially dependent on where your character is facing at any given point in time. There can be long draw distances or lots of NPCs near by taxing the CPU (i.e. Diamond City). Reply
  • Mr Perfect - Friday, May 10, 2019 - link

    Yeah, that makes sense. F4's drops are definitely depended on location and where the character is facing for me too.

    The country side, building interiors and winding city streets you can't see very far down are just fine. Even Diamond City is okay. It's when I stand at an intersection of one of the roads that runs arrow straight through Boston or get up on rooftops with a view over the city that rates die. If the engine wants pure CPU grunt for that, then the 2600 just isn't up to it.

    Strangely, Skyrim SE has been fine. The world is pretty sparse compared to F4 though.
    Reply
  • Vayra - Monday, May 13, 2019 - link

    Fallout 4 is simply a game of asset overload. That happens especially in the urban areas. It shows us that the engine is past expiry date and unable to keep up to the game's demands of this time. The game needs all those assets to at least look somewhat bearable. And its not efficient about it at all; a big part of all those little items also need to be fully interactive objects.

    So its not 'strange' at all, really. More objects = more cpu load and none of them can be 'cooked' beforehand. They are literally placed in the world as you move around in it.
    Reply
  • Vayra - Monday, May 13, 2019 - link

    This is also part of the reason why the engine has trouble with anything over 60 fps, and why you can sometimes see objects falling from the sky as you zone in. Reply
  • amrs - Saturday, May 11, 2019 - link

    But what speed of RAM did you stuff it with? Fallout 4 has been shown to benefit from RAM faster than DDR3-1600. Reply

Log in

Don't have an account? Sign up now