2017 CPU Benchmarking

For our review, we are implementing our fresh CPU testing benchmark suite, using new scripts developed specifically for this testing. This means that with a fresh OS install, we can configure the OS to be more consistent, install the new benchmarks, maintain version consistency without random updates and start running the tests in under 5 minutes. After that it's a one button press to start an 8-10hr test (with a high-performance core) with nearly 100 relevant data points in the benchmarks given below. The tests cover a wide range of segments, some of which will be familiar but some of the tests are new to benchmarking in general, but still highly relevant for the markets they come from.

Our new CPU tests go through six main areas. We cover the Web (we've got an un-updateable version of Chrome 56), general system tests (opening tricky PDFs, emulation, brain simulation, AI, 2D image to 3D model conversion), rendering (ray tracing, modeling), encoding (compression, AES, h264 and HEVC), office based tests (PCMark and others), and our legacy tests, throwbacks from another generation of bad code but interesting to compare.

Our graphs typically list CPUs with microarchitecture, SKU name, cost and power. The cost will be one of two numbers, either the 1k unit price 'tray price' for when a business customer purchases 1000 CPUs, or the MSRP likely to be found at retail. The problem here is that neither Intel nor AMD are consistent: Intel has a tray price for every CPU, but an MSRP only for parts sold at retail. AMD typically quotes MSRP for CPUs at retail, tray prices for enterprise CPUs, and doesn't say much about OEM only parts. We try to find a balance here, so prices may be $10-$20 from what you might expect.

A side note on OS preparation. As we're using Windows 10, there's a large opportunity for something to come in and disrupt our testing. So our default strategy is multiple: disable the ability to update as much as possible, disable Windows Defender, uninstall OneDrive, disable Cortana as much as possible, implement the high performance mode in the power options, and disable the internal platform clock which can drift away from being accurate if the base frequency drifts (and thus the timing ends up inaccurate).

Web Tests on Chrome 56

Sunspider 1.0.2
Mozilla Kraken 1.1
Google Octane 2.0

System Tests

PDF Opening
3DPM v2.1
Dolphin v5.0
DigiCortex v1.20
Agisoft PhotoScan v1.0

Rendering Tests

Corona 1.3
Blender 2.78
LuxMark CPU C++
LuxMark CPU OpenCL
POV-Ray 3.7.1b4
Cinebench R15 ST
Cinebench R15 MT

Encoding Tests

7-Zip 9.2
WinRAR 5.40
AES Encoding (TrueCrypt 7.2)
HandBrake v1.0.2 x264 LQ
HandBrake v1.0.2 x264-HQ
HandBrake v1.0.2 HEVC-4K

Office / Professional

Chromium Compile (v56)
SYSmark 2014 SE

Legacy Tests

3DPM v1 ST / MT
x264 HD 3 Pass 1, Pass 2
Cinebench R11.5 ST / MT
Cinebench R10 ST / MT

A side note - a couple of benchmarks (LuxMark) weren't fully 100% giving good data during testing. Need to go back and re-work this part of our testing.

2017 CPU Gaming Tests

For our new set of GPU tests, we wanted to think big. There are a lot of users in the ecosystem that prioritize gaming above all else, especially when it comes to choosing the correct CPU. If there's a chance to save $50 and get a better graphics card for no loss in performance, then this is the route that gamers would prefer to tread. The angle here though is tough - lots of games have different requirements and cause different stresses on a system, with various graphics cards having different reactions to the code flow of a game. Then users also have different resolutions and different perceptions of what feels 'normal'. This all amounts to more degrees of freedom than we could hope to test in a lifetime, only for the data to become irrelevant in a few months when a new game or new GPU comes into the mix. Just for good measure, let us add in DirectX 12 titles that make it easier to use more CPU cores in a game to enhance fidelity.

Our original list of nine games planned in February quickly became six, due to the lack of professional-grade controls on Ubisoft titles. If you want to see For Honor, Steep or Ghost Recon: Wildlands benchmarked on AnandTech, point Ubisoft Annecy or Ubisoft Montreal in my direction. While these games have in-game benchmarks worth using, unfortunately they do not provide enough frame-by-frame detail to the end user, despite using it internally to produce the data the user eventually sees (and it typically ends up obfuscated by another layer as well). I would instead perhaps choose to automate these benchmarks via inputs, however the extremely variable loading time is a strong barrier to this.

So we have the following benchmarks as part of our 4/2 script, automated to the point of a one-button run and out pops the results four hours later, per GPU. Also listed are the resolutions and settings used.

  • Civilization 6 (1080p Ultra, 4K Ultra)
  • Ashes of the Singularity: Escalation* (1080p Extreme, 4K Extreme)
  • Shadow of Mordor (1080p Ultra, 4K Ultra)
  • Rise of the Tomb Raider #1 - GeoValley (1080p High, 4K Medium)
  • Rise of the Tomb Raider #2 - Prophets (1080p High, 4K Medium)
  • Rise of the Tomb Raider #3 - Mountain (1080p High, 4K Medium)
  • Rocket League (1080p Ultra, 4K Ultra)
  • Grand Theft Auto V (1080p Very High, 4K High)

For each of the GPUs in our testing, these games (at each resolution/setting combination) are run four times each, with outliers discarded. Average frame rates, 99th percentiles and 'Time Under x FPS' data is sorted, and the raw data is archived.

The four GPUs we've managed to obtain for these tests are:

  • MSI GTX 1080 Gaming X 8G
  • ASUS GTX 1060 Strix 6G
  • Sapphire Nitro R9 Fury 4GB
  • Sapphire Nitro RX 480 8GB

In our testing script, we save a couple of special things for the GTX 1080 here. The following tests are also added:

  • Civilization 6 (8K Ultra, 16K Lowest)

This benchmark, with a little coercion, are able to be run beyond the specifications of the monitor being used, allowing for 'future' testing of GPUs at 8K and 16K with some amusing results. We are only running these tests on the GTX 1080, because there's no point watching a slideshow more than once.

Test Bed and Setup Benchmarking Performance: CPU System Tests


View All Comments

  • Santoval - Tuesday, July 25, 2017 - link

    That is not how IPC works, since it explicitly refers to single core - single thread performance. As the number of cores rises the performance of a *single* task never scales linearly because there is always some single thread code involved (Amdahl's law). For example if your task has 90% parallel and 10% serial code its performance will max out at x10 that of a single core at ~512 cores. From then on even if you had a CPU with infinite cores you couldn't extract half an ounce of additional performance. If your code was 95% parallel the performance of your task would plateau at x20. For that though you would need ~2048 cores. And so on.

    Of course Amdahl's law does not provide a complete picture. It assumes, for example, that your task and its code will remain fixed no matter how many cores you add on them. And it disregards the possibility of computing distinct tasks in parallel on separate cores. That's where Gustafson's Law comes in. This "law" is not concerned with speeding up the performance of tasks but computing larger and more complex tasks at the same amount of time.

    An example given in Wikipedia involves boot times : Amdahl's law states that you can speed up the boot process, assuming it can be made largely parallel, up to a certain number of cores. Beyond that -when you become limited by the serial code of your bootloader- adding more cores does not help. Gustafson's law, on the contrary, states that instead of speeding up the boot process by adding more cores and computing resources, you could add colorful GUIs, increase the resolution etc, while keeping the boot time largely the same. This idea could be applied to many -but not all- computing tasks, for example ray tracing (for more photorealistic renderings) and video encoding (for smaller files or videos with better quality), and many other heavily multi-threaded tasks.
  • Rickyxds - Monday, July 24, 2017 - link

    I just agree XD. Reply
  • Diji1 - Wednesday, July 26, 2017 - link

    "Overall speed increase 240%."

    LMAO. Ridiculous.
  • Alistair - Wednesday, July 26, 2017 - link

    No reason to laugh. I compared the 6600k vs the Ryzen 1700. 1 year speed increase of 144 percent (2.44 times the speed). Same as this: 1135 vs 466 points.

  • Dr. Swag - Tuesday, July 25, 2017 - link

    I disagree, best value is 1600 as it oces as well as 1600x, comes with a decent stock cooler, and is cheaper. Reply
  • vext - Monday, July 24, 2017 - link

    Interesting article but it seems intended to play down the extremely bad press x299 has received which is all over the internet and Youtube.

    Once you get past Mr. Cuttress' glowing review, it's clear that the I5-7640x is not worth the money because of lackluster performance, the I7-7740X is marginally faster than the older 7700k, and the I7-7800x is regularly beaten by the 7740X in many benchmarks that actually count and is a monstrously inefficient energy pig. Therefore the only Intel CPUs of this batch worth buying are the 7700k/7740x, and there is no real advantage to x299. In summary, it doesn't actually change anything.

    It's very telling that Mr. Cutress doesn't comment on the absolutely egregious energy consumption of the 7800x. The Test Bed setup section doesn't list the 7800x at all. The 7840x and 7740x are using a Thermalright True Copper (great choice!) but no info on the 7800x cooler. Essentially, the 7800x cameo appearance is only to challenge the extremely strong Ryzen multi-threaded results, but its negative aspects are not discussed, perhaps because it might frighten people from x299. Tsk, tsk. As my 11 year old daughter would say "No Fair." By the way, the 7800x is selling for ~ $1060 right now on Newegg, not $389.

    Proudly typed on my Ryzen 1800x/Gigabyte AB350 Gaming 3. # ;-)
  • Ian Cutress - Monday, July 24, 2017 - link

    You may not have realised but this is the Kaby Lake-X review, so it focuses on the KBL-X parts. We already have a Skylake-X review for you to mull over. There are links on the first page. Reply
  • mapesdhs - Monday, July 24, 2017 - link

    Nevertheless, the wider picture is relevant here. The X299 platform is a mess. Intel is aiming KL-X at a market which doesn't exist, they've locked out features that actually make it useful, it's more power hungry, and a consumer needs a lot of patience and plenty of coffee to work out what the heck works and what doesn't on a mbd with a KL-X fitted.

    This is *exactly* the sort of criticism of Intel which should have been much stronger in the tech journalism space when Intel started pulling these sorts of stunts back with the core-crippled 3930K, heat-crazy IB and PCIe-crippled 5820K. Instead, except for a few exceptions, the tech world has been way too forgiving of Intel's treading-on-water attitude ever since SB, and now they've panicked in response to Ryzen and released a total hodgebodge of a chipset and CPU lineup which makes no sense at all. And if you get any disagreement about what I've said by anyone at Intel, just wave a 4820K in their face and say well explain this then (quad-core chip with 40 PCIe lanes, da daa!).

    I've been a big fan of Z68 and X79, but nothing about Intel's current lineup appeals in the slightest.
  • serendip - Tuesday, July 25, 2017 - link

    There's also the funny bit about motherboards potentially killing KBL-X CPUs if a Skylake-X was used previously.

    What's with Intel's insane product segmentation strategy with all the crippling and inconsistent motherboard choices? It's like they want to make it hard to choose, so buyers either get the cheapest or most expensive chip.
  • Haawser - Tuesday, July 25, 2017 - link

    'EmergencyLake-X' is just generally embarrassing. Intel should just find a nearby landfill site and quietly bury it. Reply

Log in

Don't have an account? Sign up now