The 2017 Benchmark Suite

For our review, we are implementing our fresh CPU testing benchmark suite, using new scripts developed specifically for this testing. This means that with a fresh OS install, we can configure the OS to be more consistent, install the new benchmarks, maintain version consistency without random updates and start running the tests in under 5 minutes. After that it's a one button press to start an 8-10hr test (with a high-performance core) with nearly 100 relevant data points in the benchmarks given below for CPUs, followed by our CPU gaming tests which run for 4-5 hours for each of the GPUs used. The CPU tests cover a wide range of segments, some of which will be familiar but some of the tests are new to benchmarking in general, but still highly relevant for the markets they come from.

Our new CPU tests go through six main areas. We cover the Web (we've got an un-updateable version of Chrome 56), general system tests (opening tricky PDFs, emulation, brain simulation, AI, 2D image to 3D model conversion), rendering (ray tracing, modeling), encoding (compression, AES, h264 and HEVC), office based tests (PCMark and others), and our legacy tests, throwbacks from another generation of bad code but interesting to compare.

All of our benchmark results can also be found in our benchmark engine, Bench.

A side note on OS preparation. As we're using Windows 10, there's a large opportunity for something to come in and disrupt our testing. So our default strategy is multiple: disable the ability to update as much as possible, disable Windows Defender, uninstall OneDrive, disable Cortana as much as possible, implement the high performance mode in the power options, and disable the internal platform clock which can drift away from being accurate if the base frequency drifts (and thus the timing ends up inaccurate).

Web Tests on Chrome 56

Sunspider 1.0.2
Mozilla Kraken 1.1
Google Octane 2.0
WebXPRT15

System Tests

PDF Opening
FCAT
3DPM v2.1
Dolphin v5.0
DigiCortex v1.20
Agisoft PhotoScan v1.0

Rendering Tests

Corona 1.3
Blender 2.78
LuxMark v3.1 CPU C++
LuxMark v3.1 CPU OpenCL
POV-Ray 3.7.1b4
Cinebench R15 ST
Cinebench R15 MT

Encoding Tests

7-Zip 9.2
WinRAR 5.40
AES Encoding (TrueCrypt 7.2)
HandBrake v1.0.2 x264 LQ
HandBrake v1.0.2 x264-HQ
HandBrake v1.0.2 HEVC-4K

Office / Professional

PCMark8
Chromium Compile (v56)
SYSmark 2014 SE

Legacy Tests

3DPM v1 ST / MT
x264 HD 3 Pass 1, Pass 2
Cinebench R11.5 ST / MT
Cinebench R10 ST / MT

CPU Gaming Tests

For our new set of GPU tests, we wanted to think big. There are a lot of users in the ecosystem that prioritize gaming above all else, especially when it comes to choosing the correct CPU. If there's a chance to save $50 and get a better graphics card for no loss in performance, then this is the route that gamers would prefer to tread. The angle here though is tough - lots of games have different requirements and cause different stresses on a system, with various graphics cards having different reactions to the code flow of a game. Then users also have different resolutions and different perceptions of what feels 'normal'. This all amounts to more degrees of freedom than we could hope to test in a lifetime, only for the data to become irrelevant in a few months when a new game or new GPU comes into the mix. Just for good measure, let us add in DirectX 12 titles that make it easier to use more CPU cores in a game to enhance fidelity.

Our original list of nine games planned in February quickly became six, due to the lack of professional-grade controls on Ubisoft titles. If you want to see For Honor, Steep or Ghost Recon: Wildlands benchmarked on AnandTech, please point Ubisoft Annecy or Ubisoft Montreal in my direction. While these games have in-game benchmarks worth using, unfortunately they do not provide enough frame-by-frame detail to the end user, despite using it internally to produce the data the user eventually sees (and it typically ends up obfuscated by another layer as well). I would instead perhaps choose to automate these benchmarks via inputs, however the extremely variable loading time is a strong barrier to this.

So we have the following benchmarks as part of our 4/2 script, automated to the point of a one-button run and out pops the results four hours later, per GPU. Also listed are the resolutions and settings used.

  • Civilization 6 (1080p Ultra, 4K Ultra)
  • Ashes of the Singularity: Escalation* (1080p Extreme, 4K Extreme)
  • Shadow of Mordor (1080p Ultra, 4K Ultra)
  • Rise of the Tomb Raider #1 - GeoValley (1080p High, 4K Medium)
  • Rise of the Tomb Raider #2 - Prophets (1080p High, 4K Medium)
  • Rise of the Tomb Raider #3 - Mountain (1080p High, 4K Medium)
  • Rocket League (1080p Ultra, 4K Ultra)
  • Grand Theft Auto V (1080p Very High, 4K High)

For each of the GPUs in our testing, these games (at each resolution/setting combination) are run four times each, with outliers discarded. Average frame rates, 99th percentiles and 'Time Under x FPS' data is sorted, and the raw data is archived.

The four GPUs we've managed to obtain for these tests are:

  • MSI GTX 1080 Gaming X 8G
  • ASUS GTX 1060 Strix 6G
  • Sapphire Nitro R9 Fury 4GB
  • Sapphire Nitro RX 480 8GB

In our testing script, we save a couple of special things for the GTX 1080 here. The following tests are also added:

  • Civilization 6 (8K Ultra, 16K Lowest)

This benchmark, with a little coercion, are able to be run beyond the specifications of the monitor being used, allowing for 'future' testing of GPUs at 8K and 16K with some amusing results. We are only running these tests on the GTX 1080, because there's no point watching a slideshow more than once.

*As an additional to this review, we do not have any CPU gaming data on Skylake-X. We ran a set of tests before Threadripper arrived, but now having had a chance to analyze the data, despite being on the latest BIOS and setup, there are still issues with performance that we need to nail down once this review is out of the way.

Test Bed and Setup Benchmarking Performance: CPU System Tests
Comments Locked

347 Comments

View All Comments

  • nitin213 - Thursday, August 10, 2017 - link

    Thanks for your reply. Hopefully the test suite can be expanded as Intel's CPUs probably also move to higher core count and IO ranges in future.
    and i completely understand the frustration trying to get a 3rd party to change their defaults. Cheers
  • deathBOB - Thursday, August 10, 2017 - link

    It's clear to me . . . Ian is playing both sides and making out like a bandit! /s
  • FreckledTrout - Thursday, August 10, 2017 - link

    Ian can we get an updated comments section so we can +/- people and after x number of minuses they wont show by default. I'm saying this because some of these comments(the one in this chain included) are not meaningful responces. The comments section is by far the weakest link on Anantech.

    Nice review btw.
  • mapesdhs - Thursday, August 10, 2017 - link

    toms has that, indeed it's kinda handy for blanking out the trolls. Whether it's any useful indicator of "valid" opinion though, well, that kinda varies. :D (there's nowt to stop the trolls from voting everything under the sun, though one option would be to auto-suspend someone's ability to vote if their own posts get hidden from down voting too often, a hands-off way of slapping the trolls)

    Given the choice, I'd much rather just be able to *edit* what I've posted than up/down-vote what others have written. I still smile recalling a guy who posted a followup to apologise for the typos in his o.p., but the followup had typos aswell, after which he posted aaaaagh. :D

    Ian.
  • Johan Steyn - Thursday, August 10, 2017 - link

    Ian thanks for at least responding, I appreciate it. Please compare your review to sites like PCPer and many others. They have no problem to also point out the weak points of TR, yet clearly understand for what TR was mostly designed and focus properly on it and even though they did not test the 64 PCI lanes as an example, mention that they are planning a follow-up to do it, since it is an important point. You do mention these as well, but could have said more than just mention it by the way.

    Look at your review, most of it is about games. Are you serious?

    I have to give you credit to at least mention the problems with Sysmark.

    Let me give you an example of slanted journalism, When you do the rendering benchmarks, where AMD is known to shine, you only mention at each benchmark what they do etc, and fail to mention that AMD clearly beats Intel, even though other sites focus more ons these benchmarks. In the one benchmark where Intel get a descent score, you take time to mention that:

    "Though it's interesting just how close the 10-core Core i9-7900X gets in the CPU (C++) test despite a significant core count disadvantage, likely due to a combination of higher IPC and clockspeeds."

    Not in one of the rendering benchmarks do you give credit to AMD, yet you found it fitting to end the section of with:

    "Intel recently announced that its new 18-core chip scores 3200 on Cinebench R15. That would be an extra 6.7% performance over the Threadripper 1950X for 2x the cost."

    Not slanted journalism? At least you mention "2x the cost," but for most this will not defer them in buying the monopoly.

    After focussing so much time on game performance, I am not sure you understand TR at all. AMD still has a long way to go in many areas. Why? Because corrupt Intel basically drove them to bankruptcy, but that is a discussion for another day. I lived through those days and experienced it myself.

    Maybe I missed it, but where did you discuss the issue of memory speed? You mention in the beginning of memory overclock. Did you test the system running at 3200 or 2666? It is important to note. If you ran at 2666, then you are missing a very important point. Ryzen is known to gain a huge amount with memory speed. You should not regard 3200 as an overclock, since that is what that memory is made for, even if 2666 is standard spec. Most other sites I checked, used it like that. If you did use 3200, don't you think you should mention it?

    Why is it that your review ends up meh about TR and leave you rather wanting an i9 an almost all respects, yet most of the other site gives admiration where deserved, even though they have criticism as well. Ian I see that you clearly are disappointed with TR, which is OK, maybe you just like playing games and that is why you are so.

    It was clear how much you admire Intel in your previous article. You say that I gave no examples of slanted journalism, maybe you should read my post again. "Most Powerful, Most scalable." It is well known that people don't read the fine print. This was intentional. If not, you are a very unlucky guys for having so many unintended mishaps. Then I truly need to say I am sorry.

    For once, please be a bit excited that there is some competition against the monopoly of Intel, or maybe you are also deluded that they became so without any underhanded ways.

    By the way, sorry that I called you Anand. I actually wanted to type Anandtech, but left it like it. This site still carries his name and he should still take responsibility. After I posted, I realised I should have just checked the author, so sorry about that.
  • vanilla_gorilla - Thursday, August 10, 2017 - link

    "Intel recently announced that its new 18-core chip scores 3200 on Cinebench R15. That would be an extra 6.7% performance over the Threadripper 1950X for 2x the cost."

    How do you not understand that is a dig at Intel? He's saying you have to pay twice as much for only a 6.7% improvement.
  • smilingcrow - Thursday, August 10, 2017 - link

    The memory speed approach taken was clearly explained in the test and was stated as being consistent with how they always test.
    I don't take issue with testing at stock speeds at launch day as running memory out of spec for the system can be evaluated in depth later on.
  • Johan Steyn - Friday, August 11, 2017 - link

    That is just rubbish. Threadripper has no problem with 3200 memory and other sites has no problem running it at that speed. 3200 memory is designed to run 3200, why run it at 2666? There is just no excuse except being paid by Intel.

    Maybe then you can accuse other sites of being unscientific?
  • fanofanand - Tuesday, August 15, 2017 - link

    Anandtech always tests at JDEC, regardless of the brand.
  • Manch - Friday, August 11, 2017 - link

    ""Intel recently announced that its new 18-core chip scores 3200 on Cinebench R15. That would be an extra 6.7% performance over the Threadripper 1950X for 2x the cost."

    Not slanted journalism? At least you mention "2x the cost," but for most this will not defer them in buying the monopoly."

    You call Intel the monopoly and call him out for not wording the sentence to dissuade people from buying Intel. Who has the bias here? If he was actively promoting Intel over AMD or vice versa, you'd be OK with the latter, but to do neither. He's an Intel shill? Come on. That's unfair. HOW should he have wrote it so it would satisfy you?

    FYI Anand is gone. He's NOT responsible for anything at Anandtech. Are you going to hold Wozniak's feet to the fire for the lack of ports on a Mac too?

Log in

Don't have an account? Sign up now