2017 CPU Benchmarking

For our review, we are implementing our fresh CPU testing benchmark suite, using new scripts developed specifically for this testing. This means that with a fresh OS install, we can configure the OS to be more consistent, install the new benchmarks, maintain version consistency without random updates and start running the tests in under 5 minutes. After that it's a one button press to start an 8-10hr test (with a high-performance core) with nearly 100 relevant data points in the benchmarks given below. The tests cover a wide range of segments, some of which will be familiar but some of the tests are new to benchmarking in general, but still highly relevant for the markets they come from.

Our new CPU tests go through six main areas. We cover the Web (we've got an un-updateable version of Chrome 56), general system tests (opening tricky PDFs, emulation, brain simulation, AI, 2D image to 3D model conversion), rendering (ray tracing, modeling), encoding (compression, AES, h264 and HEVC), office based tests (PCMark and others), and our legacy tests, throwbacks from another generation of bad code but interesting to compare.

Our graphs typically list CPUs with microarchitecture, SKU name, cost and power. The cost will be one of two numbers, either the 1k unit price 'tray price' for when a business customer purchases 1000 CPUs, or the MSRP likely to be found at retail. The problem here is that neither Intel nor AMD are consistent: Intel has a tray price for every CPU, but an MSRP only for parts sold at retail. AMD typically quotes MSRP for CPUs at retail, tray prices for enterprise CPUs, and doesn't say much about OEM only parts. We try to find a balance here, so prices may be $10-$20 from what you might expect.

A side note on OS preparation. As we're using Windows 10, there's a large opportunity for something to come in and disrupt our testing. So our default strategy is multiple: disable the ability to update as much as possible, disable Windows Defender, uninstall OneDrive, disable Cortana as much as possible, implement the high performance mode in the power options, and disable the internal platform clock which can drift away from being accurate if the base frequency drifts (and thus the timing ends up inaccurate).

Additional Note for 7/28: As this review is being written, due to limited time, testing on the CPUs is still ongoing and some benchmark graphs will be added in time when the results come in and can be verified.

Web Tests on Chrome 56

Sunspider 1.0.2
Mozilla Kraken 1.1
Google Octane 2.0
WebXPRT15

System Tests

PDF Opening
FCAT
3DPM v2.1
Dolphin v5.0
DigiCortex v1.20
Agisoft PhotoScan v1.0

Rendering Tests

Corona 1.3
Blender 2.78
LuxMark CPU C++
LuxMark CPU OpenCL
POV-Ray 3.7.1b4
Cinebench R15 ST
Cinebench R15 MT

Encoding Tests

7-Zip 9.2
WinRAR 5.40
AES Encoding (TrueCrypt 7.2)
HandBrake v1.0.2 x264 LQ
HandBrake v1.0.2 x264-HQ
HandBrake v1.0.2 HEVC-4K

Office / Professional

PCMark8
Chromium Compile (v56)
SYSmark 2014 SE

Legacy Tests

3DPM v1 ST / MT
x264 HD 3 Pass 1, Pass 2
Cinebench R11.5 ST / MT
Cinebench R10 ST / MT

A side note - a couple of benchmarks (LuxMark) weren't fully 100% giving good data during testing. Need to go back and re-work this part of our testing.

2017 CPU Gaming Tests

For our new set of GPU tests, we wanted to think big. There are a lot of users in the ecosystem that prioritize gaming above all else, especially when it comes to choosing the correct CPU. If there's a chance to save $50 and get a better graphics card for no loss in performance, then this is the route that gamers would prefer to tread. The angle here though is tough - lots of games have different requirements and cause different stresses on a system, with various graphics cards having different reactions to the code flow of a game. Then users also have different resolutions and different perceptions of what feels 'normal'. This all amounts to more degrees of freedom than we could hope to test in a lifetime, only for the data to become irrelevant in a few months when a new game or new GPU comes into the mix. Just for good measure, let us add in DirectX 12 titles that make it easier to use more CPU cores in a game to enhance fidelity.

Our original list of nine games planned in February quickly became six, due to the lack of professional-grade controls on Ubisoft titles. If you want to see For Honor, Steep or Ghost Recon: Wildlands benchmarked on AnandTech, point Ubisoft Annecy or Ubisoft Montreal in my direction. While these games have in-game benchmarks worth using, unfortunately they do not provide enough frame-by-frame detail to the end user, despite using it internally to produce the data the user eventually sees (and it typically ends up obfuscated by another layer as well). I would instead perhaps choose to automate these benchmarks via inputs, however the extremely variable loading time is a strong barrier to this.

So we have the following benchmarks as part of our 4/2 script, automated to the point of a one-button run and out pops the results four hours later, per GPU. Also listed are the resolutions and settings used.

  • Civilization 6 (1080p Ultra, 4K Ultra)
  • Ashes of the Singularity: Escalation* (1080p Extreme, 4K Extreme)
  • Shadow of Mordor (1080p Ultra, 4K Ultra)
  • Rise of the Tomb Raider #1 - GeoValley (1080p High, 4K Medium)
  • Rise of the Tomb Raider #2 - Prophets (1080p High, 4K Medium)
  • Rise of the Tomb Raider #3 - Mountain (1080p High, 4K Medium)
  • Rocket League (1080p Ultra, 4K Ultra)
  • Grand Theft Auto V (1080p Very High, 4K High)

For each of the GPUs in our testing, these games (at each resolution/setting combination) are run four times each, with outliers discarded. Average frame rates, 99th percentiles and 'Time Under x FPS' data is sorted, and the raw data is archived.

The four GPUs we've managed to obtain for these tests are:

  • MSI GTX 1080 Gaming X 8G
  • ASUS GTX 1060 Strix 6G
  • Sapphire Nitro R9 Fury 4GB
  • Sapphire Nitro RX 480 8GB

In our testing script, we save a couple of special things for the GTX 1080 here. The following tests are also added:

  • Civilization 6 (8K Ultra, 16K Lowest)

This benchmark, with a little coercion, are able to be run beyond the specifications of the monitor being used, allowing for 'future' testing of GPUs at 8K and 16K with some amusing results. We are only running these tests on the GTX 1080, because there's no point watching a slideshow more than once.

Test Bed and Setup Benchmarking Performance: CPU System Tests
POST A COMMENT

140 Comments

View All Comments

  • Gavin Bonshor - Thursday, July 27, 2017 - link

    One of the hardest working men in the industry! :D Reply
  • edlee - Thursday, July 27, 2017 - link

    I dont understand the point of making a $100 cpu without an integrated gpu if you wanted to attract the lower end market, this is really silly mistake. Sort of like intel including an integrated gpu with i7-7700k, it doesnt make sense, 95% of those with a 7700k will buy a gpu, but someone who is looking for a lowend cpu is not going to buy a discrete graphics cards, its just silly Reply
  • phoenix_rizzen - Thursday, July 27, 2017 - link

    It really depends on the use case.

    For example, are there any integrated GPUs that support 3 monitors? I know a lot of them support dual monitors, but haven't come across any that support 3 (although I haven't looked that hard). My work PC is a low-profile desktop running an AMD Athlon-II x4 CPU and an Nvidia 730 GT GPU for tri-monitor setup. Upgrading the CPU/motherboard/RAM to a Ryzen 3 1300X would be a huge upgrade for this system.

    90-odd % of the desktops in the schools here use AMD Athlon-II CPUs (graphics integrated into the chipset), with the rest using Intel Pentium CPUs (graphics integrated into the CPU). And we add Nvidia 210 or 730 GPUs to those that need better multi-monitor support or better 3D performance. Why do we do it that way? Cost. We try to keep the complete desktop system (case, motherboard,
    CPU, at least 2 GB RAM, no storage of any kind) to under $200 CDN (they're diskless Linux stations). We have just shy of 5000 of those in the district right now.

    We've avoided the Bulldozer-based APUs so far as the price/performance just wasn't there compared to the Pentium line (from our suppliers). But the Ryzen 3 looks like a decent upgrade. Will be interesting to see what the prices are like for it from our suppliers this winter/spring. Will also be interesting to see what the GPU side of the Zen-based APUs will be like next year.

    The other important bit is driver support. We are a mostly Linux-using school district, so we tend to use hardware that's at least 2 steps back from the bleeding edge. That way, we get better prices, and better driver support.
    Reply
  • edlee - Thursday, July 27, 2017 - link

    i understand when upgrading from integrated to gpu like you stated in your use case, but from the low end price standpoint, a i3-7100 is cheaper because they dont need to add a gpu like the ryzen 3 needs, so its not competing on a performance standpoint or a price standpoint when you add the price of the cheapest gpu Reply
  • Outlander_04 - Friday, July 28, 2017 - link

    Using an integrated gpu is usually a poor choice. Intels drivers are so dumbed down they are worse than hopeless.
    Factor in that using integrated means less system RAM available as well so performance can be reduced
    Reply
  • Ratman6161 - Tuesday, August 01, 2017 - link

    Many people may be starting out from the position of knowing that the integrated graphics on any of the Intel CPU's in the test are not good enough for them. If you know that from the start then the argument that AMD doesn't have an IGPU is meaningless. I'm also somewhat interested in seeing overclocking tests with the R3 as that is one thing you just don't get with Intel at this level short of the 7350K. I sort of suspect that an OC'd 1200 could but just as fast or faster than a 1300X (though at only a $20 difference I'm not sure how much it matters).
    Also, in more computationally intense tasks, the 1300x really doesn't do badly against the i5 that costs $53 more so once again, if you don't care about integrated graphics it could be a good choice for some people.

    On the other hand, for someone for whom MS Office, email, and web browsing are their main uses, then something like the i3-7100 suddenly looks very attractive - or even the Pentium G.
    In this segment, AMD really needs to get a Ryzen Based APU on the market. If they did a single CCX, 4 core and used the empty space vacated by the second CCX for a decent IGPU they could definitely have an i3 killer.
    Reply
  • renw0rp - Thursday, July 27, 2017 - link

    I had HP Folio 9470m with core i5-3437U and it was driving 3 * 1920x1200 screens without an issue. And it's ~2013 processor...

    3rd gen of Core processors was the first to support 3 displays. The 2nd gen supported just 2.
    Reply
  • stuartlew - Thursday, July 27, 2017 - link

    AMD Kaveri does three monitors Reply
  • serendip - Friday, July 28, 2017 - link

    Are there motherboards with integrated chipset graphics for Ryzen?

    I understand the good thing about adding a discrete GPU only to PCs that need one but not having an integrated GPU is nuts, for the mass market at least.
    Reply
  • silverblue - Friday, July 28, 2017 - link

    No, but Bristol Ridge launched yesterday, so there are now APUs that use AM4. Reply

Log in

Don't have an account? Sign up now