Benchmarking Suite 2017

2017 CPU

For our Ryzen review, we are implementing our new CPU testing benchmark suite, fully scripted as of 2/17. This means that with a fresh OS install, we can configure the OS to be more consistent, install the new benchmarks, maintain version consistency without random updates and start running the tests in under 5 minutes. After that it's a one button press to start an 8-10hr test (with a high-performance core) with nearly 100 relevant data points in the benchmarks given below. The tests cover a wide range of segments, some of which will be familiar but some of the tests are new to benchmarking in general, but still highly relevant for the markets they come from.

Our new CPU tests go through six main areas. We cover the Web (we've got an un-updateable version of Chrome 56), general system tests (opening tricky PDFs, emulation, brain simulation, AI, 2D image to 3D model conversion), rendering (ray tracing, modeling), encoding (compression, AES, h264 and HEVC), office based tests (PCMark and others), and our legacy tests, throwbacks from another generation of bad code but interesting to compare.

A side note on OS preparation. As we're using Windows 10, there's a large opportunity for something to come in and disrupt our testing. So our default strategy is multiple: disable the ability to update as much as possible, disable Windows Defender, uninstall OneDrive, disable Cortana as much as possible, implement the high performance mode in the power options, and disable the internal platform clock which can drift away from being accurate if the base frequency drifts (and thus the timing ends up inaccurate).

Web on Chrome 56

Sunspider
Kraken
Octane
Web13
Web15

System

PDF Opening
FCAT
3DPM v21
Dolphin
DigiCortex
Civilization 6
Agisoft PS v1.0 

Rendering

Corona
Blender
LuxMark CPU C++
LuxMark CPU OpenCL
POV-Ray
CB15 ST
CB15 MT

Encoding

7-Zip
WinRAR
TrueCrypt
HandBrake 264-LQ
HandBrake 264-HQ
HandBrake 265-4K

Office

PCMark8 
SYSmark 2014 / SE

Legacy

3DPM v1 ST / MT
x264 HD 3 Pass 1, Pass 2
CB 11.5 ST / MT
CB 10 ST / MT

A side note - a couple of benchmarks (Dolphin, Civ 6) weren't fully 100% giving good data during testing. Need to go back and re-work this part of our testing.

2017 GPU

The bad news for our Ryzen review is that our new 2017 GPU testing stack not yet complete. We recieved our Ryzen CPU samples on February 21st, and tested in the hotel at the event for 6hr before flying back to Europe.

I spent two days back in London, where ~12 CPUs relevant to the review today were testing on our new CPU benchmarks. This was before I had to fly to Barcelona for Mobile World Congress, and I brought 30kg of kit with me to help with the review. I have had Ryzen set up in our shared flat for the past few days, and had Ryzen benchmarks running while attending meetings. As a result, our CPU data is good, but we lack any substantial GPU comparison data, power numbers (some idiot senior editor forgot his power meter...) or overclocking numbers. Based on a few Twitter polls conducted over at @IanCutress, people seemed more interested in CPU performance anyway, so we'll do a Pt 2 with more GPU data in the next couple of weeks.

 

 

AMD Stock Coolers and Memory: Wraith v2 and DDR4 Test Bed Setup and Hardware
Comments Locked

574 Comments

View All Comments

  • EchoWars - Thursday, March 2, 2017 - link

    No, apparently the failure was in your education, since it's obvious you did not read the article.
  • Notmyusualid - Friday, March 3, 2017 - link

    Ha...
  • sharath.naik - Thursday, March 2, 2017 - link

    I think you missed the biggest news in this information dump. The TDP is the biggest advantage amd has. Which means that for 150watt server cpu. they should be able to cram a lot more cores than intel will be able to.
  • Meteor2 - Friday, March 3, 2017 - link

    ^^^This. I think AMD's strength with Zen is going to be in servers.
  • Sttm - Friday, March 3, 2017 - link

    Yeah I can see that.
  • UpSpin - Thursday, March 2, 2017 - link

    According to a german site, in games, Ryzen is equal (sometimes higher, sometimes lower) to the Intel i7-6900K in high resolution games (WQHD). Once the resolution is set very low (720p) the Ryzen gets beaten by the Intel processor, but honestly, who cares about low resolution? For games, the probably best bet would be the i7-7700K, mainly because of the higher clock rate, for now. Once the games get better optimized for 8 cores, the 4-core i7-7700K will be beaten for sure, because in multi-threaded applications Ryzen is on par with the twice expensive Intel processor.

    I doubt it makes sense to buy the Core i7-6850K, it has the same low turbo boost frequency the 6900K has, thus low single threaded performance, but at only 6 cores. So I expect that it's the worst from both worlds. Poor multi-threaded performance compared to Ryzen, poor single threaded performance compared to i7-7700K.

    We also have to see how well Ryzen can get overclocked, thus improving single core performance.
  • fanofanand - Thursday, March 2, 2017 - link

    That is a well reasoned comment. Kudos!
  • ShieTar - Thursday, March 2, 2017 - link

    Well, the point of low-resolution testing is, that at normal resolutions you will always be GPU-restricted. So not only Ryzen and the i7-6900K are equal in this test, but so are all other modern and half-modern CPUs including any old FX-8...

    The most interesting question will be how Ryzen performs on those few modern games which manage to be CPU-restricted even in relevant resolutions, e.g. Battlefield 1 Multiplayer. But I think it will be a few more days, if not weeks, until we get that kind of in-depth review.
  • FriendlyUser - Thursday, March 2, 2017 - link

    This is true, but at the same time this artificially magnifies the differences one is going to notice in a real-world scenario. I saw reviews with a Titan X at 1080p, while many will be playing 1440p with a 1060 or RX480.

    The test case must also approximate real life.
  • khanikun - Friday, March 3, 2017 - link

    They aren't testing to show what it's like in real life though. The point of testing is to show the difference between the CPUs. Hence why they are gearing their benchmarking to stress the CPU, not other portions of the system.

Log in

Don't have an account? Sign up now