Benchmarking Suite 2017

2017 CPU

For our Ryzen review, we are implementing our new CPU testing benchmark suite, fully scripted as of 2/17. This means that with a fresh OS install, we can configure the OS to be more consistent, install the new benchmarks, maintain version consistency without random updates and start running the tests in under 5 minutes. After that it's a one button press to start an 8-10hr test (with a high-performance core) with nearly 100 relevant data points in the benchmarks given below. The tests cover a wide range of segments, some of which will be familiar but some of the tests are new to benchmarking in general, but still highly relevant for the markets they come from.

Our new CPU tests go through six main areas. We cover the Web (we've got an un-updateable version of Chrome 56), general system tests (opening tricky PDFs, emulation, brain simulation, AI, 2D image to 3D model conversion), rendering (ray tracing, modeling), encoding (compression, AES, h264 and HEVC), office based tests (PCMark and others), and our legacy tests, throwbacks from another generation of bad code but interesting to compare.

A side note on OS preparation. As we're using Windows 10, there's a large opportunity for something to come in and disrupt our testing. So our default strategy is multiple: disable the ability to update as much as possible, disable Windows Defender, uninstall OneDrive, disable Cortana as much as possible, implement the high performance mode in the power options, and disable the internal platform clock which can drift away from being accurate if the base frequency drifts (and thus the timing ends up inaccurate).

Web on Chrome 56

Sunspider
Kraken
Octane
Web13
Web15

System

PDF Opening
FCAT
3DPM v21
Dolphin
DigiCortex
Civilization 6
Agisoft PS v1.0 

Rendering

Corona
Blender
LuxMark CPU C++
LuxMark CPU OpenCL
POV-Ray
CB15 ST
CB15 MT

Encoding

7-Zip
WinRAR
TrueCrypt
HandBrake 264-LQ
HandBrake 264-HQ
HandBrake 265-4K

Office

PCMark8 
SYSmark 2014 / SE

Legacy

3DPM v1 ST / MT
x264 HD 3 Pass 1, Pass 2
CB 11.5 ST / MT
CB 10 ST / MT

A side note - a couple of benchmarks (Dolphin, Civ 6) weren't fully 100% giving good data during testing. Need to go back and re-work this part of our testing.

2017 GPU

The bad news for our Ryzen review is that our new 2017 GPU testing stack not yet complete. We recieved our Ryzen CPU samples on February 21st, and tested in the hotel at the event for 6hr before flying back to Europe.

I spent two days back in London, where ~12 CPUs relevant to the review today were testing on our new CPU benchmarks. This was before I had to fly to Barcelona for Mobile World Congress, and I brought 30kg of kit with me to help with the review. I have had Ryzen set up in our shared flat for the past few days, and had Ryzen benchmarks running while attending meetings. As a result, our CPU data is good, but we lack any substantial GPU comparison data, power numbers (some idiot senior editor forgot his power meter...) or overclocking numbers. Based on a few Twitter polls conducted over at @IanCutress, people seemed more interested in CPU performance anyway, so we'll do a Pt 2 with more GPU data in the next couple of weeks.

 

 

AMD Stock Coolers and Memory: Wraith v2 and DDR4 Test Bed Setup and Hardware
Comments Locked

574 Comments

View All Comments

  • mapesdhs - Sunday, March 5, 2017 - link

    Yet another example of manipulation which wouldn't be tolerated in other areas of commercial product. I keep coming across examples in the tech world where products are deliberately crippled, prices get hiked, etc., but because it's tech stuff, nobody cares. Media never mentions it.

    Last week I asked a seller site about why a particular 32GB 3200MHz DDR4 kit they had listed (awaiting an ETA) was so much cheaper than the official kits for Ryzen (same brand of RAM please note). Overnight, the seller site changed the ETA to next week but also increased the price by a whopping 80%, making it completely irrelevant. I've seen this happen three times with different products in the last 2 weeks.

    Ian.
  • HomeworldFound - Sunday, March 5, 2017 - link

    If they were pretty cheap then use your logic, placeholder prices happen. If they had no ETA the chances is that they had no prices. I don't see a shortage of decent DDR4 so it definitely isn't a supply and demand problem. Perhaps you need to talk to the manufacturer to get their guideline prices.
  • HomeworldFound - Sunday, March 5, 2017 - link

    Not really. If developers wanted to enhance AMD platforms, or it was actually worth it they'd have done it by now. It's now just an excuse to explain either underperformance or an inability to work with the industry.
  • Notmyusualid - Tuesday, March 7, 2017 - link

    @ sedra

    It certainly should not be forgotten, that is for sure.
  • Rene23 - Monday, March 6, 2017 - link

    yet people here mentioned multiple times "settled in 2009"; pretending it is not happening anymore, sick :-/
  • GeoffreyA - Monday, March 6, 2017 - link

    I kind of vaguely knew that benchmarks were often unfairly optimised for Intel CPUs; but I never knew this detailed information before, and from such a reputable source: Agner Fog. I know that he's an authority on CPU microarchitectures and things like that. Intel is evil. Even now with Ryzen, it seems the whole software ecosystem is somewhat suboptimal on it, because of software being tuned over the last decade for the Core microarchitecture. Yet, despite all that, Ryzen is still smashing Intel in many of the benchmarks.
  • Outlander_04 - Monday, March 6, 2017 - link

    Settled in 2009 .
    Not relevant to optimisation for Ryzen in any way
  • Rene23 - Monday, March 6, 2017 - link

    settled in 2009 does not mean their current compiler and libraries are not doing it anymore, e.g. it could simply not run the best SSE/AVX code path disguised as simply not matching new AMD cpus properly.
  • cocochanel - Saturday, March 4, 2017 - link

    One thing that is not being mentioned by many is the increase in savings when you buy a CPU + mobo. Intel knows how to milk the consumer. On their 6-8 core flagships, a mobo with a top chipset will set you back 300-400 $ or even more. That's a lot for a mobo. Add the overpriced CPU. I expect AMD mobos to offer better value. Historically, they always did.
    On top of that, a VEGA GPU will probably be a better match for Ryzen than an Nvidia card, but I say probably and not certainly.
    If I were to replace my aging gaming rig for Christmas, this would be my first choice.
  • mapesdhs - Sunday, March 5, 2017 - link

    Bang goes the saving when one asks about a RAM kit awaiting an ETA and the seller hikes the price by 80% overnight (see my comment above).

Log in

Don't have an account? Sign up now