Launching the #CPUOverload Project: Testing Every x86 Desktop Processor since 2010
by Dr. Ian Cutress on July 20, 2020 1:30 PM ESTCPU Tests: SPEC2006 1T, SPEC2017 1T, SPEC2017 nT
SPEC2017 and SPEC2006 is a series of standardized tests used to probe the overall performance between different systems, different architectures, different microarchitectures, and setups. The code has to be compiled, and then the results can be submitted to an online database for comparison. It covers a range of integer and floating point workloads, and can be very optimized for each CPU, so it is important to check how the benchmarks are being compiled and run.
We run the tests in a harness built through Windows Subsystem for Linux, developed by our own Andrei Frumusanu. WSL has some odd quirks, with one test not running due to a WSL fixed stack size, but for like-for-like testing is good enough. SPEC2006 is deprecated in favor of 2017, but remains an interesting comparison point in our data. Because our scores aren’t official submissions, as per SPEC guidelines we have to declare them as internal estimates from our part.
For compilers, we use LLVM both for C/C++ and Fortan tests, and for Fortran we’re using the Flang compiler. The rationale of using LLVM over GCC is better cross-platform comparisons to platforms that have only have LLVM support and future articles where we’ll investigate this aspect more. We’re not considering closed-sourced compilers such as MSVC or ICC.
clang version 10.0.0
clang version 7.0.1 (ssh://git@github.com/flang-compiler/flang-driver.git
24bd54da5c41af04838bbe7b68f830840d47fc03)-Ofast -fomit-frame-pointer
-march=x86-64
-mtune=core-avx2
-mfma -mavx -mavx2
Our compiler flags are straightforward, with basic –Ofast and relevant ISA switches to allow for AVX2 instructions. We decided to build our SPEC binaries on AVX2, which puts a limit on Haswell as how old we can go before the testing will fall over. This also means we don’t have AVX512 binaries, primarily because in order to get the best performance, the AVX-512 intrinsic should be packed by a proper expert, as with our AVX-512 benchmark.
To note, the requirements for the SPEC licence state that any benchmark results from SPEC have to be labelled ‘estimated’ until they are verified on the SPEC website as a meaningful representation of the expected performance. This is most often done by the big companies and OEMs to showcase performance to customers, however is quite over the top for what we do as reviewers.
For each of the SPEC targets we are doing, SPEC2006 rate-1, SPEC2017 speed-1, and SPEC2017 speed-N, rather than publish all the separate test data in our reviews, we are going to condense it down into individual data points. The main three will be the geometric means from each of the three suites.
A fourth metric will be a scaling metric, indicating how well the nT result scales to the 1T result for 2017, divided by the number of cores on the chip.
The per-test data will be a part of Bench.
Experienced users should be aware that 521.wrf_r, part of the SPEC2017 suite, does not work in WSL due to the fixed stack size. It is expected to work with WSL2, however we will cross that bridge when we get to it. For now, we’re not giving wrf_r a score, which because we are taking the geometric mean rather than the average, should not affect the results too much.
Facebook
Twitter
RSS
110 Comments
View All Comments
vasily - Monday, July 20, 2020 - link
You might want to check out Phoronix Test Suite and openbenchmarking.org.https://www.phoronix-test-suite.com/
https://openbenchmarking.org/
colinisation - Monday, July 20, 2020 - link
would love to see the following processors added5775C (overclocked to 4Ghz) - just purely to see what impact the eDRAM has on workloads
4770K
7600K
Phenom II X4
Highest Bulldozer core
VIA's highest performance x86 core
faizoff - Monday, July 20, 2020 - link
What a gargantuan project this is going to be. And I cannot wait, oddly enough I've been using the bench tool the past few weeks to get a sense of how much difference an upgrade for me would make.I am probably one of the many (or few) people that have still held on to their i5 2500k and this is one of the places I can select that CPU and compare the benchmarks with newer releases.
This project looks to be an amazing read once all done and will be especially looking forward to those segments "how well does x CPU run today?"
Alim345 - Monday, July 20, 2020 - link
Are you going to make benchmark scripts available? They should be useful for individual comparisons, since many users might have overclocked CPUs which were more common in 2010-2015.brantron - Monday, July 20, 2020 - link
Just to fill out the starting set:7700K needs a common AMD counterpart, i.e. Ryzen 2600
Sandy or Ivy Bridge i7
Haswell i7
That would also make for a good article, as it should be possible to overclock any of those to ~4.5 GHz for a more apples to apples comparison.
StormyParis - Monday, July 20, 2020 - link
Thank you for that. My main question is not "what should I buy" because that's always very well covered, and on a fixed budget there's never much choice anyway, but "should I upgrade *now* which is only worth it when last time's amount of money gets you at least 2x performance. I'ive got a 7yo Core i5... I'll look into it !eastcoast_pete - Monday, July 20, 2020 - link
Ian, thanks for this!One aspect I've wondered about for a while is whether you could include performance/Watt in your tests and comparisons going forward? I know that's usually done for server CPUs, but I also find it of interest for desktop and laptop CPUs.
thebigteam - Monday, July 20, 2020 - link
I think I have the below list of Intel CPUs available if needed, likely with working mobos too. Would be very happy to clean out the closet and get these to you guys :) Likely some 2009/2010 Athlons as wellE8400
i3 530
i3 540
i5 760
i5 2500
i5 4670K
inighthawki - Monday, July 20, 2020 - link
Thank you so much for changing your gaming benchmark methodology. I tend to play my games at 1440p on lowest settings for maximum framerates, which is far more often than not CPU bound. It was always so annoying seeing the benchmarks be GPU bound when I'm trying to see how much a new CPU helps.Smell This - Monday, July 20, 2020 - link
Chicken(lol)
With AM3, AM2+ and AM2 processors, AM3+ processors broke backwards-compatibility.
A mobo like the MSI 790FX K9A2 Platinum transitioned nearly 250 processors from S754-939, to AM2-AM3, beginning with the single-core Athlon 64 3000+ 'Orleans' up to the PhII x6 DDR3 Thubans.
These were the progeny of the K8 or 'Hammer' projects. A Real Man would never leave them behind ...
https://www.cpu-upgrade.com/mb-MSI/K9A2_Platinum_%...