Launching the #CPUOverload Project: Testing Every x86 Desktop Processor since 2010
by Dr. Ian Cutress on July 20, 2020 1:30 PM ESTCPU Tests: SPEC2006 1T, SPEC2017 1T, SPEC2017 nT
SPEC2017 and SPEC2006 is a series of standardized tests used to probe the overall performance between different systems, different architectures, different microarchitectures, and setups. The code has to be compiled, and then the results can be submitted to an online database for comparison. It covers a range of integer and floating point workloads, and can be very optimized for each CPU, so it is important to check how the benchmarks are being compiled and run.
We run the tests in a harness built through Windows Subsystem for Linux, developed by our own Andrei Frumusanu. WSL has some odd quirks, with one test not running due to a WSL fixed stack size, but for like-for-like testing is good enough. SPEC2006 is deprecated in favor of 2017, but remains an interesting comparison point in our data. Because our scores aren’t official submissions, as per SPEC guidelines we have to declare them as internal estimates from our part.
For compilers, we use LLVM both for C/C++ and Fortan tests, and for Fortran we’re using the Flang compiler. The rationale of using LLVM over GCC is better cross-platform comparisons to platforms that have only have LLVM support and future articles where we’ll investigate this aspect more. We’re not considering closed-sourced compilers such as MSVC or ICC.
clang version 10.0.0
clang version 7.0.1 (ssh://git@github.com/flang-compiler/flang-driver.git
24bd54da5c41af04838bbe7b68f830840d47fc03)-Ofast -fomit-frame-pointer
-march=x86-64
-mtune=core-avx2
-mfma -mavx -mavx2
Our compiler flags are straightforward, with basic –Ofast and relevant ISA switches to allow for AVX2 instructions. We decided to build our SPEC binaries on AVX2, which puts a limit on Haswell as how old we can go before the testing will fall over. This also means we don’t have AVX512 binaries, primarily because in order to get the best performance, the AVX-512 intrinsic should be packed by a proper expert, as with our AVX-512 benchmark.
To note, the requirements for the SPEC licence state that any benchmark results from SPEC have to be labelled ‘estimated’ until they are verified on the SPEC website as a meaningful representation of the expected performance. This is most often done by the big companies and OEMs to showcase performance to customers, however is quite over the top for what we do as reviewers.
For each of the SPEC targets we are doing, SPEC2006 rate-1, SPEC2017 speed-1, and SPEC2017 speed-N, rather than publish all the separate test data in our reviews, we are going to condense it down into individual data points. The main three will be the geometric means from each of the three suites.
A fourth metric will be a scaling metric, indicating how well the nT result scales to the 1T result for 2017, divided by the number of cores on the chip.
The per-test data will be a part of Bench.
Experienced users should be aware that 521.wrf_r, part of the SPEC2017 suite, does not work in WSL due to the fixed stack size. It is expected to work with WSL2, however we will cross that bridge when we get to it. For now, we’re not giving wrf_r a score, which because we are taking the geometric mean rather than the average, should not affect the results too much.
110 Comments
View All Comments
Arbie - Monday, July 20, 2020 - link
I din't realize how much work was being done. Thank you for maintaining this great resource.Arbie - Monday, July 20, 2020 - link
And maybe consider the technically excellent and easily benchmarked Ashes of the Singularity instead of the problematic Far Cry 5. Not as popular but modern and multi-core (and a great game).BushLin - Monday, July 20, 2020 - link
I suspect it's due to Far Cry 5's need for 8 threads which manifests in stutter for 6c6t CPUs in contrast to smooth gameplay on lower clocked 4c8t CPUs.Tilmitt - Tuesday, July 21, 2020 - link
Has anyone ever played Ashes as a game though?Arbie - Tuesday, July 21, 2020 - link
1000+ hours so far. Glorious in all respects including phenomenal AI. But when Ashes is mentioned someone always pops a comment like yours, which they probably just read somewhere else since it certainly isn't based on actual experience. Still hurts the game, though.driscoll42 - Monday, July 20, 2020 - link
This is awesome and amazing, I can't wait to see the results. And I hate to say "But what about", but maybe, if possible, go back to some of the popular older ones? No need to retest *everything*, but the most popular CPUs pre-2010 like the i7-920, Core 2 Quad Q6600, Core 2 Duo E8600, Core i7-870, etc...ltcommanderdata - Monday, July 20, 2020 - link
If they are going to test a few LGA 775 CPUs, I'd vote to also include NetBurst's last gasp, aka the Pentium Extreme Edition 965 as a really old gen reference. It'd be interesting to include it's then competitor, the Socket 939 AMD Athlon 64 FX-60, as well. I've always been curious whether Hyperthreading support allowed the Pentium EE 965 to age better than expected as multithreading became mainstream and possibly reduce the gap against the FX-60 and even early Core 2 Duo Conroe CPUs in modern software compared to the gap seen at launch.mganai - Thursday, July 23, 2020 - link
How about the dual socket LGA 771 with two Core 2 Extreme QX9650s?https://www.youtube.com/watch?v=wNo7qoLRtkQ
aryonoco - Monday, July 20, 2020 - link
Epic work Ian. Epic!Now if only your publisher implemented a subscription model (a la Ars Technica) so I could still support your work withthout being bombarded by ads and tracked, I would feel a lot less guilty enjoying the fruits of your amazing work.
lmcd - Tuesday, July 21, 2020 - link
This 100%