CPU Tests: SPEC

Page by Andrei Frumusanu

SPEC2017 is a series of standardized tests used to probe the overall performance between different systems, different architectures, different microarchitectures, and setups. The code has to be compiled, and then the results can be submitted to an online database for comparison. It covers a range of integer and floating point workloads, and can be very optimized for each CPU, so it is important to check how the benchmarks are being compiled and run.

We run the tests in a harness built through Windows Subsystem for Linux, developed by our own Andrei Frumusanu. WSL has some odd quirks, with one test not running due to a WSL fixed stack size, but for like-for-like testing is good enough. Because our scores aren’t official submissions, as per SPEC guidelines we have to declare them as internal estimates from our part.

For compilers, we use LLVM both for C/C++ and Fortan tests, and for Fortran we’re using the Flang compiler. The rationale of using LLVM over GCC is better cross-platform comparisons to platforms that have only have LLVM support and future articles where we’ll investigate this aspect more. We’re not considering closed-sourced compilers such as MSVC or ICC.

clang version 10.0.0
clang version 7.0.1 (ssh://git@github.com/flang-compiler/flang-driver.git
 24bd54da5c41af04838bbe7b68f830840d47fc03)

-Ofast -fomit-frame-pointer
-march=x86-64
-mtune=core-avx2
-mfma -mavx -mavx2

Our compiler flags are straightforward, with basic –Ofast and relevant ISA switches to allow for AVX2 instructions. We decided to build our SPEC binaries on AVX2, which puts a limit on Haswell as how old we can go before the testing will fall over. This also means we don’t have AVX512 binaries, primarily because in order to get the best performance, the AVX-512 intrinsic should be packed by a proper expert, as with our AVX-512 benchmark. All of the major vendors, AMD, Intel, and Arm, all support the way in which we are testing SPEC.

To note, the requirements for the SPEC licence state that any benchmark results from SPEC have to be labeled ‘estimated’ until they are verified on the SPEC website as a meaningful representation of the expected performance. This is most often done by the big companies and OEMs to showcase performance to customers, however is quite over the top for what we do as reviewers.

For the new Cypress Cove based i7-11700K, we haven’t had quite the time to investigate the new AVX-512 instruction differences – since this is the first consumer desktop socketed CPU with the new ISA extensions it’s something we’ll revisit in the full review. Based on our testing on the server core counterparts however, it doesn’t make any noticeable differences in SPEC.

SPECint2017 Rate-1 Estimated Scores

In the SPECint2017 suite, we’re seeing the new i7-11700K able to surpass its desktop predecessors across the board in terms of performance. The biggest performance leap is found in 523.xalancbmk which consists of XML processing at a large +54.4% leap versus the 10700K.

The rest of the improvements range in the +0% to +15% range, with an average total geomean advantage of +15.5% versus the 10700K. The IPC advantage should be in the +18.5% range.

SPECfp2017 Rate-1 Estimated Scores

In the FP scores, there’s nothing standing out too much, with general even improvements across the board. The total improvement here is +19.6%, with the IPC improvement in the +22% range.

SPEC2017 Rate-1 Estimated Total

Although the new Cypress Cove cores in the 11700K do have good generational IPC improvements, that’s all compared to the quite old predecessor, meaning that for single-thread performance, the advancements aren’t enough to quite keep up with the latest Zen3 competition from AMD, or for that matter, the Firestorm cores in Apple’s new M1.

SPEC2017 Rate-N Estimated Total

More interesting are the multi-threaded SPEC results. Here, the new generation from Intel is showcasing a +5.8% and +16.2% performance improvement over its direct predecessor. Given the power draw increases we’ve seen this generation, those are rather unimpressive results, and actually represent a perf/W regression. AMD’s current 6-core 5600X actually is very near to the new 11700K, but consuming a fraction of the power.

CPU Tests: Encoding and Legacy/Web Gaming Tests: Deus Ex Mankind Divided
Comments Locked

541 Comments

View All Comments

  • zzzxtreme - Sunday, March 7, 2021 - link

    I wished you would have tested the XE graphics
  • Fman4 - Monday, March 8, 2021 - link

    Am I the only one find that OP plugged 4 RAMs on an X570 ITX motherboard?
  • Fman4 - Monday, March 8, 2021 - link

    @Dr. Ian Cutress
  • zodiacfml - Monday, March 8, 2021 - link

    bored. just here to say this is unsurprising though this strongly reminds me of the time where AMD is releasing new, well designed CPUs but two process node generations behind intel. I think AMD was 32nm and 28nm while Intel is 22 and 14nm. most comments were really harsh with AMD but I reasoned that it is simply due to the manufacturing superiority of Intel
  • blppt - Monday, March 8, 2021 - link

    Bulldozer and Piledriver are not the examples I would put up for "well designed".
  • GeoffreyA - Tuesday, March 9, 2021 - link

    Still, within that mess, AMD did a pretty good job raising Bulldozer's IPC and cutting down its power each generation. But the foundation being fatally flawed, it was hopeless. I believe it taught them a lot about cutting power and so on, and when they poured that into Zen, we saw the result. Bulldozer was a fantastic training ground, if one looks at it humorously.
  • Oxford Guy - Tuesday, March 9, 2021 - link

    No, AMD did an extremely poor job.

    Firstly, Bulldozer had worse IPC than Phenom. No engineers with brains release a CPU to replace the entire line while giving it worse IPC. The trap of going for high clocks was a lesson shown to the entire industry via Netburst. AMD's engineers knew all about it, yet someone at the company decided to try Netburst 2.0.

    Secondly, AMD was so sloppy and lazy that Piledriver shipped with a performance regression in AVX. It was worse to use AVX than to not use it. How incredibly incompetent can the company have been? It doesn't take a high IQ to understand that one doesn't ship broken AVX.

    AMD then refused to replace Piledriver until Zen came out. It tinkered half-heartedly with APU rubbish and focused on pushing junk like Jaguar.

    While it's true that the extreme failure of AMD (the construction core line) is due, to a large degree, to Intel abusing its monopoly to starve AMD of customers and cash — cash it needed to do R&D, one does not release a new chip with worse IPC and then very shortly after break AVX and refuse to stop feeding that junk to customers for many years. Just tinkering with Phenom would have been better (Phenom 3).

    As for the foundation claim... we have no idea how well the CMT concept could have worked out with competent engineering. Remember, they literally broke AVX in the Piledriver revision that was supposed to fix Bulldozer enough to make it sellable. Operations caching could have been stronger. The L3 cache was almost as slow as main memory. The RAM controller was weak, just like Phenom's. Etc.

    We paid for Intel's monopoly and we're still paying today. Only its monopoly and the lack of adequate competition is enabling the company to be so profitable despite failing so badly. Relying on two companies (or one 1/2, when it comes to R&D money ratio and other factors) to deliver adequate competition doesn't work.

    Google and Microsoft = Google owns the clearnet. Apparently, they have some sort of cooperation agreement which helps to explain why Bing has such a tiny index and such a poor-quality search.

    TSMC and Samsung = Can't meet demand.

    AMD and Nvidia = Nvidia keeps breaking profit records while utterly failing to meet demand. Both companies refuse to stop making their cards attractive for mining and have for a long long time. AMD refused to adequately compete beyond the lower midrange (Polaris forever, or you can buy a 'console'!) for a long time, leaving us to pay through the nose for Nvidia's prices. AMD literally competes against the PC market by pushing the console scam. Consoles are gaming PCs in disguise and they're parasitic in multiple ways, including in terms of wafer allocations. AMD's many many years of refusal to compete with Nvidia beyond the Polaris price point caused so much pent-up demand and now the company can enjoy the artificially high price points from that. It let Nvidia keep raising prices to get consumers used to that. Now that it has finally been forced to improve the 'consoles' beyond the garbage-tier Jaguar CPU it has to offer a bit more value to the PC gaming market. And so, after all these years, we have something decent that one can't buy. I can go on about this so-called competition but why bother. People will go to the most extravagant lengths to excuse the problem of lack of adequate competition — like the person who recently said it's easier to create Google's empire from scratch than it is to make a competitive GPU and sell it as a third GPU company.

    There are plenty of other areas in tech with inadequate competition, too.
  • blppt - Tuesday, March 9, 2021 - link

    "AMD then refused to replace Piledriver until Zen came out. It tinkered half-heartedly with APU rubbish and focused on pushing junk like Jaguar."

    To be fair, AMD had put a LOT of time, money and effort into Bulldozer/Piledriver, and were never a company with bottomless wells of cash to toss an architecture out immediately. Plus, Zen took a long time to design and finalize---thankfully, they made literally ALL the right moves in designing it, including hiring the brilliant Jim Keller.

    I think if Zen had been another BD like failure, that would have been the almost the end of AMD in the cpu market (leaving them basically as ATI was) The consoles likely would have gone with Intel or ARM for their next iteration. AMD once again spent tons of money that they don't have as disposable income in designing Zen. Two failures in a row would have been disastrous.

    Heck, the consoles might go with their own custom ARM design for PS6/Xbox(whatever) anyways.
  • GeoffreyA - Wednesday, March 10, 2021 - link

    blppt. Agreed, that would have been the end of AMD.
  • Oxford Guy - Wednesday, March 10, 2021 - link

    AMD did not put a lot of resources into fixing Bulldozer.

    It shipped Piledriver with broken AVX and never bothered to replace Piledriver on the desktop until Zen.

    Inexcusable. It shipped Steamroller and Excavator in cost-cut mode, cutting cores, cutting clocks, cutting the socket standards, and cutting cache. It used a dense library to save money by keeping the die small and used the inferior 28nm bulk process.

    Pathetic in basically every respect.

Log in

Don't have an account? Sign up now