SPEC2006 & 2017: Industry Standard - ST Performance

One big talking point around the new Ryzen 3000 series is the new augmented single-threaded performance of the new Zen 2 core. In order to investigate the topic in a more controlled manner with better documented workloads, we’ve fallen back to the industry standard SPEC benchmark suite.

We’ll be investigating the previous generation SPEC CPU2006 test suite giving us some better context to past platforms, as well as introducing the new SPEC CPU2017 suite. We have to note that SPEC2006 has been deprecated in favour of 2017, and we must also mention that the scores posted today are noted as estimates as they’re not officially submitted to the SPEC organisation.

For SPEC2006, we’re still using the same setup as on our mobile suite, meaning all the C/C++ benchmarks, while for SPEC2017 I’ve also went ahead and prepared all the Fortran tests for a near complete suite for desktop systems. I say near complete as due to time constraints we’re running the suite via WSL on Windows. I’ve checked that there are no noticeable performance differences to native Linux (we’re also compiling statically), however one bug on WSL is that it has a fixed stack size so we’ll be missing 521.wrf_r from the SPECfp2017 collection.

In terms of compilers, I’ve opted to use LLVM both for C/C++ and Fortran tests. For Fortran, we’re using the Flang compiler. The rationale of using LLVM over GCC is better cross-platform comparisons to platforms that have only have LLVM support and future articles where we’ll investigate this aspect more. We’re not considering closed-sourced compilers such as MSVC or ICC.

clang version 8.0.0-svn350067-1~exp1+0~20181226174230.701~1.gbp6019f2 (trunk)
clang version 7.0.1 (ssh://git@github.com/flang-compiler/flang-driver.git 

-Ofast -fomit-frame-pointer
-mfma -mavx -mavx2

Our compiler flags are straightforward, with basic –Ofast and relevant ISA switches to allow for AVX2 instructions.

The Ryzen 3900X system was run in the same way as the rest of our article with DDR4-3200CL16, same as with the i9-9900K, whilst the Ryzen 2700X had DDR-2933 with similar CL16 16-16-16-38 timings.

SPECint2006 Speed Estimated Scores

In terms of the int2006 benchmarks, the improvements of the new Zen2 based Ryzen 3900X is quite even across the board when compared to the Zen+ based Ryzen 2700X. We do note however somewhat larger performance increases in 403.gcc and 483.xalancbmk – it’s not immediately clear as to why as the benchmarks don’t have one particular characteristic that would fit Zen2’s design improvements, however I suspect it’s linked to the larger L3 cache.

445.gobmk in particular is a branch-heavy workload, and the 35% increase in performance here would be better explained by Zen2’s new additional TAGE branch predictor which is able to reduce overall branch misses.

It’s also interesting that although Ryzen3900X posted worse memory latency results than the 2700X, it’s still able to outperform the latter in memory sensitive workloads such as 429.mcf, although the increases for 471.omnetpp is amongst the smallest in the suite.

However we still see that AMD has an overall larger disadvantage to Intel in these memory sensitive tests, as the 9900K has large advantages in 429.mcf, and posting a large lead in the very memory bandwidth intensive 462.libquantum, the two tests that put the most pressure on the caches and memory subsystem.

SPECfp2006(C/C++) Speed Estimated Scores

In the fp2006 benchmarks, we gain see some larger jumps on the part of the Ryzen 3900X, particularly in 482.sphinx3. These two tests along with 450.soplex are characterized by higher data cache misses, so Zen2’s 16MB L3 cache should definitely be part of the reason we see such larger jumps.

I found it interesting that we’re not seeing much improvements in 470.lbm even though this is a test that is data store heavy, so I would have expected Zen2’s additional store AGU to greatly benefit this workload. There must be some higher level memory limitations which is bottlenecking the test.

453.povray isn’t data heavy nor branch heavy, as it’s one of the more simple workloads in the suite. Here it’s mostly up to the execution backend throughput and the ability of the front-end to feed it fast enough that are the bottlenecks. So while the Ryzen 3900X provides a big boost over the 2700X, it’s still largely lagging behind the 9900K, a characteristic we’re also seeing in the similar execution bottlenecked 456.hmmer of the integer suite.

SPEC2006 Speed Estimated Total

Overall, the 3900X is 25% faster in the integer and floating point tests of the SPEC2006 suite, which corresponds to an 17% IPC increase, above AMD's officially published figures for IPC increases.

Moving on to the 2017 suite, we have to clarify that we’re using the Rate benchmark variations. The 2017 suite’s speed and rate benchmarks differ from each other in terms of workloads. The speed tests were designed for single-threaded testing and have large memory demands of up to 11GB, while the rate tests were meant for multi-process tests. We’re using the rate variations of the benchmarks because we don’t see any large differentiation between the two variations in terms of their characterisation and thus the performance scaling between the both should be extremely similar. On top of that, the rate benchmarks take up to 5x less time (+1 hour vs +6 hours), and we're able run them on more memory limited platforms (which we plan on to do in the future).

SPECint2017 Rate-1 Estimated Scores

In the int2017 suite, we’re seeing similar performance differences and improvements, although this time around there’s a few workloads that are a bit more limited in terms of their performance boosts on the new Ryzen 3900X.

Unfortunately I’m not quite as familiar with the exact characteristics of these tests as I am with the 2006 suite, so a more detailed analysis should follow in the next few months as we delve deeper into microarchitectural counters.

SPECfp2017 Rate-1 Estimated Scores

In the fp2017 suite, things are also quite even. Interesting enough here in particular AMD is able to leapfrog Intel’s 9900K in a lot more workloads, sometimes winning in terms of absolute performance and sometimes losing.

SPEC2017 Rate-1 Estimated Total

As for the overall performance scores, the new Ryzen 3900X improves by 23% over the 2700X. Although closing the gap greatly and completely, it’s just a hair's width shy of actually beating the 9900K’s absolute single-threaded performance.

SPEC2017 Rate-1 Estimated Performance Per GHz

Normalising the scores for frequency, we see that AMD has achieved something that the company hasn’t been able to claim in over 15 years: It has beat Intel in terms of overall IPC. Overall here, the IPC improvements over Zen+ are 15%, which is a bit lower than the 17% figure for SPEC2006.

We already know about Intel’s new upcoming Sunny Cove microarchitecture which should undoubtedly be able to regain the IPC crown with relative ease, but the question for Intel is if they’ll be able to still maintain the single-thread absolute performance crown and continue to see 5GHz or similar clock speeds with the new core design.

Test Bed and Setup Benchmarking Performance: Web Tests


View All Comments

  • TEAMSWITCHER - Monday, July 8, 2019 - link

    "Intel fleecing the market?" - Your hatred is showing. Reply
  • DrKlahn - Monday, July 8, 2019 - link

    No, I've bought both. But you'd have to be very naive to call the "progress" made after Bulldozer flopped and the Core architecture dominated anything but milking the market while moving things at a snails pace. Intel had every chance to continue to boldly innovate, but instead chose to parcel out small incremental changes and charge a hefty premium for them.

    I'll buy whatever makes sense. Right now that isn't Intel in my opinion. May change when Sunny Cove hits.
  • Xyler94 - Tuesday, July 9, 2019 - link

    The fact people are still holding onto Sandy Bridge because they don't feel like Coffee Lake isn't a good upgrade should be your reasoning that yes, Intel wasn't really doing much. Heck, I'm on my 4790k, and the only CPUs that peeked my interest is the AMD 3900x and 3950x, because those are great looking processors. If I were more into overclocking, maybe I'd spring for the 9900k, but it's not a processor that peeked my interest... Reply
  • Meteor2 - Monday, July 15, 2019 - link

    Well there was the "benchmarking Sandy Bridge in 2019" article a couple of months back. That showed that a 9700K is about 1.5-2x faster than a 2600K. Yes that's not the same rate of improvement, over seven years, which we saw up to Sandy Bridge, but it is still a hell of a lot faster for roughly the same cash price.

    It's just the increments -- a few percent a generation -- have been small. But they have compounded.
  • Xyler94 - Monday, July 22, 2019 - link

    Your 1.5 to 2 times faster was in what, productivity? What about gaming? That's what I was alluding to. Reply
  • Yaldabaoth - Sunday, July 7, 2019 - link

    Thank you so much!

    What more needs to be said about the 3900X relative to its peers? So glad I didn't spring for the i9-9900K a few months back!

    The 3700K is in a more interesting position at its price, but it looks like there are still some conditional performance advantages for the i7-9700K (read: single threaded workloads), but many times not. In any case, it's a great value and product. Kudos to AMD.

    What I _really_ want to see is the 3600X (and the 3600) vs the i5-9600K. Historically, AMD has been seen as a budget play, but I suspect the single-threaded performance of the Intel CPU will shore it up vs the higher threads of these AMD chips in many user-relevant workloads vs. benchmarks. (Heck, it probably is just as good as the 3900X in gaming due to GPUs not catching up to increases in effects and resolution, right?) There could be very interesting values at these price points. Could you imagine 3 years ago telling someone, "For a good budget gaming PC, the Intel chips are alright, but if you want something REALLY nice once your GPU budget is used, put another $80 toward an AMD CPU!" Amazing.
  • BloodyBunnySlippers - Sunday, July 7, 2019 - link

    I too am interested in the 3600x. I don't know if my math is relevant but the TDP per core count is higher for the 3600x than for the 3700x AND 3800x. Isn't it possible that it would then have a higher OC and better gaming performance? I mostly game, but moving from 4 threads on my current system to 12 threads isn't going to hurt either. Reply
  • allenb - Sunday, July 7, 2019 - link

    This is a great result for AMD. They’ve won some benchmarks, conceded some others to Intel, and embarrassed themselves nowhere. A few thoughts:

    1. Was worried about the memory latency graphs at the start, but it was a relief to see how effectively those regressions have been mitigated in actual workloads.

    2. In the 3D particle movement with AVX test, the 7920X jumps out to a *huge* lead. While everything else roughly doubles, it shows 9x speed up. Granted it is the only AVX512-capable cpu but it shouldn’t be *that* good unless the revised code is able to brilliantly utilize some of the new -512 features beyond just doubling vector length.

    3. I work in an industry that’s heavily focused on performance and relatively cost-insensitive. That’s traditionally meant Intel but suddenly we are looking very hard at AMD on the server side.

    4. Can’t wait to see what the 3950X can do!
  • Kevin G - Sunday, July 7, 2019 - link

    1) Yes, AMD seemingly knew what they were doing and addressed that concern in the end product. I'd be curious what impact reducing the L3 capacity would have on benchmarks to valid some of the theories (or better yet, AMD release a larger L3 cache chiplet).

    2) There are a couple of other instructions with AVX-512 that can make vector code more efficient beyond just doubling the vector width for a linear increase in performance. Also worth noting that leveraging AVX-512 comes at a clock speed hit so the gains per clock is even higher.

    3) While this is consumer side, the 12 core results are the most important to look like for Rome performance projection. That is the one that shares the IO die. (OTOH, a single CPU chiplet Rome that is allowed to clock to the moon with a 512 bit memory interface and 130 PCIe 4.0 lanes is an interesting single threaded proposal.)

    4) Agreed but I would predict a very small down tick in IPC. The 3900 has all the cache enabled with fewer cores competing for contention. Everything else looks to be upward.
  • just4U - Sunday, July 7, 2019 - link

    It's been a long time since I actually was chomping at the bit to see a review.. A Big day for Amd. Reply

Log in

Don't have an account? Sign up now