ABOUT BENCH FORUMS PODCAST LOGIN REGISTER
PC Components▼
Smartphones & tablets▼
Systems▼
ENTERPRISE & IT
GUIDES▼
DEALS

The Intel Xeon E5 v4 Review: Testing Broadwell-EP With Demanding Server Workloads

 by Johan De Gelas on March 31, 2016 12:30 PM EST
112 Comments | Add A Comment
112 Comments

Single Core Integer Performance With SPEC CPU2006

In past server reviews, I used LZMA (7-zip) compression and decompression to evaluate single threaded performance. But I was well aware that while it was a decent integer test, it also gave a very myopic view in the process. After noticing that my colleagues used SPEC CPU2006, and after discussing the matter with several people, I realized that running SPEC CPU2006 was a much better way to evaluate single core performance. Even though SPEC CPU2006 is more HPC and workstation oriented, it contains a good variety of integer workloads.

I also wanted to keep the settings as "normal" as possible. So I used:

  • 64 bit gcc : most used compiler on linux, good all round compiler that does not try to "break" benchmarks (libquantum...)
  • gcc version 4.8.4: 4.8.x has been around for a long time, very mature version
  • -O2 -fno-strict-aliasing: standard compiler settings that many developers use
  • Run 2 copies and bind them to the first core

The ultimate objective is to measure performance in non-"aggressively optimized" applications where for some reason - as is frequently the case - a "multi thread unfriendly" task keeps us waiting. As we want to be able to compare these numbers to other architectures such as the IBM POWER 8, we decided to use all threads available on a single core. In case of Intel, this means one physical and two simultaneous threads running on top of it.

We included the Opteron 6376 for nostalgic reasons. We are showing the results of 2 threads running on top of one module with 2 "integer cores".

Subtest Xeon E5-2690 Opteron 6376 Xeon E5-2697v2 Xeon E5-2667 v3 Xeon E5-2699 v3 Xeon E5-2699 v4
400.perlbench 41.1 29.3 37.6 42.6 39.9 36.6
401.bzip2 33.4 24.1 30.1 33.1 29.9 25.3
403.gcc 40.2 26.7 38.9 42.4 36.4 33.3
429.mcf 45.1 31.7 46.8 46.4 41.6 43.9
445.gobmk 36.4 25.5 33.2 34.9 31.7 27.7
456.hmmer 30.4 26.1 27.6 31 27.1 28.4
458.sjeng 35.2 24.7 32.8 35.2 30.5 28.3
462.libquantum 74.9 39.9 79.3 84.4 62.2 67.3
464.h264ref 51.7 34.2 48.1 52.1 45.2 40.7
471.omnetpp 24.5 25.3 26.8 29.4 26.6 29.9
473.astar 28.2 20.7 26.1 27.9 24 23.6
483.xalancbmk 41.5 28.2 41.4 48.2 42.4 41.8

Unless you are used to seeing these numbers, this does not tell you too much. As Sandy Bridge EP (Xeon E5 v1) is about 4 years old, the servers based upon this CPU are going to get replaced by newer ones. So Sandy Bridge is our reference, and Sandy Bridge performance is considered to be 100%.

Subtest Application type Xeon E5-2690 Opteron 6376 Xeon E5-2697v2 Xeon E5-2667 v3 Xeon E5-2699 v3 Xeon E5-2699 v4
400.perlbench Spam filter 100% 71% 91% 104% 97% 89%
401.bzip2 Compression 100% 72% 90% 99% 90% 76%
403.gcc Compiling 100% 66% 97% 105% 91% 83%
429.mcf Vehicle scheduling 100% 70% 104% 103% 92% 97%
445.gobmk Game AI 100% 70% 91% 96% 87% 76%
456.hmmer Protein seq. analyses 100% 86% 91% 102% 89% 93%
458.sjeng Chess 100% 70% 93% 100% 87% 80%
462.libquantum Quantum sim 100% 53% 106% 113% 83% 90%
464.h264ref Video encoding 100% 66% 93% 101% 87% 79%
471.omnetpp Network sim 100% 103% 109% 120% 110% 122%
473.astar Pathfinding 100% 73% 93% 99% 85% 84%
483.xalancbmk XML processing 100% 68% 100% 116% 102% 101%

Many smart people have spent weeks - if not months - on SPEC CPU2006 analysis, so we will not pretend we can offer you a complete picture in a few days. If you want a detailed analysis of compilers and CPU 2006, I recommend the very detailed article of SPEC CPU 2006 meister Andreas Stiller in the February issue of C'T (German computer magazine). 

We need much more profiling data than we could gather in the past weeks. But for what we can do, we'll start with the most important parameter: clockspeed.

One of the most important things to realize is that - especially with badly threaded workloads - these massive multi-core CPUs almost never work at their advertised clockspeed.

  • The Xeon E5-2690 can run at 3.3 GHz with all cores busy, and is capable of boosting up to 3.8 GHz
  • The Xeon E5-2697 v2 can run at 3 GHz with all cores busy, and is capable of boosting up to 3.5 GHz
  • The Xeon E5-2699 v3 can run at 2.8 GHz with all cores busy, and is capable of boosting up to 3.6 GHz
  • The Xeon E5-2667 v3 3.2 GHz is a specialized high frequency model. It can run at 3.4 GHz with all cores busy, and is capable of boosting up to 3.6 GHz
  • The Xeon E5-2699 v4 can run at 2.8 GHz with all cores busy, and is capable of boosting up to 3.6 GHz

So that already explains a lot. In contrast to the many benchmark applications, SPEC CPU2006 runs for a long time (5 to 15 minutes per test), and our first impression is that the HCC parts are not able to keep all of their cores at their maximum turbo boost. Otherwise there is no reason why a Xeon E5-2699 v3 or v4 would perform worse than a Xeon E5-2667 v3: both can run at 3.6 GHz when one core is active.

The low IPC, memory intensive network simulator omnetppp seems to be the only test that runs significantly better on the newer cores (Haswell, Broadwell) compared to Sandy Bridge. That also seems to be the only benchmark where the high core count chips (E5-2699 v4, E5-2699 v3) continue to outperform Sandy Bridge. We could pinpoint the reason by testing with different memory speeds and channels. The E5-2699 v4 can offer the highest performance thanks to the larger L3-cache (55 MB) and the higher DIMM speed (DDR4-2400) compared to Sandy Bridge (20 MB, DDR3-1600). Otherwise when we keep the clockspeed more or less constant, by looking at the Xeon E5-2667v3 and the Xeon E5-2690, we get a 1-5% speed difference, and only the memory intensive subtests (omnetpp, Libquantum) and xalancbmk (low IPC, branch intensive) show higher improvements.

Once we test both top SKUs with "-Ofast" (a more aggressive compiler setting), the results change quite a bit:.

Subtest Application type Xeon E5-2699 v4 vs Xeon E5-2690 (-Ofast) Xeon E5-2699 v4 vs Xeon E5-2690 (-O2)
400.perlbench Spam filter 111% 89%
401.bzip2 Compression 94% 76%
403.gcc Compiling 95% 83%
429.mcf Vehicle scheduling 114% 97%
445.gobmk Game AI 90% 76%
456.hmmer Protein seq. analyses 106% 93%
458.sjeng Chess 93% 80%
462.libquantum Quantum sim 101% 90%
464.h264ref Video encoding 89% 79%
471.omnetpp Network sim 132% 122%
473.astar Pathfinding 98% 84%
483.xalancbmk XML processing 105% 101%

Switching from -O2 to -Ofast improves Broadwell-EP's absolute performance by over 19%. Meanwhile the relative performance advantage versus the Xeon E5-2690 averages 3%. As a result, the clockspeed disadvantage of the latest Xeon is negated by the increase in IPC. Clearly the latest generation of Xeons benefit more from aggressive optimizations than the previous ones. That is unsurprising of course, but it is interesting that the newest Xeons need more optimization to "hold the line" in single core performance.

So far we can conclude that if you were to upgrade from a Xeon E5-2xxx v1 to a similar v4 model, your single threaded integer code will not run faster without recompiling and optimizing. The process improvements have been used mostly to add more cores in the same power envelope, while at same time Intel also traded a few speed bins in to add even more cores in the top models. As a result single core integer performance basically holds the line, nothing more. The only exception are memory intensive applications who benefit from every growing L3-cache and the faster DRAM technology.

Benchmark Configuration and Methodology Memory Subsystem
PRINT THIS ARTICLE
Comments Locked

112 Comments

View All Comments

  • PowerOfFacts - Thursday, June 23, 2016 - link

    And now Oracle marketing speaks. Their HammerDB results are bogus. Oracle continues to site socket results when the majority of the world has moved on to per core results. They cite the results from a 32 core HammerDB then compare it to a 1 chip (1/2 of 1 socket) POWER8 because Phil has a hard-on for how "HE" believes IBM has packaged the processor and similarly chooses an Intel configuration to ensure "THEY" get the result they want. Phil & Oracle (appear) to always speak with forked tongue.

  • patrickjp93 - Sunday, April 3, 2016 - link

    "Best" only at specific scale-up workloads. There's a reason Sparc is not particularly popular for clusters and supercomputing (and it's NOT software compatibility). It sucks at a lot of workloads when compared to x86. As for the SAP benchmarks, that's to be expected since x86 doesn't yet support transactional memories. That changes with Skylake Purley though.

  • Brutalizer - Wednesday, April 6, 2016 - link

    In these 25ish benchmarks, the SPARC M7 is 2-3x faster on all kinds of workloads, not just some specific scale up workloads. The reason SPARC M7 is not popular for clusters (supercomputers are clusters) is not because of low raw compute performance, it is because of cost and wattage. The M7 is much more expensive than x86, and draws much more power. I guess somewhere 250 watt or so? M7 are in big enterprise servers, some have water cooling, etc. Whereas clusters have many cheap nodes, with no water cooling.

    Clusters can have x86 because the highest wattage x86 cpu, uses 140 watt or so. Not more. So it would be feasible to use 140 watt cpus in clusters. But not 250 watt cpus, they draw too much power.

    For instance, the IBM Blue Gene supercomputer that hold spot nr 5 in top500 for a couple of years, used 850 MHz powerpc cpus, when everyone else used 2.4 GHz x86 or so. The 850 MHz cpu dont use lot of power, so that is the reason it was used in Blue Gene, not because it was faster (it wasnt). A large supercomputer can draw 10 MegaWatt, and that costs very much. Power is a huge issue in super computers. SPARC M7 draws too much power to be useful in a large cluster, and costs too much.

    If we talk about raw compute power for SPARC M7, it reaches 1200 SPECint2006, whereas E5-2699v3 reaches 715 SPECint2006. Not really 2-3x faster, but still much faster.
    In SPECfp2006, the M7 reaches 832, whereas the E5-2699v3 reach 474.
    https://blogs.oracle.com/BestPerf/entry/201510_spe...

    So, as you can see yourself, the SPARC M7 is faster on scale-up business workloads (it was designed for that type of workloads) and also faster on raw compute power. And faster in everything in between. Just look at the wide diversity among these 25 ish benchmarks.

  • Brutalizer - Wednesday, April 6, 2016 - link

    BTW, do you really expect a 150 watt x86 cpu, to outperform a 250 watt SPARC M7 cpu? Have you seen benchmarks where they compare 250 watt graphics card vs a 150 watt graphics card? Which GPU do you think is faster? Do you expect a 150 watt GPU to outperform a 250 watt gpu?

    The SPARC M7 has 50% more cores, twice the cpu cache, twice the GHz, twice the Wattage, twice the RAM bandwidth, twice the nr of transistors (10 billions) - and you are surprised it is 2-3x faster than x86?

    BTW, the SPARC M7 has stronger cores than x86. If you look at all these benchmarks, typically one M7 with 32 cores, is faster than two E5-2699v3 with 2x18 = 36 cores. This must mean that one SPARC M7 core, packs more punch than a E5-2699v3 core, because 32 SPARC cores are faster than 36 x86 cores in all benchmarks.

  • adamod - Friday, June 3, 2016 - link

    i know this is an old post but i am confused (this isnt something i have learned much about yet) i am hoping you can help some...if the sparc has 2 to 3x performance and is 250w compared to 140w then wouldnt that make it MORE efficient? and if you need two 2699's to compare to a sparc m7 then wouldnt that be 280w, more than the 250w of the xeons? i realize there are other factors here but this doesnt make sense to me. also yea there are graphics cards that are a lower wattage and perform better...i am an AMD fan but nvidia has had some faster cards with better performance in the past...i have an R9 280X, a mid grade card rated at i believe 225w, kinda crazy when it can get beaten by 17w nvidia cards

  • tqth - Sunday, April 3, 2016 - link

    The SPARC and POWER servers are for people with unlimited pocket where compactness and reliability worth the premium it's spent on. If you have to ask how much it costs, you'd probably can't afford it.
    Xeons are commodity hardware where you could purchase the best bang for your buck.
    They are not aiming at the same market. Most software wouldn't even work on both system.
    Besides, benchmarks are worthless - unless the performance of the specific software is tested. And that's rare.

  • PowerOfFacts - Thursday, June 23, 2016 - link

    Depends on which Xeon processors you are referring to. The latest Broadwell EP & EX chips can cost over $7K each. Well on par if not exceeding POWER8 chips and definitely more than OpenPOWER chips. Times are changing. Intel has milked their clients for a long time feeding them the marketing line of open, commodity & low cost. They are no longer open buying up ecosystem integrating into the silicone, what exactly does commodity mean anyway and as low cost goes ... as I just said, pretty salty.

  • yuhong - Thursday, March 31, 2016 - link

    64GB LR-DIMMs will probably not come out at reasonable prices until 8Gbit DDR4 is more mainstream.

  • iwod - Thursday, March 31, 2016 - link

    I thought Samsung announced a 128GB DIMM with some type of 3D / TSV RAM.

  • Casper42 - Thursday, March 31, 2016 - link

    Not shipping just yet though.
    Should be sometime this year though.
Copyright © 2024. All rights reserved.
BENCH
TOPICS
FOLLOW
ABOUT

Log in

Don't have an account? Sign up now