The Intel Xeon E5 v4 Review: Testing Broadwell-EP With Demanding Server Workloads
by Johan De Gelas on March 31, 2016 12:30 PM EST- Posted in
- CPUs
- Intel
- Xeon
- Enterprise
- Enterprise CPUs
- Broadwell
Single Core Integer Performance With SPEC CPU2006
In past server reviews, I used LZMA (7-zip) compression and decompression to evaluate single threaded performance. But I was well aware that while it was a decent integer test, it also gave a very myopic view in the process. After noticing that my colleagues used SPEC CPU2006, and after discussing the matter with several people, I realized that running SPEC CPU2006 was a much better way to evaluate single core performance. Even though SPEC CPU2006 is more HPC and workstation oriented, it contains a good variety of integer workloads.
I also wanted to keep the settings as "normal" as possible. So I used:
- 64 bit gcc : most used compiler on linux, good all round compiler that does not try to "break" benchmarks (libquantum...)
- gcc version 4.8.4: 4.8.x has been around for a long time, very mature version
- -O2 -fno-strict-aliasing: standard compiler settings that many developers use
- Run 2 copies and bind them to the first core
The ultimate objective is to measure performance in non-"aggressively optimized" applications where for some reason - as is frequently the case - a "multi thread unfriendly" task keeps us waiting. As we want to be able to compare these numbers to other architectures such as the IBM POWER 8, we decided to use all threads available on a single core. In case of Intel, this means one physical and two simultaneous threads running on top of it.
We included the Opteron 6376 for nostalgic reasons. We are showing the results of 2 threads running on top of one module with 2 "integer cores".
| Subtest | Xeon E5-2690 | Opteron 6376 | Xeon E5-2697v2 | Xeon E5-2667 v3 | Xeon E5-2699 v3 | Xeon E5-2699 v4 |
| 400.perlbench | 41.1 | 29.3 | 37.6 | 42.6 | 39.9 | 36.6 |
| 401.bzip2 | 33.4 | 24.1 | 30.1 | 33.1 | 29.9 | 25.3 |
| 403.gcc | 40.2 | 26.7 | 38.9 | 42.4 | 36.4 | 33.3 |
| 429.mcf | 45.1 | 31.7 | 46.8 | 46.4 | 41.6 | 43.9 |
| 445.gobmk | 36.4 | 25.5 | 33.2 | 34.9 | 31.7 | 27.7 |
| 456.hmmer | 30.4 | 26.1 | 27.6 | 31 | 27.1 | 28.4 |
| 458.sjeng | 35.2 | 24.7 | 32.8 | 35.2 | 30.5 | 28.3 |
| 462.libquantum | 74.9 | 39.9 | 79.3 | 84.4 | 62.2 | 67.3 |
| 464.h264ref | 51.7 | 34.2 | 48.1 | 52.1 | 45.2 | 40.7 |
| 471.omnetpp | 24.5 | 25.3 | 26.8 | 29.4 | 26.6 | 29.9 |
| 473.astar | 28.2 | 20.7 | 26.1 | 27.9 | 24 | 23.6 |
| 483.xalancbmk | 41.5 | 28.2 | 41.4 | 48.2 | 42.4 | 41.8 |
Unless you are used to seeing these numbers, this does not tell you too much. As Sandy Bridge EP (Xeon E5 v1) is about 4 years old, the servers based upon this CPU are going to get replaced by newer ones. So Sandy Bridge is our reference, and Sandy Bridge performance is considered to be 100%.
| Subtest | Application type | Xeon E5-2690 | Opteron 6376 | Xeon E5-2697v2 | Xeon E5-2667 v3 | Xeon E5-2699 v3 | Xeon E5-2699 v4 |
| 400.perlbench | Spam filter | 100% | 71% | 91% | 104% | 97% | 89% |
| 401.bzip2 | Compression | 100% | 72% | 90% | 99% | 90% | 76% |
| 403.gcc | Compiling | 100% | 66% | 97% | 105% | 91% | 83% |
| 429.mcf | Vehicle scheduling | 100% | 70% | 104% | 103% | 92% | 97% |
| 445.gobmk | Game AI | 100% | 70% | 91% | 96% | 87% | 76% |
| 456.hmmer | Protein seq. analyses | 100% | 86% | 91% | 102% | 89% | 93% |
| 458.sjeng | Chess | 100% | 70% | 93% | 100% | 87% | 80% |
| 462.libquantum | Quantum sim | 100% | 53% | 106% | 113% | 83% | 90% |
| 464.h264ref | Video encoding | 100% | 66% | 93% | 101% | 87% | 79% |
| 471.omnetpp | Network sim | 100% | 103% | 109% | 120% | 110% | 122% |
| 473.astar | Pathfinding | 100% | 73% | 93% | 99% | 85% | 84% |
| 483.xalancbmk | XML processing | 100% | 68% | 100% | 116% | 102% | 101% |
Many smart people have spent weeks - if not months - on SPEC CPU2006 analysis, so we will not pretend we can offer you a complete picture in a few days. If you want a detailed analysis of compilers and CPU 2006, I recommend the very detailed article of SPEC CPU 2006 meister Andreas Stiller in the February issue of C'T (German computer magazine).
We need much more profiling data than we could gather in the past weeks. But for what we can do, we'll start with the most important parameter: clockspeed.
One of the most important things to realize is that - especially with badly threaded workloads - these massive multi-core CPUs almost never work at their advertised clockspeed.
- The Xeon E5-2690 can run at 3.3 GHz with all cores busy, and is capable of boosting up to 3.8 GHz
- The Xeon E5-2697 v2 can run at 3 GHz with all cores busy, and is capable of boosting up to 3.5 GHz
- The Xeon E5-2699 v3 can run at 2.8 GHz with all cores busy, and is capable of boosting up to 3.6 GHz
- The Xeon E5-2667 v3 3.2 GHz is a specialized high frequency model. It can run at 3.4 GHz with all cores busy, and is capable of boosting up to 3.6 GHz
- The Xeon E5-2699 v4 can run at 2.8 GHz with all cores busy, and is capable of boosting up to 3.6 GHz
So that already explains a lot. In contrast to the many benchmark applications, SPEC CPU2006 runs for a long time (5 to 15 minutes per test), and our first impression is that the HCC parts are not able to keep all of their cores at their maximum turbo boost. Otherwise there is no reason why a Xeon E5-2699 v3 or v4 would perform worse than a Xeon E5-2667 v3: both can run at 3.6 GHz when one core is active.
The low IPC, memory intensive network simulator omnetppp seems to be the only test that runs significantly better on the newer cores (Haswell, Broadwell) compared to Sandy Bridge. That also seems to be the only benchmark where the high core count chips (E5-2699 v4, E5-2699 v3) continue to outperform Sandy Bridge. We could pinpoint the reason by testing with different memory speeds and channels. The E5-2699 v4 can offer the highest performance thanks to the larger L3-cache (55 MB) and the higher DIMM speed (DDR4-2400) compared to Sandy Bridge (20 MB, DDR3-1600). Otherwise when we keep the clockspeed more or less constant, by looking at the Xeon E5-2667v3 and the Xeon E5-2690, we get a 1-5% speed difference, and only the memory intensive subtests (omnetpp, Libquantum) and xalancbmk (low IPC, branch intensive) show higher improvements.
Once we test both top SKUs with "-Ofast" (a more aggressive compiler setting), the results change quite a bit:.
| Subtest | Application type | Xeon E5-2699 v4 vs Xeon E5-2690 (-Ofast) | Xeon E5-2699 v4 vs Xeon E5-2690 (-O2) |
| 400.perlbench | Spam filter | 111% | 89% |
| 401.bzip2 | Compression | 94% | 76% |
| 403.gcc | Compiling | 95% | 83% |
| 429.mcf | Vehicle scheduling | 114% | 97% |
| 445.gobmk | Game AI | 90% | 76% |
| 456.hmmer | Protein seq. analyses | 106% | 93% |
| 458.sjeng | Chess | 93% | 80% |
| 462.libquantum | Quantum sim | 101% | 90% |
| 464.h264ref | Video encoding | 89% | 79% |
| 471.omnetpp | Network sim | 132% | 122% |
| 473.astar | Pathfinding | 98% | 84% |
| 483.xalancbmk | XML processing | 105% | 101% |
Switching from -O2 to -Ofast improves Broadwell-EP's absolute performance by over 19%. Meanwhile the relative performance advantage versus the Xeon E5-2690 averages 3%. As a result, the clockspeed disadvantage of the latest Xeon is negated by the increase in IPC. Clearly the latest generation of Xeons benefit more from aggressive optimizations than the previous ones. That is unsurprising of course, but it is interesting that the newest Xeons need more optimization to "hold the line" in single core performance.
So far we can conclude that if you were to upgrade from a Xeon E5-2xxx v1 to a similar v4 model, your single threaded integer code will not run faster without recompiling and optimizing. The process improvements have been used mostly to add more cores in the same power envelope, while at same time Intel also traded a few speed bins in to add even more cores in the top models. As a result single core integer performance basically holds the line, nothing more. The only exception are memory intensive applications who benefit from every growing L3-cache and the faster DRAM technology.
Facebook
Twitter
RSS
112 Comments
View All Comments
xrror - Tuesday, April 5, 2016 - link
Even at 3.3Ghz though, they shouldn't be that slow. I'm taking a guess - if this was a student lab, and they bothered to specifically order xeon (or opteron back in the day) workstations - I'm guessing this was a CAD/CAM lab or something running a boatload of expensive licenced software (like, autodesk, solidworks, etc) and some of that stuff is horrible at thrashing on the hard drive, constantly.And I doubt your school could spring the cash for SSD drives in them (because Workstation SKU == you pay dearly OEM workstation 'certified' drive cost).
This is all guesses though. And not trying to defend - it does suck when you have what should be a sweet machine choking for whatever reason, and you're there trying to get your assignments done and you just want to smash the screen cause it just chhhuuuuuuggggsss... ;p
SkipPerk - Friday, April 8, 2016 - link
I have seen this many times, even in the for-profit sector. I once saw a compute cluster that was choking on server with slow storage. They had a 10 gb network and fast Xeon machines running on flash, but the primary storage was too slow. When they get a proper SAN it was an order of magnitude improvement.Back in the day storage was often the bottleneck, but it still comes up today.
someonesomewherelse - Thursday, September 1, 2016 - link
We ran everything in virtual machines with the actual disk images not stored locally.... and the lans in the classrooms were 100mbit, idk about the connection from the classroom to the server with the image. How's that for slow?I would have loved it if our stuff was as 'slow' as yours. The wifi in the classrooms was very fast too..... especially since I doubt anyone bothered with turning of their torrents (well I mean it's completely understandable, you are going to watch the new episode of your favorite show once you are back home and not everyone had (well has, but most people can get it now) fth with at least 100Mbit line (ideally symmetrical, but some isps are too gready with ul speeds so 300/50 is cheaper than 100/100...... and good luck getting 1000/1000 on a residential package (the hw isn't the problem since you can get 1000/1000 with a commercial (aka over priced) package..... using the same hw... basically I would just need to sign a new contract, send it back, and enjoy the faster line in 1 business day or less)...well at least there are no bw caps (if I didn't read foreign boards bw caps on non mobile connections would be something I'd think no isp could do and not lose all customers) and there's we have no dmca (or something similar) and afaik no plans for one either (if they tried to pass such a law I can imagine that you'd have enough support for a referendum which you would win with a huge mayority), even better, the methods used to catch people downloading torrents are illegal anyway so any evidence obtained with them or derived from them is inadmissible anyway and just by presenting it you have admitted to several crimes which the police and prosecution are obliged to investigate/prosecute.... copyright infringment however is a civil matter).
donwilde1 - Tuesday, April 5, 2016 - link
One of the more interesting Intel features, in my opinion, is that Broadwell carries an on-board encryption engine with its own interpreter similar to a small-memory, embedded JVM. This enables full Trusted Boot capability, which I view as a necessity in today's hackable world. Would you consider a follow-on article on this? The project was a clean-room development called BeiHai, done in China.JamesAnthony - Wednesday, April 6, 2016 - link
From what I can tell in looking over the benchmarks, there is not much of an increase in performance at all in core vs core performance speeds going from the V1 CPUs to the V4 CPUsAs if you look at the benchmarks, and calculate that you are comparing 16 cores to 44 cores, the 44 core setup is not 2.75x faster.
So while your overall speed goes up, your work accomplished per core is not increasing at the same rate.
Why does this matter? Well thanks to software licensing costs, as you add cores it gets very expensive quickly. So if your software costs (which can easily exceed the hardware costs very quickly) go up with each core you add, but the work done does not, you quickly wind up in a negative cost / performance ratio.
For quite a few people the E5-2667 v2 CPU with 8 cores at 3.5 GHz (Turbo 4) comes out around the best value for the software licensing cost.
So while Intel puts out processors that overall can do more work than the previous ones, the move to per core software licensing is making it a negative value proposition. This is why people keep wanting higher clock speed lower core count processors, but we seem stuck around 3.5 GHz for many years.
SkipPerk - Friday, April 8, 2016 - link
Although you are right for workstations, so much demand is for generic virtualized machines. Many buyers are fine with 2 ghz with as many cores as they can get. They load as little RAM as the spec requires and throw out the cheapest single core, dual thread 2 GB RAM VM they can. This is how call centers work, not to mention many low-level office jobs. They do not care about performance because this is more than enough.I have had specialty applications where prosumer 6-core or 8-core CPUs were the better deal (usually liquid cooled and overclocked), but not many buyers are licensing insanely expensive analytical software by the core.
SeanJ76 - Sunday, April 10, 2016 - link
@Xeon chips!! TOTAL GARBAGE!legolasyiu - Wednesday, April 20, 2016 - link
The ASUS Workstation/Server board with V4 boards are very stable and they have 10% OC. I am very interested how the processor with those boards.Bulat Ziganshin - Saturday, May 7, 2016 - link
>This increases AES (symmetric) encryption performance by 20-25%PCLMULQDQ implements part of Galois Field multiplication and bdw actually improved only GCM part of AES-GCM algo. neither AES nor other popular symmetric encryption algos became faster
oceanwave1000 - Monday, May 9, 2016 - link
This article mentioned that the Broadwell EP e5-v4 family has 3 die configurations. I got the 306mm2 and 454mm2. Did anyone catch the third one?Thanks.