Intel Xeon E5 Version 3: Up to 18 Haswell EP Cores
by Johan De Gelas on September 8, 2014 12:30 PM ESTMemory Subsystem Bandwidth
Let's set the stage first and perform some meaningful low level benchmarks. First, we measured the memory bandwidth in Linux. The binary was compiled with the Open64 compiler 5.0 (Opencc). It is a multi-threaded, OpenMP based, 64-bit binary. The following compiler switches were used:
-Ofast -mp -ipa
The results are expressed in GB per second. Note that we also tested with gcc 4.8.1 and compiler options
-O3 –fopenmp –static
Results were consistently 20% to 30% lower with gcc, so we feel our choice of Open64 is appropriate. Everybody can reproduce our results (Open64 is freely available) and since the binary is capable of reaching higher speeds, it is easier to spot speed differences. First we compared our DDR4-2133 LRDIMMs with the Registered DDR4-2133 DIMMs on the Xeon E5-2695 v3 (14 cores at 2.3GHz, Turbo up to 3.6GHz).
Registered DIMMs are slightly faster at 1DPC, but LRDIMMs are clearly faster when you insert more than one DIMM per channel. We measured a 16% to 18% difference in performance. It's interesting to note that LRDIMMs are supposed to run at 1600 at 3DPC according to Intel's documentation, but our bandwidth measurement points to 1866. The command "dmidecode -type 17" that reads out the BIOS confirmed this.
Next, we compared the different Xeon platforms.
The new Xeon E5-2600 v3 has access to 15-21% more bandwidth than the E5-2600 v2, which uses DDR3-1866, and almost 50% more than the first Xeon E5s (DDR3-1600). Interestingly, the previous generation Xeons and the Xeon E5-2667 v3 need to use one thread per logical thread to use the full potential of the memory controller. The reason that the Xeon E5-2667 v3 shows similar behavior as the previous Xeons is that it is also a die with one dual ring and one memory controller. Also, 16 threads (one per physical core) is probably not enough to get the full potential of a quad channel DDR4-2133 memory subsystem. The new High Core Count (HCC, 14-18 core) Xeon E5 chips perform better with one thread per physical processor.
Although it makes sense that a CPU needs a certain number of threads to get its memory controller working at full speed, it's still interesting to note that the previous 12-core Xeon E5-2697 v2 can only offer 41GB/s at 24 threads while the 14-core Xeon E5-2695 v3 is already delivering more than twice as much bandwidth at 28 threads. Of course, those kind of bandwidth numbers only matter for specific HPC benchmarks as the L3 cache (30-45MB L3) will take care of most of the requests. Latency however always matters.
Facebook
Twitter
RSS
85 Comments
View All Comments
bsd228 - Friday, September 12, 2014 - link
Now go price memory for M class Sun servers...even small upgrades are 5 figures and going 4 years back, a mid sized M4000 type server was going to cost you around 100k with moderate amounts of memory.And take up a large portion of the rack. Whereas you can stick two of these 18 core guys in a 1U server and have 10 of them (180 cores) for around the same sort of money.
Big iron still has its place, but the economics will always be lousy.
platinumjsi - Tuesday, September 9, 2014 - link
ASRock are selling boards with DDR3 support, any idea how that works?http://www.asrockrack.com/general/productdetail.as...
TiGr1982 - Tuesday, September 9, 2014 - link
Well... ASRock is generally famous "marrying" different gen hardware.But here, since this is about DDR RAM, governed by the CPU itself (because memory controller is inside the CPU), then my only guess is Xeon E5 v3 may have dual-mode memory controller (supporting either DDR4 or DDR3), similarly as Phenom II had back in 2009-2011, which supported either DDR2 or DDR3, depending on where you plugged it in.
If so, then probably just the performance of E5 v3 with DDR3 may be somewhat inferior in comparison with DDR4.
alpha754293 - Tuesday, September 9, 2014 - link
No LS-DYNA runs? And yes, for HPC applications, you actually CAN have too many cores (because you can't keep the working cores pegged with work/something to do, so you end up with a lot of data migration between cores, which is bad, since moving data means that you're not doing any useful work ON the data).And how you decompose the domain (for both LS-DYNA and CFD makes a HUGE difference on total runtime performance).
JohanAnandtech - Tuesday, September 9, 2014 - link
No, I hope to get that one done in the more Windows/ESXi oriented review.Klimax - Tuesday, September 9, 2014 - link
Nice review. Next stop: Windows Server. (And MS-SQL..)JohanAnandtech - Tuesday, September 9, 2014 - link
Agreed. PCIe Flash and SQL server look like a nice combination to test this new Xeons.TiGr1982 - Tuesday, September 9, 2014 - link
Xeon 5500 series (Nehalem-EP): up to 4 cores (45 nm)Xeon 5600 series (Westmere-EP): up to 6 cores (32 nm)
Xeon E5 v1 (Sandy Bridge-EP): up to 8 cores (32 nm)
Xeon E5 v2 (Ivy Bridge-EP): up to 12 cores (22 nm)
Xeon E5 v3 (Haswell-EP): up to 18 cores (22 nm)
So, in this progression, core count increases by 50% (1.5 times) almost each generation.
So, what's gonna be next:
Xeon E5 v4 (Broadwell-EP): up to 27 cores (14 nm) ?
Maybe four rows with 5 cores and one row with 7 cores (4 x 5 + 7 = 27) ?
wallysb01 - Wednesday, September 10, 2014 - link
My money is on 24 cores.SuperVeloce - Tuesday, September 9, 2014 - link
What's the story with 2637v3? Only 4 cores and the same freqency and $1k price as 6core 2637v2? By far the most pointless cpu on the list.