The Intel Xeon E7 v2 Review: Quad Socket, Up to 60 Cores/120 Threads
by Johan De Gelas on February 21, 2014 6:00 AM EST- Posted in
- IT Computing
- Intel
- Xeon
- Ivy Bridge EX
- server
- Brickland
Our Benchmark Choices
To make the comparison more interesting, we decided to include both the Quad Xeon "Westmere-EX" as well as the "Nehalem-EX". Remember these heavy duty, high RAS servers continue to be used for much longer in the data center than their dual socket counterparts. Many people considering the newest Xeon E7-4800 v2 probably still own a Xeon X7500.
Of course, the comparison would not be complete without the latest dual Xeon 2600 v2 server and at least one Opteron based server. Due to the large number of platforms and the fact that we developed a brand new HPC test (see further), we quickly ran out of time. These time constrains and the fact that we have neglected our Linux testing in recent reviews in favor of Windows 2012 and ESXi led to the decision to limit ourself to testing on top of Ubuntu Linux 13.10 (kernel 3.11). You'll see our typical ESXi and Windows benchmarks in a later review.
Benchmark Configuration
There are some differences in the RAM and SSD configurations. The use of different SSDs was due to time constraints as we wanted to test the servers as much as possible in parallel. The RAM configuration differences are a result of the platforms: for example, the quad Intel CPUs only perform at their best when each CPU gets eight DIMMs. The Opteron and Dual Xeon E5-2680 v2 server perform best with one DIMM per channel (1 DPC).
None of these differences have a tangible influence on the results of our benchmarks, as none of them were bottlenecked by the storage system or the amount of RAM that was used. The minimum amount of 64GB of RAM was more than enough for all benchmarks in this review.
We also did not attempt to do power measurements. We will try to do an apples-to-apples power comparison at a later time.
Intel S4TR1SY3Q "Brickland" IVT-EX 4U-server
The latest and greatest from Intel consists of the following components:
| CPU |
4x Xeon E7-4890 v2 (D1 stepping) 2.8GHz 15 cores, 37.5MB L3, 155W TDP |
| RAM |
256GB, 32x8GB Samsung 8GB DDR3 M393B1K70DH0-YK0 at 1333MHz |
| Motherboard | Intel CRB Baseboard "Thunder Ridge" |
| Chipset | Intel C602J |
| PSU | 2x1200W (2+0) |
Total amount of DIMM slots is 96. When using 64GB LRDIMMs, this server can offer up to 6TB of RAM! In some cases, we have tested the E7-4890 v2 at a lower maximum clock in order to do clock-for-clock comparisons with the previous generation, and in a few cases we have also disabled three of the cores in order to simulate performance of some of the 12-core Ivy Bridge EX parts. For example, a E7-4890 v2 at 2.8 GHz with 3 cores disabled (12 cores total) gives you a good idea how the much less expensive E7- 8857 v2 at 3 GHz would perform: it would perform about 7% higher than the 12-core E7-4890 v2.
Intel Quanta QSCC-4R Benchmark Configuration
The previous quad Xeon E7 server, as reviewed here.
| CPU |
4x Xeon X7560 at 2.26GHz or 4x Xeon E7-4870 at 2.4GHz |
| RAM |
16x8GB Samsung 8GB DDR3 M393B1K70DH0-YK0 at 1066MHz |
| Motherboard | QCI QSSC-S4R 31S4RMB00B0 |
| Chipset | Intel 7500 |
| BIOS version | QSSC-S4R.QCI.01.00.S012,031420111618 |
| PSU | 4x850W Delta DPS-850FB A S3F E62433-004 850W |
The server can accept up to 64 32GB Load Reduced DIMMs (LR-DIMMs) or 2TB.
Intel's Xeon E5 server R2208GZ4GSSPP (2U Chassis)
This is the server we used in our Xeon "Ivy bridge EP" review.
| CPU | 2x Xeon processor E5-2680 (2.8GHz, 10c, 25MB L3, 115W) |
| RAM |
128GB (8 x 16GB) Micron MT36JSF2G72PZ – BDDR3-1866 |
| Internal Disks | 2 x Intel MLC SSD710 200GB |
| Motherboard | Intel Server Board S2600GZ "Grizzly Pass" |
| Chipset | Intel C600 |
| BIOS version | SE5C600.86B (August the 6th, 2013) |
| PSU | Intel 750W DPS-750XB A (80+ Platinum) |
The Xeon E5 CPUs have four memory channels per CPU and support up to DDR3-1866, and thus our dual CPU configuration gets eight DIMMs for maximum bandwidth.
Supermicro A+ Opteron server 1022G-URG (1U Chassis)
This Opteron server is not comparable in any way with the featured Intel systems as it is not targeted at the same market and costs a fraction of the other machines. Nevertheless, here's our test configuration.
| CPU | 2x Opteron "Abu Dhabi" 6376 at 2.3GHz |
| RAM | 64GB (8x8GB) DDR3-1600 Samsung M393B1K70DH0-CK0 |
| Motherboard | SuperMicro H8DGU-F |
| Internal Disks | 2 x Intel MLC SSD710 200GB |
| Chipset | AMD Chipset SR5670 + SP5100 |
| BIOS version | R3.5 |
| PSU | SuperMicro PWS-704P-1R 750Watt |
The Opteron server in this review is only here to satisfy curiosity. We want to see how well the Opteron fares in our new Linux benchmarks.
Facebook
Twitter
RSS
125 Comments
View All Comments
Kevin G - Monday, February 24, 2014 - link
Even with Itanium's poor performnace, it doesn't stop you from citing the Big Tux experiment to slander overall Linux performance.Brutalizer - Tuesday, February 25, 2014 - link
The reason I cite Big Tux, is because that is the only benchmarks I have seen for Linux running on 64 sockets. If you have other benchmarks, please link to them so I can stop refer to Big Tux.I have never attributed Linux bad performance on Big Tux, because the Itanium has poor performance. I attribute Linux bad performance on Big Tux, because of this: Linux had ~40% cpu utilization on 64 socket Big Tux Itanium server. This means every other cpu idles under full load when using Linux. Is this bad or not? This has nothing to do with Itanium. If Linux ran 64 socket SPARC or POWER - it would still idle ~40%.
Thus, my conclusion of Linux bad performance, is because of the low cpu utilization. It has nothing to do with how fast or slow the hardware. Instead, how good does Linux utilize all resources on large servers? Answer: very bad.
Talking about slandering Linux, have you read this from a prominent Linux kernel developer?
http://vger.kernel.org/~davem/cgi-bin/blog.cgi/200...
"...And here's the punch line, Solaris has never even run on a 1024 cpu system let alone one as big this new SGI system, and Linux has handled it just fine for years. Yet Mr. Bonwick feels compelled to imply that Linux doesn't scale and Solaris does. To claim that Solaris is more ready to scale on large multi-core systems is pure FUD, and I'm saddened to see someone as technically gifted as Jeff stoop to this level..."
Who is slandering who? Is it FUD to say that Linux has scalability problems over 8 sockets? Is it FUD to say that there has never been a 32 socket Linux server for sale? Or is it just that he is not aware of different types of scalability: clusters or SMP servers? Is it just pure ignorance, when he believes a 4096 core Linux cluster can replace a 32 socket SMP server? What do you think? Is it FUD when the ZFS creator claims that Linux does not scale on 32 socket servers, or is it in fact a true claim? Who is FUDing who?
Kevin G - Tuesday, February 25, 2014 - link
Linux scales just as well as Unix on large socket counts. Case in point are IBM's own benchmarks on their p795 systems with 32 sockets, 256 cores and 1024 threads: AIX only beats Linux by a mere 2.7% Source: http://www-03.ibm.com/systems/power/hardware/795/p...I should also point out that your link is 7 years old. Things have changed in the Linux kernel.
hoboville - Monday, February 24, 2014 - link
Well you're right, but it's not as bad for x86 as you make it sound. Systems like TITAN were examples of scale-out compute, if ever there was one. I'll grant it's not the same in terms of what they calculate (Titan is simulation focused and GPU focused) and less on pure RAS and rapid DB access like ERP (not transactional / real time). But that's essentially irrelevant. The point is how they scale in terms of number of nodes and the cost of nodes.Intel's newest chip is cool, but not practical in terms of price competition (why Titan used more Opteron nodes instead of Xeon, for example). What you're focused on is price competition at the ultimate upper end of the spectrum, where SPARC and Power live. And that, in turn, the price of the highest end single system. Intel may be trying to break into that space, but no, it doesn't make sense because x86 wasn't designed for it as an architecture. Their single systems won't compete, yet.
But that's not to say this new Xeon irrelevant. It isn't. It will, however, have problems because of the price-per-performance isn't competitive. In a scale-out design you want more, cheaper nodes and beat the competition by volume. These nodes are just too expensive when you want performance per dollar.
What most mid-to-large companies need is a scalable setup that grows with their business. A lot of IT is bean counting and cost cutting. If you want to start SMP, you start small and tack on additional systems, because your budget people won't let you get a SPARC system or Unix setup. Oracle just doesn't offer systems or prices that are reasonable, and because of this, many businesses that SMP won't give them a second glance. This is where x86 and Xeon fit into the picture, scale out, starting small and building up. But these new systems are asking too much and people aren't going to be interested.
Kevin G - Monday, February 24, 2014 - link
Intel has effectively killed off the Itanium. The original 22 nm Kitson has been scrapped and the successor to Poulson is going to be on 32 nm as well. After that, nothing appears on Intel's roadmap for the chip.HP, the largest Itanium customer, has already announced that their NonStop mainframe line is moving to x86:
Kevin G - Monday, February 24, 2014 - link
Forgot the link: http://h17007.www1.hp.com/us/en/enterprise/servers...Kevin G - Monday, February 24, 2014 - link
"So, instead of you telling me I am wrong, I suggest you just show us links with SMP workloads for the SGI UV2000 server... then you are right, and I am wrong. And I will shut up."United States Post Office running Oracle Data Warehouse software on a SGI UV1000 (the older sibling of the UV2000, still shared memory and cache coherent):
https://www.fbo.gov/index?s=opportunity&mode=f...
SGI and MarkLogic for Big Data:
http://www.v3.co.uk/v3-uk/news/2216603/sgi-and-mar...
I've also found passing references other government (No Such Agency?) installations of a UV2000 installation running Hadoop.
Brutalizer - Tuesday, February 25, 2014 - link
But please, Kevin G, dont you know that Hadoop is a clustered solution? Why do you think people are running clustered database solutiosn as Hadoop on a SGI UV2000 server? Is it because SGI says it is for clustered benchmarks only?And yes, there are clustered databases.
Kevin G - Tuesday, February 25, 2014 - link
Did you not see the link where the USPS is running Oracle workloads on a UV1000? I'll post it again so that you may see: https://www.fbo.gov/index?s=opportunity&mode=f...Kevin G - Tuesday, February 25, 2014 - link
There a couple of reasons why someone would have to run Hadoop on a UV2000: the UV2000 has a large global address space which data could directly reside (ie. no disks access necessary!). If the raw data can reside in 64 TB, performance should be very good. Secondly, Hadoop is free under the Apache license. Traditional database software like Oracle charge a premium the more sockets there are installed on a system. I'd imagine that 256 socket UV2000 system would incur an Oracle licensing fee in the tens of millions of US dollars. So between the choice of free or tens of millions of dollars, most organizations would at least try to work with the free solution.