The IBM POWER8 Review: Challenging the Intel Xeon
by Johan De Gelas on November 6, 2015 8:00 AM EST- Posted in
- IT Computing
- CPUs
- Enterprise
- Enterprise CPUs
- IBM
- POWER
- POWER8
Comparing Benchmarks: AT vs IBM
Before we close things out, let's spend a moment summarizing our results and comparing the performance we saw to the kind of performance advantages that IBM advertises POWER8 is capable of.
From a high level perspective, the S822L is more expensive and consumes a lot more power than a comparable Xeon system.
With limited optimization and with the current Ubuntu 15.04, the performance-per-watt ratio favors Intel even more as the POWER8 barely outperforms the very efficient 120W TDP Xeons. So there is no denying that the Intel systems offer a superior performance/watt ratio.
However, it would be unfair to base our judgement on our first attempt as we have to admit this our first real attempt to benchmark and test the POWER8 platform. It is very likely that we will manage to extract quite a bit more performance out of the system on our second attempt. IBM POWER8 also has a big advantage in memory bandwidth. But we did not manage to port OpenFOAM to the POWER platform, probably the most likely candidate for leveraging that advantage.
We are less convinced that the POWER8 platform has a huge "raw CPU compute advantage," contrary to what for example IBM's SPECJBB (85% faster ) and SAP (29% faster) results seem to suggest.
For example, IBM's own SPECjEnterprise®2010 benchmarking shows that:
- Two Xeon E5-2697 v3, 14 cores at 2.6-3.1 GHz score 19282
- Two POWER8, 12 cores at 3.5-3.8 GHz score 22543
SAP is "low IPC" (hard to run many instructions in parallel in one thread) software that benefits much from low latency caches. The massive L3-cache (12-cores, 96 MB) and huge thread count are probably giving the IBM POWER8 the edge. The RAM bandwidth also helps, but in a lesser degree. IBM clearly built POWER8 with this kind of software in mind. We had individual threadcount intensive benchmarks (LZMA decompression) and L3-cache sensitive benchmarks (ElasticSearch), but t o be fair to IBM, none of our benchmarks leveraged the three strongest points (threadcount, L3-cache size and memory bandwidth) all at once like SAP.
SPECJBB2013 has recently been discontinued as it was not reliable enough. We tend to trust the jEnterprise test a lot more. In any case, the best POWER8 has a 17% advantage there.
Considering that the POWER8 inside that S824 has 20% more cores and a 3% higher clockspeed, our 3.4 GHz 10-core CPU would probably be slightly behind the Xeon E5-2697 v3. We found out that the 10-core POWER8 is slightly faster than Xeon E5-2695 v3. The Xeon E5-2695 v3 is very similar to the E5-2697 v3, it is just running at a 10% lower clockspeed (All core turbo: 2.8GHz vs 3.1GHz). So all in all, our benchmarks seems to be close to the official benchmarks, albeit slightly lower.
Closing Thoughts: A Mix of Xeon "E5" and "E7"
So let's sum things up. The IBM S822L is definitely not a good choice for those looking to lower their energy bills or operate in a location with limited cooling. The pricing of the CDIMMs causes it to be more expensive than a comparable Xeon E5 based server. However, you get something in return: the CDIMMs should offer higher reliability and are more similar to the memory subsystem of the E7 than the E5. Also, PCIe adapters are hot-pluggable on the S822L and can be replaced without bringing down the system. With most Xeon E5 systems, only disks, fans and PSU are hot-pluggable.
In a number of ways then, the S822L is more a competitor to dual Xeon E7 systems than it is to dual Xeon E5 systems. In fact, a dual Xeon E7 server consumes in the 600-700W range, and in that scenario the power usage of S822L (700-800W) does not seem outrageous anymore.
The extra reliability is definitely a bonus when running real time data analytics or virtualization. A failing memory chip may cost a lot when you running fifty virtual machines on top of a server. Even in some HPC or batch data analytics applications where you have to wait for hours for a certain result that is being computed in an enormous amount of memory, the cost savings of being able to survive a failing memory chip might be considerable.
One more thing: for those who need full control, the fact that every layer in the software stack is open makes the S822L very attractive. For now, the available "OpenCompute" Xeon servers that are also "open" seem to mostly density optimized servers and the openess seems limited on several levels. Rackspace felt that the current OpenCompute servers are not "open enough", and went for OpenPOWER servers instead. In all those markets, the S822L is very interesting alternative to the dual Xeon E5 servers.
Ultimately however, the performance-per-dollar Xeon E5 competitors will most likely be OpenPOWER third party servers. Those servers do not use CDIMMS, but regular RDIMMs. Other components such as disks, networkcards and PSUs will probably be cheaper but potentially also slightly less reliable.
All in all, the arrival of OpenPOWER servers is much more exciting than most of us anticipated. Although the IBM POWER8 servers can not beat the performance/watt ratio of the Xeon, we now have a server processor that is not only cheaper than Intel's best Xeons, but that can also keep up with them. Combine that with the fact that IBM has lined up POWER8+ for next year and a whole range of server vendors is building their own POWER8 based servers, and we have a lot to look forward to!
Facebook
Twitter
RSS
146 Comments
View All Comments
jesperfrimann - Monday, November 9, 2015 - link
Well, I think you should kick Franz Bourlet, for not hooking you up with with a IBM technical Advocate who actually knew the technology. Such a person could have shown you the robes and helped you understand the kit better. Again Franz is a sales guy.IMHO selecting Ubuntu as the Linux distro, did not help you. It's new to the POWER platform and does not have the same robustness as for example SLES which have been around for 10+ years on POWER.
The fact that you are getting better results using gcc generated code rather than xLC, shows me that something is not right.
And that the IBM JDK isn't working is well also an indicator that something is now right.
IMHO selecting Ubuntu, did not make Things easier for you Guys.
And for really optimized code you need to install and use High performance math libraries for POWER (MASS), which is an addon math library.
And AFAIR having 8 memory modules, only enables half the memory bandwidth of the system.
So IMHO IBM didn't help you make their system look good.
But again that is what you get when you get rid of all the clever people :)
// Jesper
nils_ - Wednesday, November 11, 2015 - link
You can always rent a box at OVH, they offer a huge chunk of an OpenPower System, albeit virtualized through Runlabs.stefstef - Sunday, November 8, 2015 - link
compared to the pentium 4 the mips r16k with loads of l3 cache was a bzip2 beast, outperforming the pentium 4 which ran at twice the clock speed and more. despite that the usage of zip programs is what these server processors are build.mapesdhs - Tuesday, November 10, 2015 - link
Just curious, do you know of any comparative results anywhere for bzip2 on old MIPS vs. other CPUs? It's not something I've seen mentioned before, at least not with respect to SGIs, but perhaps I can run som tests the next time I obtain a quad-R16K/1GHz (16MB L2) Tezro. Best I have at is only an R16K/900MHz (8MB L2) single-CPU Fuel and various configs of Tezro and Onyx350 from 4 to 16x 700MHz with 8MB L2. Just a pity SGI never got to employ multi-core MIPS (it was planned, but alas never happened).Oddly, back when current, MIPS' real strength was fp. Over time it fell behind badly for general int, though for SGI's core markets that didn't really matter ("It's the bandwidth, stupid!" - famous quote from Mashey IIRC). MIPS could have caught up with MDMX and MIPS V ISA, especially with the initially intended merged Cray vector stuff, but again that all fell away once the design talent moved to Intel in 1996/7.
Ian.
Freen the merciless - Sunday, November 8, 2015 - link
Heh! Sparc T5 eats Xeon and power for breakfast.kgardas - Monday, November 9, 2015 - link
I guess you mean T7 with SPARC M7 inside and not T5. If so, then yes, M7 looks quite capable, but unfortunately provides horrible price/performance ratio. POWER8 box starts at ~6.5k $ while T7-1 on ~40k $. So on SPARC front we'll need to see if Oracle is going to change that with Sonoma chip.Michael Bay - Monday, November 9, 2015 - link
In parallel only.aryonoco - Tuesday, November 10, 2015 - link
Thank you Johan for this amazingly well written and well researched article.I have to agree with a few people here that question your choice of using LE Ubuntu to test. Traditionally people who use Linux on POWER use SUSE, and some use RHEL, but Ubuntu? Nothing against them, and I love apt, but it's just not a mature platform.
Try with something more representative such as BE SLES and you will find a vastly different types ecosystem maturity.
But thanks again, and also thanks to AT for caring about such subjects and publishing these tests.
JohanAnandtech - Wednesday, November 11, 2015 - link
Thank you for taking the time to write up some constructive feedback. I have years of experience with ubuntu and linux and I wanted to play it safe. Running benchmarks on "new" hardware with a new ISA (from my perspective) is pretty complex. C-ray and 7-zip are the only exceptions, but most real server apps (NAMD, ElasticSearch, Spark) depends on many layers of software.In theory the OS/ distro is more important to get applications working than the ISA. In practice, it might have been better to bet on the distro with the most maturity and adapt our scripts and installation procedures to Suse.
But as soon as I get the chance, I'll try out BE suse or redhat on a POWER system.
mapesdhs - Tuesday, November 10, 2015 - link
Johan,A minor point, please note my home page for C-ray is here:
http://www.sgidepot.co.uk/c-ray.html
Blinkenlights is just a mirror, and not the primary mirror either (that would be the vintagecomputers site).
Btw, it's a pity you didn't use the same image sizes & settings as used on the main c-ray site, because then I could have included the results on my page (ie. 'sphfract' at 800x600, 1024x768 with 8X oversampling, and 7500x3500), or did you just use the same settings that Phoronix employs?
Also, John Tsiombikas, the guy who wrote C-ray, told me some interesting things about the test and how it works (info included on the page), most especially that it is highly vulnerable to compiler optimisations which can produce results that are even less realistic than real life workloads. I'm glad thought that you did at least use the sphfract test, since at a sensible resolution or with oversampling it easily pushes the test out of just L1 (the 'scene' test is much smaller). But yeah, overall, c-ray was never intended to be used as a benchmark, it's just taken off somehow, perhaps because the scanline method of threading makes it scale very well.
Hmm, I really must sort out the page formatting one of these days, and move the most complex test tables to the top. Never seem to find the time...
Thanks!!
Ian.
PS. I always obtained the best results by having more threads than the no. of cores/CPUs, or is this something which doesn't work with non-MIPS systems?