Comparing Benchmarks: AT vs IBM

Before we close things out, let's spend a moment summarizing our results and comparing the performance we saw to the kind of performance advantages that IBM advertises POWER8 is capable of.

From a high level perspective, the S822L is more expensive and consumes a lot more power than a comparable Xeon system.

With limited optimization and with the current Ubuntu 15.04, the performance-per-watt ratio favors Intel even more as the POWER8 barely outperforms the very efficient 120W TDP Xeons. So there is no denying that the Intel systems offer a superior performance/watt ratio.

However, it would be unfair to base our judgement on our first attempt as we have to admit this our first real attempt to benchmark and test the POWER8 platform. It is very likely that we will manage to extract quite a bit more performance out of the system on our second attempt. IBM POWER8 also has a big advantage in memory bandwidth. But we did not manage to port OpenFOAM to the POWER platform, probably the most likely candidate for leveraging that advantage.

We are less convinced that the POWER8 platform has a huge "raw CPU compute advantage," contrary to what for example IBM's SPECJBB (85% faster ) and SAP (29% faster) results seem to suggest.

For example, IBM's own SPECjEnterprise®2010 benchmarking shows that:

SAP is "low IPC" (hard to run many instructions in parallel in one thread) software that benefits much from low latency caches. The massive L3-cache (12-cores, 96 MB) and huge thread count are probably giving the IBM POWER8 the edge. The RAM bandwidth also helps, but in a lesser degree. IBM clearly built POWER8 with this kind of software in mind. We had individual threadcount intensive benchmarks (LZMA decompression) and L3-cache sensitive benchmarks (ElasticSearch), but t o be fair to IBM, none of our benchmarks leveraged the three strongest points (threadcount, L3-cache size and memory bandwidth) all at once like SAP.

SPECJBB2013 has recently been discontinued as it was not reliable enough. We tend to trust the jEnterprise test a lot more. In any case, the best POWER8 has a 17% advantage there.

Considering that the POWER8 inside that S824 has 20% more cores and a 3% higher clockspeed, our 3.4 GHz 10-core CPU would probably be slightly behind the Xeon E5-2697 v3. We found out that the 10-core POWER8 is slightly faster than Xeon E5-2695 v3. The Xeon E5-2695 v3 is very similar to the E5-2697 v3, it is just running at a 10% lower clockspeed (All core turbo: 2.8GHz vs 3.1GHz). So all in all, our benchmarks seems to be close to the official benchmarks, albeit slightly lower.

Closing Thoughts: A Mix of Xeon "E5" and "E7"

So let's sum things up. The IBM S822L is definitely not a good choice for those looking to lower their energy bills or operate in a location with limited cooling. The pricing of the CDIMMs causes it to be more expensive than a comparable Xeon E5 based server. However, you get something in return: the CDIMMs should offer higher reliability and are more similar to the memory subsystem of the E7 than the E5. Also, PCIe adapters are hot-pluggable on the S822L and can be replaced without bringing down the system. With most Xeon E5 systems, only disks, fans and PSU are hot-pluggable.

In a number of ways then, the S822L is more a competitor to dual Xeon E7 systems than it is to dual Xeon E5 systems. In fact, a dual Xeon E7 server consumes in the 600-700W range, and in that scenario the power usage of S822L (700-800W) does not seem outrageous anymore.

The extra reliability is definitely a bonus when running real time data analytics or virtualization. A failing memory chip may cost a lot when you running fifty virtual machines on top of a server. Even in some HPC or batch data analytics applications where you have to wait for hours for a certain result that is being computed in an enormous amount of memory, the cost savings of being able to survive a failing memory chip might be considerable.

One more thing: for those who need full control, the fact that every layer in the software stack is open makes the S822L very attractive. For now, the available "OpenCompute" Xeon servers that are also "open" seem to mostly density optimized servers and the openess seems limited on several levels. Rackspace felt that the current OpenCompute servers are not "open enough", and went for OpenPOWER servers instead. In all those markets, the S822L is very interesting alternative to the dual Xeon E5 servers.

Ultimately however, the performance-per-dollar Xeon E5 competitors will most likely be OpenPOWER third party servers. Those servers do not use CDIMMS, but regular RDIMMs. Other components such as disks, networkcards and PSUs will probably be cheaper but potentially also slightly less reliable.

All in all, the arrival of OpenPOWER servers is much more exciting than most of us anticipated. Although the IBM POWER8 servers can not beat the performance/watt ratio of the Xeon, we now have a server processor that is not only cheaper than Intel's best Xeons, but that can also keep up with them. Combine that with the fact that IBM has lined up POWER8+ for next year and a whole range of server vendors is building their own POWER8 based servers, and we have a lot to look forward to!

Energy and Pricing
POST A COMMENT

146 Comments

View All Comments

  • JohanAnandtech - Wednesday, November 11, 2015 - link

    I did not know you used 7500x3500, my testing was inspired on what the rest of the benchmarking community (Phoronix, Serverthehome) was using (obviously, 1024x768 is too small for current servers).

    http://www.anandtech.com/show/9567/the-power-8-rev...
    This answers your question about threads, right?
    Reply
  • JohanAnandtech - Wednesday, November 11, 2015 - link

    Oh yes, changed the link. Thanks for the feedback! Reply
  • mapesdhs - Thursday, November 12, 2015 - link

    Most welcome! And I really should move the more complex tests to the top of the page...

    Oh, my wording about threads was not what I'd intended. What I meant was, the no. of threads being larger than the supported no. of hardware threads. Thus, for a 12-core Power8 with 8 threads per core, try using 192 or 384 threads, instead of just the nominal 96 one might assume would make sense.

    Ian.
    Reply
  • MB13 - Wednesday, November 11, 2015 - link

    POWER8 is full of innovation and brings change! An S812LC only costs $6,595 from IBM's external website! http://www-03.ibm.com/systems/power/hardware/s812l...

    The Power scale out boxes will save on your running and software costs as you can reduce your software licensing and server footprint.

    With the OpenPOWER Foundation, you now have companies such as Tyan and Wistron who also create their own POWER8 servers and sell them independently of IBM. If you have not looked at The OpenPOWER Foundation and the innovation it brings through community and collaboration, your missing out big time!

    There is change! Don't get left behind!
    Reply
  • MB13 - Wednesday, November 11, 2015 - link

    and don't forget - POWER8 runs Little Endian and support the latest versions of RedHat, SUSE and Ubuntu!
    The OpenPOWER servers are Linux only!
    Reply
  • Gasaraki88 - Wednesday, November 11, 2015 - link

    It's funny how this article is trying to "sell" me the system but I'm still not impressed. Costs more, less performance, and uses more power at idle and load than the Intel system. Reply
  • nils_ - Thursday, November 12, 2015 - link

    What I found the most off putting is that you have to do a lot of work to get some things running with Linux. That's a big cost factor. Reply
  • nils_ - Thursday, November 12, 2015 - link

    Having a lot of software that isn't really well ported is probably going to remain a problem for Power8 for years to come since so few people have access to these kinds of systems and the cost is prohibitive. The great thing with x86 and ARM is that you can use it at home/work pretty easily without shelling out a lot of money. On x86 you can be sure if your software builds locally and runs locally it will also run on your server. Reply
  • svj - Thursday, November 12, 2015 - link

    Well written articles.

    1. I submit that the headline is misleading. Intel x86 does not compete with POWER at the high end. POWER L & LC line of servers are comparable to x86 based servers. IBM POWER is taking the battle to Intel's home turf.
    2. The analysis leaves out cost of SW. Many organizations use commercial software which are priced per core. If POWER can do with 10 cores what Intel does with 18 cores, that means HUGE savings.
    3. OPEN POWER is a huge move. I think the market will start seeing the results soon.
    Reply
  • alpha754293 - Thursday, November 12, 2015 - link

    An excellent review as always Johan. (haha...to zeeBomb. It is my understanding that Johan doesn't post probably as often as he might have otherwise like to because testing servers/enterprise computing solutions takes a LOT longer than testing/benching consumer-level systems. Some of the HPC applications that I run takes hours to days for each run, so when you're running it, you're running those tests over and over again, and before you know it, a month has gone by (or you've ran out of time with the system) or you have to purposely cut it short so that you can test a variety of software.)

    It's unfortunate that IBM never ported AIX to x86 (unlike Solaris.) I think that there would be more people trying to get into it if the cost of entry (even just to learn) isn't so high. I've looked at getting an old POWER4 system before for that purpose, but by then, the systems are so old and slow that it's like "what's the point?" I think that IBM is literally pricing themselves into extinction (along with their entire hardware/software ecosystem). Unfortunately for many businesses, AIX POWER servers still run their mainframe/backend which means that if you want to get paid $100k+ outta college - go learn AIX on POWER. As the current generation of sysadmins are starting to age and retire out, and they're going to have a hard time finding qualified people, the only way eventually would be that they would have to pay top dollar just to attract people into the field. (Unless they decide to move everything over to the x86/Xeon/Linux world. But for some mainframes (like financial institutions), that's DEFINITELY easier said than don).
    Reply

Log in

Don't have an account? Sign up now