Assessing IBM's POWER8, Part 1: A Low Level Look at Little Endian
by Johan De Gelas on July 21, 2016 8:45 AM ESTMemory Subsystem: Bandwidth
As we mentioned before, the IBM POWER8 has a memory subsystem which is more similar to the Xeon E7's than the E5's. The IBM POWER8 connects to 4 "Centaur" buffer cache chips, which have both a 19.2 GB/s read and 9.6 GB/s write link to the processor, or 28.8 GB/s in total. So the 105 GB/s aggregate bandwidth of the POWER8 is not comparable to Intel's peak bandwidth. Intel's peak bandwidth is the result of 4 channels of DDR4-2400 that can either write or read at 76.8 GB/s (2.4 GHz x 8 bytes per channel x 4 channels).
Bandwidth is of course measured with John McCalpin's Stream bandwidth benchmark. We compiled the stream 5.10 source code with gcc 5.2.1 64 bit. The following compiler switches were used on gcc:
-Ofast -fopenmp -static -DSTREAM_ARRAY_SIZE=120000000
The latter option makes sure that stream tests with array sizes which are not cacheable by the Xeons' huge L3 caches.
It is important to note why we use the GCC compiler and not vendors' specialized compilers: the GCC compiler is not as good at vectorizing the code. Intel's ICC compiler does that very well, and as result shows the bandwidth available to highly optimized HPC code, which is great for that code in the real world, but it's not realistic for multi-threaded server applications.
With ICC, Intel can use the very wide 256-bit load units to their full potential and we measured up to 65 GB/s per socket. But you also have to consider that ICC is not free, and GCC is much easier to integrate and automate into the daily operations of any developer. No licensing headaches, no time consuming registrations.
The combination of the powerful four load and two store subsystem of the POWER8 and the read/write interconnect between the CPU and the Centaur chips makes it much easier to offer more bandwidth. The IBM POWER8 delivers a solid 90 GB/s despite using old DDR3-1333 memory technology.
Intel claims higher bandwidth numbers, but those numbers can only be delivered in vectorized software.
Facebook
Twitter
RSS
124 Comments
View All Comments
zodiacfml - Thursday, July 21, 2016 - link
Like a good TV series, I can't wait for the next episode.aryonoco - Friday, July 22, 2016 - link
OK, this is literally why Anandtech is the best in the tech journalism industry.There is nowhere else on the net that you can find a head to head comparison between POWER and Xeon, and unless you work in the tech department of a Fortune 500 company, this information has just not been available, until now.
Johan, thank you for your work on this article. I did give you beef in your previous article about using LE Ubuntu but I concede your point. Very happy to you are writing more for Anandtech these days.
Xeons really need some competition. Whether that competition comes from POWER or ARM or Zen, I am happy to see some competition. IBM has big plans for POWER9. Hopefully this is just the start of things to come.
JohanAnandtech - Friday, July 22, 2016 - link
Thanks! it is very exciting to perform benchmarks that nobody has published yet :-).In hindsight, I have to admit that the first article contained too few benchmarks that really mattered for POWER8. Most of our usual testing and scripting did not work, and so after lot of tinkering, swearing and sweat I got some benchmarks working on this "exotic to me" platform. The contrast between what one would expect to see on POWER8 and me being proud of being able to somewhat "tame the beast" could not have been greater :-). In other words, there was a learning curve.
tipoo - Friday, July 22, 2016 - link
I found it very interesting as well and would certainly not mind seeing more from this space, like maybe Xeon Phi and SPARC M7jospoortvliet - Tuesday, July 26, 2016 - link
Amen. But, to not ask to much, just the prospect of part 2 of the Power benchmark is already super exciting. Yes, the Internetz need more of this!Daniel Egger - Friday, July 22, 2016 - link
Not quite sure what the Endianess of a systems adds to the competitive factor. Maybe someone could elaborate why it is so important to run a system in LE?ZeDestructor - Friday, July 22, 2016 - link
Not much, really, with the compilers being good and all that.Really, it's quite clearly there just for some excellent alliteration.
JohanAnandtech - Friday, July 22, 2016 - link
Basically LE reduces the barrier for an IBM server being integrated in x86 dominated datacenter.see https://www.ibm.com/developerworks/community/blogs...
Just a few reasons:
"Numerous clients, software partners, and IBM’s own software developers have told us that porting their software to Power becomes simpler if the Linux environment on Power supports little endian mode, more closely matching the environment provided by Linux on x86. This new level of support will *** lower the barrier to entry for porting Linux on x86 software to Linux on Power **."
"A system accelerator programmer (GPU or FPGA) who needs to share memory with applications running in the system processor must share data in an pre-determined endianness for correct application functionality."
Daniel Egger - Friday, July 22, 2016 - link
While correct in theory, this hasn't been a problem for the last 20 years. People are used to using BE on PPC/POWER, the software, the drivers and the infrastructure are very mature (as a matter of fact it was my job 15 years ago to make sure they are). PPC/POWER actually have configurable endianess so if someone wanted to go LE earlier it would have easily been possible but only few ever attempted that stunt; so why have the big disruption now?KAlmquist - Friday, July 22, 2016 - link
I assume that this is about selling POWER boxes to companies that currently run all x86 servers, and have a bunch of custom software that they might be willing to recompile. If the customer has to spend a bunch of time fixing endian dependencies in his software in order to get it to work on POWER, it will probably be less expensive for them to simply stick with x86.