Assessing IBM's POWER8, Part 1: A Low Level Look at Little Endian
by Johan De Gelas on July 21, 2016 8:45 AM ESTMemory Subsystem: Latency Measurements
There is no doubt about it: the performance of modern CPUs depends heavily on the cache subsystem. And some applications depend heavily on the DRAM subsystem too. We used LMBench in an effort to try to measure latency. Our favorite tool to do this, Tinymembench, does not support the POWER architecture yet. That is a pity, because it is a lot more accurate and modern (as it can test with two outstanding requests).
The numbers we looked at were "Random load latency stride=16 Bytes" (LMBench).
| Mem Hierarchy |
IBM POWER8 | Intel Broadwell Xeon E5-2640v4 DDR4-2133 |
Intel Broadwell Xeon E5-2699v4 DDR4-2400 |
| L1 Cache (cycles) | 3 | 4 | 4 |
| L2 Cache (cycles) | 13 | 12-15 | 12-15 |
| L3 Cache 4-8 MB(cycles) | 27-28 (8 ns) | 49-50 | 50 |
| 16 MB (ns) | 55 ns | 26 ns | 21 ns |
| 32-64 MB (ns) | 55-57 ns | 75-92 ns | 80-96 ns |
| Memory 96-128 MB (ns) | 67-74 ns | 90-91 ns | 96 ns |
| Memory 384-512 MB (ns) | 89-91 ns | 91-93 ns | 95 ns |
(Note that the numbers for Intel are higher than what we reported in our Cavium ThunderX review. The reason is that we are now using the numbers of LMBench and not those of Tinymembench.)
A 64 KB L1 cache with 4 read ports that can run at 4+ GHz speeds and still maintain a 3 cycle load latency is nothing less than the pinnacle of engineering. The L2 cache excels too, being twice as large (512 KB) and still offering the same latency as Intel's L2.
Once we get to the eDRAM L3 cache, our readings get a lot more confusing. The L3 cache is blistering fast as long as you only access the part that is closest to the core (8 MB). Go beyond that limit (16 MB), and you get a latency that is no less than 7 times worse. It looks like we actually hitting the Centaur chips, because the latency stays the same at 32 and 64 MB.
Intel has a much more predictable latency chart. Xeon's L3 cache needs about 50 cycles, and once you get into DRAM, you get a 90-96 ns latency. The "transistion phase" from 26 ns L3 to 90 ns DRAM is much smaller.
Comparatively, that "transition phase" seems relatively large on the IBM POWER8. We have to go beyond 128 MB before we get the full DRAM latency. And even then the Centaur chip seems to handle things well: the octal DDR-3 1333 MHz DRAM system delivers the same or even slightly better latency as the DDR4-2400 memory on the Xeon.
In summary, IBM's POWER8 has a twice as fast 8 MB L3, while Intel's L3 is vastly better in the 9-32 MB zone. But once you go beyond 32 MB, the IBM memory subsystem delivers better latency. At a significant power cost we must add, because those 4 memory buffers need about 64 Watts.
Facebook
Twitter
RSS
124 Comments
View All Comments
JohanAnandtech - Thursday, July 28, 2016 - link
Send me a mail at johan@anandtech.comabufrejoval - Thursday, August 4, 2016 - link
Hmm, a bit fuzzy after the first paragraph or so and evidently because I dislike malwaretizement: Such links should be banned!mystic-pokemon - Friday, July 22, 2016 - link
Hi floobitFor virtualization: powerVM and out of the box KVM (tested on Fedora 23, Ubuntu 15.04 / 15.10 / 16.04) work quite well. Xen doesn't work well or hasn't been officially tested / released.
tipoo - Thursday, July 21, 2016 - link
Fun! I was always curious about this processor.tipoo - Thursday, July 21, 2016 - link
Interesting that the L3 eDRAM not only allows them to pack in much more L3 (what was it, 3 SRAM transistors per eDRAM or something?), but it's also low latency which was a cited concern with eDARM by some people. Appears to be an unfounded fear.And then on top of that they put another large L4 eDRAM cache on.
Maybe Intel needs to play with eDRAM more...
tipoo - Thursday, July 21, 2016 - link
Lol, eDRAM, not eDARMKevin G - Thursday, July 21, 2016 - link
There was a change in how the L4 cache works from Broadwell to SkyLake on the mobile parts. The implication is that Intel was exploring the idea of a large L4 eDRAM for SkyLake-EP/EX parts. We'll see how that turns out as Intel also has explored using HMC as a cache for high bandwidth applications in Knights Landing. So either way, Intel has thus idea on there radar and we'll see how it pans out next year.tsk2k - Thursday, July 21, 2016 - link
Is it possible to run Windows on one of these?ZeDestructor - Thursday, July 21, 2016 - link
At the moment, a very solid no.That said, if enough partners ask for it and/or if the numbers make sense for Azure, MS will at the very least have a damn good look at porting Windows over.
DanNeely - Thursday, July 21, 2016 - link
It's probably just a case of doing QA and releasing it. They've sold a PPC build in the past; and maintain internal builds for a number of other CPU architectures to avoid accidentally baking x86isms into the core code.