Assessing IBM's POWER8, Part 1: A Low Level Look at Little Endian
by Johan De Gelas on July 21, 2016 8:45 AM ESTMemory Subsystem: Latency Measurements
There is no doubt about it: the performance of modern CPUs depends heavily on the cache subsystem. And some applications depend heavily on the DRAM subsystem too. We used LMBench in an effort to try to measure latency. Our favorite tool to do this, Tinymembench, does not support the POWER architecture yet. That is a pity, because it is a lot more accurate and modern (as it can test with two outstanding requests).
The numbers we looked at were "Random load latency stride=16 Bytes" (LMBench).
| Mem Hierarchy |
IBM POWER8 | Intel Broadwell Xeon E5-2640v4 DDR4-2133 |
Intel Broadwell Xeon E5-2699v4 DDR4-2400 |
| L1 Cache (cycles) | 3 | 4 | 4 |
| L2 Cache (cycles) | 13 | 12-15 | 12-15 |
| L3 Cache 4-8 MB(cycles) | 27-28 (8 ns) | 49-50 | 50 |
| 16 MB (ns) | 55 ns | 26 ns | 21 ns |
| 32-64 MB (ns) | 55-57 ns | 75-92 ns | 80-96 ns |
| Memory 96-128 MB (ns) | 67-74 ns | 90-91 ns | 96 ns |
| Memory 384-512 MB (ns) | 89-91 ns | 91-93 ns | 95 ns |
(Note that the numbers for Intel are higher than what we reported in our Cavium ThunderX review. The reason is that we are now using the numbers of LMBench and not those of Tinymembench.)
A 64 KB L1 cache with 4 read ports that can run at 4+ GHz speeds and still maintain a 3 cycle load latency is nothing less than the pinnacle of engineering. The L2 cache excels too, being twice as large (512 KB) and still offering the same latency as Intel's L2.
Once we get to the eDRAM L3 cache, our readings get a lot more confusing. The L3 cache is blistering fast as long as you only access the part that is closest to the core (8 MB). Go beyond that limit (16 MB), and you get a latency that is no less than 7 times worse. It looks like we actually hitting the Centaur chips, because the latency stays the same at 32 and 64 MB.
Intel has a much more predictable latency chart. Xeon's L3 cache needs about 50 cycles, and once you get into DRAM, you get a 90-96 ns latency. The "transistion phase" from 26 ns L3 to 90 ns DRAM is much smaller.
Comparatively, that "transition phase" seems relatively large on the IBM POWER8. We have to go beyond 128 MB before we get the full DRAM latency. And even then the Centaur chip seems to handle things well: the octal DDR-3 1333 MHz DRAM system delivers the same or even slightly better latency as the DDR4-2400 memory on the Xeon.
In summary, IBM's POWER8 has a twice as fast 8 MB L3, while Intel's L3 is vastly better in the 9-32 MB zone. But once you go beyond 32 MB, the IBM memory subsystem delivers better latency. At a significant power cost we must add, because those 4 memory buffers need about 64 Watts.
Facebook
Twitter
RSS
124 Comments
View All Comments
tipoo - Thursday, July 21, 2016 - link
They made PowerPC Windows? Source? I remember the Powermac G5s were the early dev kits for the xbox 360 due to the architecture similarity, but I assumed those stories meant they were just working in OSX or Linux on them.thunderbird32 - Thursday, July 21, 2016 - link
AFAIK, the last build of Windows for PPC was NT 4. So, it's been a while.Sunner - Thursday, July 21, 2016 - link
There were early builds of Windows 2000 for the RISC's as well, during the times when it was still called NT5. I had one of those from WinHEC, but alas I lost it when moving at some point. :(yuhong - Thursday, July 21, 2016 - link
AFAIK, the little endian PowerPC mode that NT4 used was killed when they went to 64-bit and is different from today's POWER8 little endian mode that was only recently introduced.Kevin G - Thursday, July 21, 2016 - link
I used to have such a disc for Windows NT4. That disk also had binaries for DEC Alpha and MIPS.BillyONeal - Thursday, July 21, 2016 - link
The Xbox 360 is a PPC machine, and runs a (heavily modified) version of Windows. My understanding is that most x86 assumptions had to be ferreted out to run on Itanium (early) and then on ARM (later).Einy0 - Thursday, July 21, 2016 - link
MS has builds that will run on anything. The real question is why would you want to? These chips are designed from the ground up to run massive work loads. It's a completely different style of computing than a Windows machine. Even MS server OSes aren't designed for this type of work. We are talking Banking, ERP and other big data applications. MS is still dreaming about scaling on that level. Right now their answer is clustering but that comes with it's own obstacles too.abufrejoval - Thursday, August 4, 2016 - link
Well there is always QEMU.And IBM has a much better binary translator from when they bought QuickTransit. That one originally translated Power to x86 for the Mac, then Sparc to x86 for Quicktransit and eventually x86 to Power for IBM so they could run Linux workloads on AIX.
Then what exactly do you mean with Windows (assuming this is actually a reasonable question)?
Server applications or desktop?
.NET has been ported to Linux and I guess could be made to run on Power. A Power runtime could certainly be done by Microsoft, if they wanted to.
I don't see why anyone would want to run Windows desktop workloads on this hardware, other than to show that it can be done: QEMU to that!
BedfordTim - Thursday, July 21, 2016 - link
I was intrigued to see how little effect hyper-threading with your Xeon. My own experience is that it gives a 50% boost although I appreciate there are many variables.Taracta - Thursday, July 21, 2016 - link
Something seems to be wrong with the Mem Hierarchy charts in the Intel L3 and 16MB section.