Assessing IBM's POWER8, Part 1: A Low Level Look at Little Endian
by Johan De Gelas on July 21, 2016 8:45 AM ESTMemory Subsystem: Latency Measurements
There is no doubt about it: the performance of modern CPUs depends heavily on the cache subsystem. And some applications depend heavily on the DRAM subsystem too. We used LMBench in an effort to try to measure latency. Our favorite tool to do this, Tinymembench, does not support the POWER architecture yet. That is a pity, because it is a lot more accurate and modern (as it can test with two outstanding requests).
The numbers we looked at were "Random load latency stride=16 Bytes" (LMBench).
| Mem Hierarchy |
IBM POWER8 | Intel Broadwell Xeon E5-2640v4 DDR4-2133 |
Intel Broadwell Xeon E5-2699v4 DDR4-2400 |
| L1 Cache (cycles) | 3 | 4 | 4 |
| L2 Cache (cycles) | 13 | 12-15 | 12-15 |
| L3 Cache 4-8 MB(cycles) | 27-28 (8 ns) | 49-50 | 50 |
| 16 MB (ns) | 55 ns | 26 ns | 21 ns |
| 32-64 MB (ns) | 55-57 ns | 75-92 ns | 80-96 ns |
| Memory 96-128 MB (ns) | 67-74 ns | 90-91 ns | 96 ns |
| Memory 384-512 MB (ns) | 89-91 ns | 91-93 ns | 95 ns |
(Note that the numbers for Intel are higher than what we reported in our Cavium ThunderX review. The reason is that we are now using the numbers of LMBench and not those of Tinymembench.)
A 64 KB L1 cache with 4 read ports that can run at 4+ GHz speeds and still maintain a 3 cycle load latency is nothing less than the pinnacle of engineering. The L2 cache excels too, being twice as large (512 KB) and still offering the same latency as Intel's L2.
Once we get to the eDRAM L3 cache, our readings get a lot more confusing. The L3 cache is blistering fast as long as you only access the part that is closest to the core (8 MB). Go beyond that limit (16 MB), and you get a latency that is no less than 7 times worse. It looks like we actually hitting the Centaur chips, because the latency stays the same at 32 and 64 MB.
Intel has a much more predictable latency chart. Xeon's L3 cache needs about 50 cycles, and once you get into DRAM, you get a 90-96 ns latency. The "transistion phase" from 26 ns L3 to 90 ns DRAM is much smaller.
Comparatively, that "transition phase" seems relatively large on the IBM POWER8. We have to go beyond 128 MB before we get the full DRAM latency. And even then the Centaur chip seems to handle things well: the octal DDR-3 1333 MHz DRAM system delivers the same or even slightly better latency as the DDR4-2400 memory on the Xeon.
In summary, IBM's POWER8 has a twice as fast 8 MB L3, while Intel's L3 is vastly better in the 9-32 MB zone. But once you go beyond 32 MB, the IBM memory subsystem delivers better latency. At a significant power cost we must add, because those 4 memory buffers need about 64 Watts.
Facebook
Twitter
RSS
124 Comments
View All Comments
JohanAnandtech - Thursday, July 28, 2016 - link
Ah, you will have to wait for the improved P8 which is the first Power going after HPC :-)RISC is RISKY! - Tuesday, August 2, 2016 - link
I would support "Brutalizer". Every processor has its strength and weakness. If memory architecture is considered, for the same capacity, Intel is conjested memory, IBM is very distributed and Oracle-Sun is something in between. So Intel will always have memory B/W problem every way. IBM has memory efficiency problem. Oracle in theory doesn't have problem, but with 2 dimm per ch, that look like have problem. Oracle-Sun is for highly branched workload in the real world. Intel is for 1T/Core more of single threaded workloads and IBM is for mixed workloads with 2T-4T/Core priority. So supercomputing workloads will work fast on IBM now, compared to intel and sparc, while analytics and graph and other distributed will work faster on SPARC M7 and S7 (although S7 is resource limited). While for intel, a soft mix of applications and highly customized os is better. Leave the business decisions and the sales price. List prices are twice as much as sales price in the real world. These three processors (xeon e5v4, power8-9, sparc m7-s7) are thoroughly tuned for different work spaces with very little overlap. So there's no point in comparing them other than their specs. Its like comparing a falcon and a lion and a swordfish. Their environments are different even though all of them hunt. Thats in the real world. So benchmarks are not the real proof. We at the university of IITD have lots and lots of intel xeon e5v4, some P8 (10-15 single and dual sockets), and a very few (1-2 two socket M7 and 2 two socket S7). We run anything and every thing on any of these, we get our hands on. And this is the real world conclusion. So don't fight. Its a context centric supply.RISC is RISKY! - Tuesday, August 2, 2016 - link
of processors!rootvgnet - Friday, August 12, 2016 - link
Johan - interesting article, I enjoyed it - especially after I discovered how to get to the next page.As far as the comments go - 1) a good article will get a diverse response (from those with an open, read querying, mind.
2) I agree with those who, in other words are saying: "there is no 'one size fits all'." And my gut reaction is that you are providing a level of detail that assists in determining which platform/processor "fits my need"
Looking forward to part2.