AMD Rome Second Generation EPYC Review: 2x 64-core Benchmarked
by Johan De Gelas on August 7, 2019 7:00 PM ESTMemory Subsystem: TinyMemBench
We doublechecked our LMBench numbers with Andrei's custom memory latency test.
The latency tool also measures bandwidth and it became clear than once we move beyond 16 MB, DRAM is accessed. When Andrei compared with our Ryzen 9 3900x numbers, he noted:
The prefetchers on the Rome platform don't look nearly as aggressive as on the Ryzen unit on the L2 and L3
It would appear that parts of the prefetchers are adjusted for Rome compared to Ryzen 3000. In effect, the prefetchers are less aggressive than on the consumer parts, and we believe that AMD has made this choice by the fact that quite a few applications (Java and HPC) suffer a bit if the prefetchers take up too much bandwidth. By making the prefetchers less aggressive in Rome, it could aid performance in those tests.
While we could not retest all our servers with Andrei's memory latency test by the deadline (see the "Murphy's Law" section on page 5), we turned to our open source TinyMemBench benchmark results. The source was compiled for x86 with GCC and the optimization level was set to "-O3". The measurement is described well by the manual of TinyMemBench:
Average time is measured for random memory accesses in the buffers of different sizes. The larger the buffer, the more significant the relative contributions of TLB, L1/L2 cache misses, and DRAM accesses become. All the numbers represent extra time, which needs to be added to L1 cache latency (4 cycles).
We tested with dual random read, as we wanted to see how the memory system coped with multiple read requests.
The graph shows how the larger L3 cache of the EPYC 7742 resulting in a much lower latency between 4 and 16 MB, compared to the EPYC 7601. The L3 cache inside the CCX is also very fast (2-8 MB) compared to Intel's Mesh (8280) and Ring topologies (E5).
However, once we access more than 16 MB, Intel has a clear advantage due to the slower but much larger shared L3 cache. When we tested the new EPYC CPUs in a more advanced NUMA setting (with NPS = 4 setting, meaning 4 nodes per socket), latency at 64 MB lowered from 129 to 119. We quote AMD's engineering:
In NPS4, the NUMA domains are reported to software in such a way as it chiplets always access the near (2 channels) DRAM. In NPS1 the 8ch are hardware-interleaved and there is more latency to get to further ones. It varies by pairs of DRAM channels, with the furthest one being ~20-25ns (depending on the various speeds) further away than the nearest. Generally, the latencies are +~6-8ns, +~8-10ns, +~20-25ns in pairs of channels vs the physically nearest ones."
So that also explains why AMD states that select workloads achieve better performance with NPS = 4.
Facebook
Twitter
RSS
180 Comments
View All Comments
JoeBraga - Wednesday, August 14, 2019 - link
It can happen if Intel uses the new archtecture Sunny Cove and MCM/Chiplet design instead of Monolithic DesignSanX - Thursday, August 15, 2019 - link
7zip is not a legacy test, it is important for anyone who sends big data over always damn slow network. Do you know all those ZIPs, GZs and other zippers which people mostly use, compress with turtle speeds as low as 20 MB/s even on supercomputers ? The 7Zip though parallelizes that nicely. So do not diminish this good test calling it "legacy"imaskar - Friday, August 16, 2019 - link
7zip is a particular program, doing LZMA in parallel, that's why it is faster that lets say gzip. But on server you often do not want to parallel things, because other cores are doing other jobs and switching is costly. There are a lot of compressing algorithms which are better in certain situations. LZMA rarely fits. More often it is it's LZ4 or zstd for "generate once, consume many" or basic gzip (DEFLATE) for "generate once, consume once". Yes, you would be surprised, but the very basic 30 years old DEFLATE is still the king if you care for sum of compress, send, decompress AND your nodes are inside one datacenter (which is most of the times).SanX - Thursday, August 15, 2019 - link
What you can say about Ian's own test he developed to demonstrate avx512 speed boost which shows some crazy up to 3-4x or more speedups ? Does your test of Molecular Dynamics tell that Ian's test mostly irrelevant for such huge improvement of speed of the real life complex programs?imaskar - Friday, August 16, 2019 - link
Probably because you can't use ONLY avx512. You still need regular things like jumps and conditions. And this is only the best case. Usually you also need to process part of the vector differently. For example, your vector has size 20, but your width is 16. You either do another vector pass, or 4 regular computations. Often second thing is faster or just the only option.realbabilu - Sunday, August 18, 2019 - link
Most of finite element software use Intel mkl to get every juice power spec of processor.it works for Intel ones not for amdAmd math kernel not heavily programmed, otnwaa just for Linux.
Other third party like gotoblas openblas still trying hard to detect cache and type for zen2.
I mean for workstation floating point still hard for amd.
peevee - Monday, August 19, 2019 - link
Prices per core-GHz:EPYC 7742 $48.26
EPYC 7702 $50.39
EPYC 7642 $43.25
EPYC 7552 $38.12
EPYC 7542 $36.64
EPYC 7502 $32.50
EPYC 7452 $26.93
EPYC 7402 $26.53
EPYC 7352 $24.46
EPYC 7302 $20.38
EPYC 7282 $14.51
EPYC 7272 $17.96
EPYC 7262 $22.46
EPYC 7252 $19.15
Value in this 7282 is INSANE.
peevee - Tuesday, August 20, 2019 - link
"Even though our testing is not the ideal case for AMD (you would probably choose 8 or even 16 back-ends), the EPYC edges out the Xeon 8176. Using 8 JVMs increases the gap from 1% to 4-5%."1%? 36917 / 27716 = 1.3319...
33%. Without 8 JVMs.
KathyMilligan - Wednesday, August 21, 2019 - link
University of Illinois Urbana-Champaign is very good university. I am too poorly prepared for this level of education. But I'm getting ready. I read a lot of articles and books, communicate with many smart former students of this university. I also buy research papers on site and this gives me a lot of useful information, which is not so easy to find on the Internet.YB1064 - Wednesday, August 28, 2019 - link
Looks like Intel has been outclassed, out-priced and completely out-maneuvered by AMD. What a disaster!