AMD Rome Second Generation EPYC Review: 2x 64-core Benchmarked
by Johan De Gelas on August 7, 2019 7:00 PM ESTMemory Subsystem: TinyMemBench
We doublechecked our LMBench numbers with Andrei's custom memory latency test.
The latency tool also measures bandwidth and it became clear than once we move beyond 16 MB, DRAM is accessed. When Andrei compared with our Ryzen 9 3900x numbers, he noted:
The prefetchers on the Rome platform don't look nearly as aggressive as on the Ryzen unit on the L2 and L3
It would appear that parts of the prefetchers are adjusted for Rome compared to Ryzen 3000. In effect, the prefetchers are less aggressive than on the consumer parts, and we believe that AMD has made this choice by the fact that quite a few applications (Java and HPC) suffer a bit if the prefetchers take up too much bandwidth. By making the prefetchers less aggressive in Rome, it could aid performance in those tests.
While we could not retest all our servers with Andrei's memory latency test by the deadline (see the "Murphy's Law" section on page 5), we turned to our open source TinyMemBench benchmark results. The source was compiled for x86 with GCC and the optimization level was set to "-O3". The measurement is described well by the manual of TinyMemBench:
Average time is measured for random memory accesses in the buffers of different sizes. The larger the buffer, the more significant the relative contributions of TLB, L1/L2 cache misses, and DRAM accesses become. All the numbers represent extra time, which needs to be added to L1 cache latency (4 cycles).
We tested with dual random read, as we wanted to see how the memory system coped with multiple read requests.
The graph shows how the larger L3 cache of the EPYC 7742 resulting in a much lower latency between 4 and 16 MB, compared to the EPYC 7601. The L3 cache inside the CCX is also very fast (2-8 MB) compared to Intel's Mesh (8280) and Ring topologies (E5).
However, once we access more than 16 MB, Intel has a clear advantage due to the slower but much larger shared L3 cache. When we tested the new EPYC CPUs in a more advanced NUMA setting (with NPS = 4 setting, meaning 4 nodes per socket), latency at 64 MB lowered from 129 to 119. We quote AMD's engineering:
In NPS4, the NUMA domains are reported to software in such a way as it chiplets always access the near (2 channels) DRAM. In NPS1 the 8ch are hardware-interleaved and there is more latency to get to further ones. It varies by pairs of DRAM channels, with the furthest one being ~20-25ns (depending on the various speeds) further away than the nearest. Generally, the latencies are +~6-8ns, +~8-10ns, +~20-25ns in pairs of channels vs the physically nearest ones."
So that also explains why AMD states that select workloads achieve better performance with NPS = 4.
180 Comments
View All Comments
MDD1963 - Friday, August 9, 2019 - link
of the '1800 servers', how many of those are virtual, just out of curiosity? ('1800 servers' is not quite as impressive if there were, for example, 10 hosts w/ 180 Windows VMs each, for example) U.S.A.F offices are still mostly Windows 10...I'd suspect the are datacenters at each base having a large Windows Server presence as well.. (But, we used Redhat onboard assorted recce aircraft for many years now....; which seems stupid in light of the fact they could easily use CentOS for free; presumably, a Senator's family members work at Redhat, and enjoy the large income from support contracts)eek2121 - Friday, August 9, 2019 - link
Employers use Windows, data centers use Linux. All the major cloud providers, including Microsoft, have reported that Linux has the highest market share.gylgamesh - Friday, August 9, 2019 - link
Could you please specify what kind of servers those are and what tasks do they perform, and also which MS Windows OS are they using? Thanks.Slickest - Thursday, August 8, 2019 - link
I work for one of the largest colleges in the nation, and 90% of our servers are Windows.69369369 - Thursday, August 8, 2019 - link
"LUL"Go back to Twitch kiddo.
azfacea - Thursday, August 8, 2019 - link
and what would u do if i dont?Oliseo - Thursday, August 8, 2019 - link
Tell your mum you're up late again and she will force you to go outside, and we all know how much your dislike having to do that.prophet001 - Monday, August 12, 2019 - link
I don't get the hate for windows server? How you gonna run a domain and active directory in linux?CaedenV - Wednesday, August 21, 2019 - link
I use to work in schools, and a few that could not afford Windows Server would run AD through some Linux application. It was not exactly full-featured, but it worked well enough for 'free' solution.deltaFx2 - Wednesday, August 7, 2019 - link
@Pancakes: Are you kidding? 1T perf is at par with Skylake. Windows licenses per core. Why would anyone buy a SKU with more cores than they need? And if they did do that, why would they not run them on a VM? Do these people also buy more racks than they need and run windows just for the fun of it?