Memory Subsystem: Latency

AMD chose to share a core design among mobile, desktop and server for scalability and economic reasons. The Core Complex (CCX) is still used in Rome like it was in the previous generation. 

What has changed is that each CCX communicates with the central IO hub, instead of four dies communicating in 4 node NUMA layout (This option is still available to use via the NPS4 switch, keeping each CCD local to its quadrant of the sIOD as well as those local memory controllers, avoiding hops between sIOD quadrants which encour a slight latency penalty). So as the performance of modern CPUs depends heavily on the cache subsystem, we were more than curious what kind of latency a server thread would see as it accesses more and more pages in the cache hierarchy. 

We're using our own in-house latency test. In particular what we're interested in publishing is the estimated structural latency of the processors, meaning we're trying to account for TLB misses and disregard them in these numbers, except for the DRAM latencies where latency measurements get a bit more complex between platforms, and we revert to full random figures.

Mem
Hierarchy
AMD EPYC 7742
DDR4-3200

(ns @ 3.4GHz)
AMD EPYC 7601
DDR4-2400

(ns @ 3.2GHz)
Intel Xeon 8280
DDR-2666

(ns @ 2.7GHz)
L1 Cache 32KB

4 cycles
1.18ns
32KB

4 cycles
1.25ns
32KB

4 cycles
1.48ns
L2 Cache 512KB

13 cycles
3.86ns
512KB

12 cycles
3.76ns
1024KB

14 cycles
5.18ns
L3 Cache 16MB / CCX (4C)
256MB Total

~34 cycles (avg)
~10.27 ns
16MB / CCX (4C)
64MB Total

 
38.5MB / (28C)
Shared

~46 cycles (avg)
~17.5ns
DRAM

128MB Full Random
~122ns (NPS1)

~113ns (NPS4)

~116ns

~89ns
DRAM

512MB Full Random
~134ns (NPS1)

~125ns (NPS4)
 
~109ns

Update 2019/10/1: We've discovered inaccuracies with our originally published latency numbers, and have subsequently updated the article with more representative figures with a new testing tool.

Things get really interesting when starting to look at cache depths beyond the L2. Naturally Intel here this happens at 1MB while for AMD this is after 512KB, however AMD’s L2 has a speed advantage over Intel’s larger cache.

Where AMD has an ever more clearer speed advantage is in the L3 caches that are clearly significantly faster than Intel’s chips. The big difference here is that AMD’s L3’s here are only local to a CCX of 4 cores – for the EPYC 7742 this is now doubled to 16MB up from 8MB on the 7601.

Currently this is a two-edged sword for the AMD platforms: On one hand, the EPYC processors have significantly more total cache, coming in at a whopping 256MB for the 7742, quadruple the amount over the 64MB of the 7601, and a lot more than Intel’s platforms, which come in at 38.5MB for the Xeon 8180, 8176, 8280, and a larger 55MB for the Xeon E5-2699 v4.

The disadvantage for AMD is that while they have more cache, the EPYC 7742 rather consist of 16 CCX which all have a very fast 16 MB L3. Although the 64 cores are one big NUMA node now, the 64-core chip is basically 16x 4 cores, each with 16 MB L3-caches. Once you get beyond that 16 MB cache, the prefetchers can soften the blow, but you will be accessing the main DRAM.

A little bit weird is the fact that accessing data that resides at the same die (CCD) but is not within the same CCX is just as slow as accessing data is on a totally different die. This is because regardless of where the other CCX is, whether it is nearby on the same die or on the other side of the chip, the data access still has to go through the IF to the IO die and back again.

Is that necessarily a bad thing? The answer: most of the time it is not. First of all, in most applications only a low percentage of accesses must be answered by the L3 cache. Secondly, each core on the CCX has no less than 4 MB of L3 available, which is far more than the Intel cores have at their disposal (1.375 MB). The prefetchers have a lot more space to make sure that the data is there before it is needed.

But database performance might still suffer somewhat. For example, keeping a large part of the index in the cache improve performance, and especially OLTP accesses tend to quite random. Secondly the relatively slow communication over a central hub slow down synchronization communication. That is a real thing is shown by the fact that Intel states that the OLTP hammerDB runs 60% faster on a 28-core Intel Xeon 8280 than on EPYC 7601. We were not able to check it before the deadline, but it seems reasonable.

But for the vast majority of these high-end CPUs, they will be running many parallel applications, like running microservices, docker containers, virtual machines, map/reducing smaller chunks of data and parallel HPC Jobs. In almost all cases 16 MB L3 for 4 cores is more than enough.

Although come to think of it, when running an 8-core virtual machine there might be small corner cases where performance suffers a (little) bit.

In short, AMD leaves still a bit of performance on table by not using a larger 8-core CCX. We await to see what happens in future platforms.

Memory Subsystem: Bandwidth Latency Part Two: Beating The Prefetchers
POST A COMMENT

184 Comments

View All Comments

  • npz - Thursday, August 8, 2019 - link

    Let me tell you: Lockheed? Linux Raytheon? Linux + Solaris, etc Aerospace Corp? Linux. You won't find Windows ANYWHERE but desktops and laptops. All confidential info and actual aerospace work resides on Linux and legacy Solaris systems.

    Big gov or Telcom? Linux and legacy Solaris. HPC? ALL -- ALL Linux. Big database? All linux. Storage? Linux. Virtualization needs? All Linux. What does Rackspace run? I've worked on and for (2nd hand as dev. engineering support) big -- I mean giant state infrastructure (think electrical grid), financial (banks, trading) and aerospace and computing corp enviornement and every single one runs Linux and/or legacy Solaris (after they moved from AIX usually, now they're moving from Solaris to Linux)

    Again you mention Active Directory -- what did I say? I stated: "companies use Windows because of familiarity for desktop support such as Active Directory for domains, but none of major critical data center centric, HPC, military, infrastructure are running Windows. Most especially not with EPYC since the Windows scheduler is broken. "

    Go outside any Active Directory needs and servers that needs to support RDP or Windows dekstops you will see Windows isn't used anywhere else
    Reply
  • npz - Thursday, August 8, 2019 - link

    examples:

    Aerospace Corp:
    https://www.glassdoor.com/job-listing/space-vehicl...
    "Space Vehicle Modeling & Simulation Engineer"
    > Familiarity with Linux and high-performance computing environments

    https://www.glassdoor.com/job-listing/data-scienti...
    "Data Scientist"
    > Working knowledge of Unix/Linux operating systems

    Citi Group:
    https://www.glassdoor.com/job-listing/infrastructu...
    "Infrastructure Solutions Engineer"
    > Must have good understanding of Linux and networking concepts

    https://www.glassdoor.com/job-listing/senior-data-...
    "Senior Data Engineer - Hadoop, VP"
    > Hands on experience with open source software platforms Linux
    Reply
  • Oliseo - Thursday, August 8, 2019 - link

    "Completely baseless claims. I have worked large scale government and military IT and Windows servers are the most common by far. "

    Working on reception will most likely be the reason for coming to that assumption, so we can go easy on you. It's an easy mistake to make.
    Reply
  • James5mith - Tuesday, August 13, 2019 - link

    FreeIPA works pretty well for us. Reply
  • FunBunny2 - Thursday, August 8, 2019 - link

    if you run an industrial strength RDBMS (Oracle, DB2, and even SS) you run on some variant of linux. Reply
  • AlyxSharkBite - Friday, August 9, 2019 - link

    NPZ, I’d say you should do some fact checking before you make a broad statement. AWS allows you to choose your OS there’s a lot of Windows Servers there. The Department of Defense uses a lot of Windows Servers see a 2016 article (https://blogs.windows.com/windowsexperience/2016/0...

    Also here’s a handy graph on server os market share a little old but numbers won’t change that much. Notice *nix only have about 20% of the market. The other 80% is Windows Server

    https://static.spiceworks.com/shared/blog_entry_in...
    Reply
  • Bonez0r - Wednesday, August 14, 2019 - link

    Wasn't the Windows scheduler fixed in a Windows update two months ago?
    https://www.reddit.com/r/Amd/comments/bz6egc/windo...
    Reply
  • Jorsher - Tuesday, September 3, 2019 - link

    NPZ - what are you talking about? I work on a military network spanning 10+ countries and tens of thousands of users and computers. The enterprise runs on Windows, with specific cases running in RHEL. I'm a fan of *nix but quit drinking the kool-aid. Windows still owns a rather large portion of the market and I don't see it changing any time soon. Reply
  • Deshi! - Thursday, August 8, 2019 - link

    I work as an application engineer for a major global finance company that develops and hosts banking and e-commerce software used by banks and major shopping outlets. 90% of all our servers are either Linux or AIX mainly running websphere or standalone Java instances. We only have a handful of Windows servers, mainly for stuff like active directory and Outlook/ SharePoint. So yeah allot of it depends on the use case, but allot of the big boys do use Linux or AIX. It's cheaper and performs better for these use cases. Reply
  • cyberguyz - Thursday, August 8, 2019 - link

    I guess we all have to ask ourselves, who are the customers that would benefit most from a 64-core, 128 gen 4 PCIe processors? SMB or huge customers that would shell out many millions of $$$ for their middleware & backend systems? @Deshi! I or one of my L3 colleagues an L3 engineer contacted by your global finance company to fix Websphere problems some years back ;) Reply

Log in

Don't have an account? Sign up now