Memory Subsystem: Latency

AMD chose to share a core design among mobile, desktop and server for scalability and economic reasons. The Core Complex (CCX) is still used in Rome like it was in the previous generation. 

What has changed is that each CCX communicates with the central IO hub, instead of four dies communicating in 4 node NUMA layout (This option is still available to use via the NPS4 switch, keeping each CCD local to its quadrant of the sIOD as well as those local memory controllers, avoiding hops between sIOD quadrants which encour a slight latency penalty). So as the performance of modern CPUs depends heavily on the cache subsystem, we were more than curious what kind of latency a server thread would see as it accesses more and more pages in the cache hierarchy. 

We're using our own in-house latency test. In particular what we're interested in publishing is the estimated structural latency of the processors, meaning we're trying to account for TLB misses and disregard them in these numbers, except for the DRAM latencies where latency measurements get a bit more complex between platforms, and we revert to full random figures.


(ns @ 3.4GHz)

(ns @ 3.2GHz)
Intel Xeon 8280

(ns @ 2.7GHz)
L1 Cache 32KB

4 cycles

4 cycles

4 cycles
L2 Cache 512KB

13 cycles

12 cycles

14 cycles
L3 Cache 16MB / CCX (4C)
256MB Total

~34 cycles (avg)
~10.27 ns
16MB / CCX (4C)
64MB Total

38.5MB / (28C)

~46 cycles (avg)

128MB Full Random
~122ns (NPS1)

~113ns (NPS4)



512MB Full Random
~134ns (NPS1)

~125ns (NPS4)

Update 2019/10/1: We've discovered inaccuracies with our originally published latency numbers, and have subsequently updated the article with more representative figures with a new testing tool.

Things get really interesting when starting to look at cache depths beyond the L2. Naturally Intel here this happens at 1MB while for AMD this is after 512KB, however AMD’s L2 has a speed advantage over Intel’s larger cache.

Where AMD has an ever more clearer speed advantage is in the L3 caches that are clearly significantly faster than Intel’s chips. The big difference here is that AMD’s L3’s here are only local to a CCX of 4 cores – for the EPYC 7742 this is now doubled to 16MB up from 8MB on the 7601.

Currently this is a two-edged sword for the AMD platforms: On one hand, the EPYC processors have significantly more total cache, coming in at a whopping 256MB for the 7742, quadruple the amount over the 64MB of the 7601, and a lot more than Intel’s platforms, which come in at 38.5MB for the Xeon 8180, 8176, 8280, and a larger 55MB for the Xeon E5-2699 v4.

The disadvantage for AMD is that while they have more cache, the EPYC 7742 rather consist of 16 CCX which all have a very fast 16 MB L3. Although the 64 cores are one big NUMA node now, the 64-core chip is basically 16x 4 cores, each with 16 MB L3-caches. Once you get beyond that 16 MB cache, the prefetchers can soften the blow, but you will be accessing the main DRAM.

A little bit weird is the fact that accessing data that resides at the same die (CCD) but is not within the same CCX is just as slow as accessing data is on a totally different die. This is because regardless of where the other CCX is, whether it is nearby on the same die or on the other side of the chip, the data access still has to go through the IF to the IO die and back again.

Is that necessarily a bad thing? The answer: most of the time it is not. First of all, in most applications only a low percentage of accesses must be answered by the L3 cache. Secondly, each core on the CCX has no less than 4 MB of L3 available, which is far more than the Intel cores have at their disposal (1.375 MB). The prefetchers have a lot more space to make sure that the data is there before it is needed.

But database performance might still suffer somewhat. For example, keeping a large part of the index in the cache improve performance, and especially OLTP accesses tend to quite random. Secondly the relatively slow communication over a central hub slow down synchronization communication. That is a real thing is shown by the fact that Intel states that the OLTP hammerDB runs 60% faster on a 28-core Intel Xeon 8280 than on EPYC 7601. We were not able to check it before the deadline, but it seems reasonable.

But for the vast majority of these high-end CPUs, they will be running many parallel applications, like running microservices, docker containers, virtual machines, map/reducing smaller chunks of data and parallel HPC Jobs. In almost all cases 16 MB L3 for 4 cores is more than enough.

Although come to think of it, when running an 8-core virtual machine there might be small corner cases where performance suffers a (little) bit.

In short, AMD leaves still a bit of performance on table by not using a larger 8-core CCX. We await to see what happens in future platforms.

Memory Subsystem: Bandwidth Latency Part Two: Beating The Prefetchers
Comments Locked


View All Comments

  • nathanddrews - Wednesday, August 7, 2019 - link

    Binned for OC? We'll find out soon enough!
  • DigitalFreak - Thursday, August 8, 2019 - link

    At this point it looks like all TR will get your is "official" ECC support and more PCIe lanes. Maybe cheaper motherboards than EPYC.
  • willis936 - Thursday, August 8, 2019 - link

    Half the memory lanes (this is a big one), half the pcie lanes, max of 1 socket per mobo. Those are important features for datacenter customers and their absence from threadripper makes threadripper less desirable than epyc in the datacenter.
  • rocky12345 - Thursday, August 8, 2019 - link

    Yes but Threadripper is made for high end desktops for video editing etc etc and some gaming. I do not see the big data center guys going after TR all that much. Yes you may see some of the TR go there but that is not what TR is made for that is why we have EPYC & XEON CPU's.

    I do have to agree though where some said where does TR fit in price wise since we are going to have a 16/32 main stream desktop CPU shortly from AMD. I do also think this time around the 32/64 3990 TR will be 10x better than the older 2990 TR just from the memory controller not being in each CPU complex and in the 2990x because of bandwidth and latency from the memory performance really suffered when all cores were being used. On the 3990x (or whatever it will be called) this should not be an issue. If AMD is smart they will not release a 64/128 3000 series TR since it would have to be priced to far out of reach for even the most techy guy with money and the only ones that would have them would be review sites and YT reviewers and that would be only because them got them sent for free for reviews. 32/64 and the better memory performance as a whole for the new chips would be more than enough to make the 32/64 TR 3990x an instant success. Just my opinion of coarse and AMD will probably do something stupid and release a higher core count TR series CPU that next to no one will be able to afford just to be able to say hey we got the best high end CPU on the planet but to bad no one is gonna buy them because the price is to high but we have the best so who cares.
  • rocky12345 - Thursday, August 8, 2019 - link

    Oops dammit forgot to make paragraph's did not mean to have it all bunched up like that.
  • Mark Rose - Friday, August 9, 2019 - link

    Why wouldn't they release a 64 core Threadripper? Assuming they double the price of the 32 core, it would be $3400. That's affordable to a lot of people working in tech, and should be affordable to just about any business that has employees waiting on their 32 core Threadripper. AMD would sell a ton.

    That being said, I wouldn't personally buy one as I don't have a need. I'd be more likely buy a 16 core 3000 series Threadripper myself.
  • Manch - Friday, August 9, 2019 - link

    Higher Clocks
  • sor - Wednesday, August 7, 2019 - link

    It will be a feature/packaging thing. The motherboards would be TR4 and feature enthusiast features, overclocked memory, etc, not highly reliable server oriented boards. The processors themselves might be fairly comparable to their EPYC counterparts, as some Xeons were occasionally comparable to their desktop ones.
  • close - Thursday, August 8, 2019 - link

    TR was supposed to be a stopgap measure until the consume Ryzen range stretched high enough and the server EPYC range stretched low enough. I guess there is a place for further differentiation especially in terms of the platform (motherboard) used, where you have server like CPU on a more consumer like MB to create basically a workstation. Maybe OC will also fit in here.
  • Death666Angel - Friday, August 9, 2019 - link

    "TR was supposed to be a stopgap measure" where can I see AMD stating that? Considering Intel has fared pretty well with the consumer/HEDT/server differentiation, I don't think AMD needs to axe TR. I don't see them giving us EPYC with OC functions and 8 memory channles seems overkill for 16 or 32 desktop cores. I also haven't seen a statement to the effect you claim, so I highly doubt it at the moment.

Log in

Don't have an account? Sign up now