AMD Zen 2 Microarchitecture Analysis: Ryzen 3000 and EPYC Rome
by Dr. Ian Cutress on June 10, 2019 7:22 PM EST- Posted in
- CPUs
- AMD
- Ryzen
- EPYC
- Infinity Fabric
- PCIe 4.0
- Zen 2
- Rome
- Ryzen 3000
- Ryzen 3rd Gen
CCX Size
Moving down in node size brings up a number of challenges in the core and beyond. Even disregarding power and frequency, the ability to put structures into silicon and then integrate that silicon into the package, as well as providing power to the right parts of the silicon through the right connections becomes an exercise in itself. AMD gave us some insight into how 7nm changed some of its designs, as well as the packaging challenges therein.
A key metric given up by AMD relates to the core complex: four cores, the associated core structures, and then L2 and L3 caches. With 12nm and the Zen+ core, AMD stated that a single core complex was ~60 square millimeters, which separates into 44mm2 for the cores and 16mm2 for the 8MB of L3 per CCX. Add two of these 60mm2 complexes with a memory controller, PCIe lanes, four IF links, and other IO, and a Zen+ zeppelin die was 213 mm2 in total.
For Zen 2, a single chiplet is 74mm2, of which 31.3 mm2 is a core complex with 16 MB of L3. AMD did not breakdown this 31.3 number into cores and L3, but one might imagine that the L3 might be approaching 50% of that number. The reason the chiplet is so much smaller is that it doesn’t need memory controllers, it only has one IF link, and has no IO, because all of the platform requirements are on the IO die. This allows AMD to make the chiplets extremely compact. However if AMD intends to keep increasing the L3 cache, we might end up with most of the chip as L3.
Overall however, AMD has stated that the CCX (cores plus L3) has decreased in size by 47%. That is showing great scaling, especially if the +15% raw instruction throughput and increased frequency comes into play. Performance per mm2 is going to be a very exciting metric.
Packaging
With Matisse staying in the AM4 socket, and Rome in the EPYC socket, AMD stated that they had to make some bets on its packaging technology in order to maintain compatibility. Invariably some of these bets end up being tradeoffs for continual support, however AMD believes that the extra effort has been worth the continued compatibility.
One of the key points AMD spoke about with relation to packaging is how each of the silicon dies are attached to the package. In order to enable a pin-grid array desktop processor, the silicon has to be affixed to the processor in a BGA fashion. AMD stated that due to the 7nm process, the bump pitch (the distance between the solder balls on the silicon die and package) reduced from 150 microns on 12nm to 130 microns on 7nm. This doesn’t sound like much, however AMD stated that there are only two vendors in the world with technology sufficient to do this. The only alternative would be to have a bigger bit of silicon to support a larger bump pitch, ultimately leading to a lot of empty silicon (or a different design paradigm).
One of the ways in order to enable the tighter bump pitch is to adjust how the bumps are processed on the underside of the die. Normally a solder bump on a package is a blob/ball of lead-free solder, relying on the physics of surface tension and reflow to ensure it is consistent and regular. In order to enable the tighter bump pitches however, AMD had to move to a copper pillar solder bump topology.
In order to enable this feature, copper is epitaxially deposited within a mask in order to create a ‘stand’ on which the reflow solder sits. Due to the diameter of the pillar, less solder mask is needed and it creates a smaller solder radius. AMD also came across another issue, due to its dual die design inside Matisse: if the IO die uses standard solder bump masks, and the chiplets use copper pillars, there needs to be a level of height consistency for integrated heat spreaders. For the smaller copper pillars, this means managing the level of copper pillar growth.
AMD explained that it was actually easier to manage this connection implementation than it would be to build different height heatspreaders, as the stamping process used for heatspreaders would not enable such a low tolerance. AMD expects all of its 7nm designs in the future to use the copper pillar implementation.
Routing
Beyond just putting the silicon onto the organic substrate, that substrate has to manage connections between the die and externally to the die. AMD had to increase the number of substrate layers in the package to 12 for Matisse in order to handle the extra routing (no word on how many layers are required in Rome, perhaps 14). This also becomes somewhat complicated for single core chiplet and dual core chiplet processors, especially when testing the silicon before placing it onto the package.
From the diagram we can clearly see the IF links from the two chiplets going to the IO die, with the IO die also handling the memory controllers and what looks like power plane duties as well. There are no in-package links between the chiplets, in case anyone was still wondering: the chiplets have no way of direct communication – all communication between chiplets is handled through the IO die.
AMD stated that with this layout they also had to be mindful of how the processor was placed in the system, as well as cooling and memory layout. Also, when it comes to faster memory support, or the tighter tolerances of PCIe 4.0, all of this also needs to be taken into consideration as provide the optimal path for signaling without interference from other traces and other routing.
Facebook
Twitter
RSS
216 Comments
View All Comments
nonoverclock - Wednesday, June 12, 2019 - link
It's related to platform power management.wurizen - Wednesday, June 12, 2019 - link
"Raw Memory Latency" graph shows 69ns for for 3200 and 3600 Mhz RAM. This "69ns" is irrelevant, right? Isn't the "high latency" associated with Ryzen and IF due to "Cross-CCX-Memory-Latency? This is suppose to be ~110ns at 3200 Mhz RAM as tested by PCPER/etc.... This in my experiences causes "micro-stuttering" in games like BO3/BF4/etc.... And, a "Ryzen-micro-stutter/pause" is different than a micro-stutter/pause associated with Intel. With Intel the micro-stutter/pause happens in BFV, for example, but they happen once or twice per match. With Ryzen, not only is the quality/feeling of the "micro-stutter/pause" different (seems worst), but it is constant throughout the match. One gets a feeling that it is not server-side, GPU side, nor WIndows 10 side. But, CPU-side issue... Infinity Fabric side. So, now Inifinity Fabric 2 is out. Is it 2.0 as in better? No more high latency? Is that 69ns Cross-CCX-memory latency? Why is AMD and Tech sites like Anand so... like... not talking about this?igavus - Wednesday, June 12, 2019 - link
You are misattributing things here. Your stutter is most def. not caused by memory access latency variations. For it to be visible on even an 144Hz monitor with the game running at the native rate, the differences would have to be obscenely high. That's just unrealistic.Not that it helps to determine what is causing your issues, but that's not it.
wurizen - Wednesday, June 12, 2019 - link
What?wurizen - Wednesday, June 12, 2019 - link
Maybe, you guys don't know what Cross-CCX-Memory-Latyency is... my main goal of commenting was what that SLIDE showing "Raw Memory Latency" refers to? Is it Inter-core-memory or Intra-core-memory (intra-core is the same as cross-ccx-memory)...inter-core memory is data being shuffled within the cores in a CCX module. Ryzen and Ryzen + had two CCX modules with 4 cores each, totaling 8 cores for the 2700x, as an example. If, the memory/data is traveling in the same CCX, the latency is fine and is even faster than Intel. This was true with Ryzen and Ryzen +.
The issue is when data and memory is being shuffled between the CCX modules, and when traversing the so called "Infinity Fabric." Intel uses a Ring Bus and doesn't have an equivalent measurement and data. Intel does have MESH with the x299 which is similar-ish to AMD's CCX and IF. But, Intel Mesh latency is lower (I think. But havent dug around since I dont care about it since I cant afford it)....
So... that is what Cross-CCX-memory-latency is... and that SLIDE shown on this article... WTF does that refer to? 69ns is similar to Intel Ring Bus memory latency, which have shown to be fast enough and is the standard in regards to latency that won't cause issues...
So... as PCPER tested, Ryzen Infinity Fanri 1.0 has a cross-ccx-latnecy to be around 110ns... and I stand my ground (its not bios/reinstallwindows/or windows scheduler/or user-error/or imperceptible/or a misunderstanding / or a mis-atribution (I think)) that it was the reason why I suffered "micro-pauses/stutters in some games. I had two systems at the time (3700k and R7-1700x) and so I was able to diagnose/observe/interpret what was happening....
Also.. I would like to add that the "Ryzen Micro-stutter-Pause" FEELS/LOOKS/BEHAVES different... weird, right?
deltaFx2 - Thursday, June 13, 2019 - link
You might "stand your ground" but that doesn't make it true. First of all, it's pretty clear you don't understand what you're talking about. Intel's Mesh is NOTHING like AMD's CCX. Intel Mesh is an alternative interconnect to ring bus; mesh scales better to many cores relative to ring. In theory mesh should be faster but for whatever reason intel's memory latency on skylake X parts are quite bad relative to skylake client (i.e. no bueno for gaming). I recall 70ns-ish for Skylake X vs 60ns for the Skylake client.Cross CCX memory latency should not matter unless you have shared memory across threads that span CCXs. Games don't need that many threads: 8 is overkill in many cases and each CCX can comfortably handle 8. Unless you pinned threads to cores and ran an experiment that conclusively showed that the issue was inter-ccx latency (I doubt it), your standing ground doesn't mean much. One could just as well argue that the microstutter was due to driver issues or other software/bios issues. Zen has been around for quite some time and if this was a widespread problem, we'd know.
wurizen - Friday, June 14, 2019 - link
Well, I did mention "similar-ish" of Mesh to Infinity Fabric. It's meshy. And, i guess, you get "comraderie" points for calling me out as "pretty clear you don't understand what you're talking about." That hurts, man! :("In theory... Mesh should be faster..." nice way to switch subjects, bruh. yeh, i can throw some at ya, bruh! what?
Cross-CCX-High-Memory-Latency DOES MATTER!
You know why? Because a game shuffles data randomly. It doesn't know that traversing said Data from Core 0 (residing in CCX 1) to Core 3 (in CCX 2) via Infinity Fabric means that there is a latency penalty.
Bruh
deltaFx2 - Friday, June 14, 2019 - link
Actually, no, you're wrong about the mesh. Intel has a logically unified L3 cache; i.e. any core can access any slice of the L3, or even one core can use the entire L3 for itself. AMD has a logically distributed L3 cache which means only the cores from the CCX can access its cache. You simply cannot have core 3 (CCX 0) fetch a line into CCX1's cache. The tradeoff is that the distributed L3 is much faster than the logically unified one but the logically unified one obviously offers better hit rates and does not suffer from sharing issues."Cross-CCX-High-Memory-Latency DOES MATTER!" Yes it does, no question about that. It matters when you have lock contention or shared memory that spans CCXs. In order to span CCXs, you should be using more than 8 threads (4 cores to a CCX, 2 threads per core). I don't think games are _that_ multithreaded. This article mentions a Windows 10 patch to ensure that threads get assigned to the same CCX before going to the adjacent one. It can be a problem for compute-intensive applications (y'know, real work), but games? I doubt it, and you should be able to fix it easily by pinning threads to cores in the same CCX.
deltaFx2 - Friday, June 14, 2019 - link
"shared memory that spans CCXs." -> shared DIRTY memory. i.e. core 8 writes data, core 0 wants to read. All other kinds of sharing are a non-issue. Each CCX gets a local copy of the data.wurizen - Friday, June 14, 2019 - link
Why do you keep on blabbing on about this? Are you trying to fix some sort of muscle?