Silicon, Glue, & NUMA Too

In the Ryzen family, AMD designed an 8 core silicon die known as a Zeppelin die. This consisted of two core complexes (CCX) of four cores each, with each CCX having access to 8 MB of L3 cache. The Zeppelin die had access to two DRAM channels, and was fixed with 16 PCIe lanes for add-in cards. With Threadripper, AMD has doubled up the silicon.

If you were to delid a Threadripper CPU, you would actually see four silicon dies, similar to what an EPYC processor would have, making Threadripper a Multi Core Module (MCM) design. Two of these are reinforcing spacers – empty silicon with no use other than to help distribute the weight of the cooler and assist in cooling. The other two dies (in opposite corners for thermal performance and routing) are basically the same Zeppelin dies as Ryzen, containing eight cores each and having access to two memory channels each. They communicate through Infinity Fabric, which AMD lists as 102 GB/s die-to-die bandwidth (full duplex bidirectional), along with 78ns to reach the near memory (DRAM connected to the same die) and 133ns to reach the far memory (DRAM on another die). We confirmed those numbers on DDR4-2400 memory, also achieving 65 ns and 108 ns respectively using DDR4-3200. 


Despite this AMD slide showing two silicon dies, there are four units of silicon in the package. Only two of the dies are active, so AMD has 'simplified' the diagram'

By comparison, EPYC lists die-to-die bandwidth as 42.6 GB/s at DDR4-2666. This is because EPYC runs fabric links to three dies internally and one die externally (on the next socket), which maximizes all the links available. The dies in Threadripper only have to communicate with one other die, so has more flexibility. To that extent, we’re under the impression that Threadripper is using two of these links at 10.4 GT/s using the following method:

  • Die to Die for EPYC is quoted as 42.6 GB/s at DDR4-2667
  • Die to Die for Threadripper is quoted as 102.2 GB/s at DDR4-3200
  • 42.6 GB/s * 2 links * 3200/2667 = 102.2 GB/s
  • 42.6 GB/s * 3 links * 3200/2667 at 8.0 GT/s = 115.8 GB/s (too high)
  • 42.6 GB/s * 3 links * 3200/2667 at 6.4 GT/s = 92.6 GB/s (too low)

This configuration for AMD is essentially what the industry calls a NUMA configuration: non-uniform memory access. If left as it, it means that code cannot rely on a regular (and low) latency between requesting something from DRAM and receiving it. This can be an issue for high-performance code, which is why some software is designed NUMA-aware, so that it can intelligently pin the memory it needs to the closest DRAM controller, lowering potential bandwidth but prioritizing latency.

NUMA is nothing new in the x86 space. Once CPUs began shipping with on-die memory controllers rather than using an off-die memory controller in the Northbridge, NUMA became an inherent part of multi-socket systems. In this respect AMD was the leader here right from the start, as they beat Intel to on-die memory controllers for x86 CPUs by years. So AMD has been working with NUMA for years, and similarly NUMA has been the state of affairs for Intel's multi-socket server systems for almost a decade.

What's new with Threadripper however is that NUMA has never been a consumer concern. MCM consumer CPUs have been few and far between, and we'd have to go all the way back to the Core 2 Quad family to find a CPU with cores on multiple dies, which was a design that predates on-die memory controllers for Intel. So with Threadripper, this is the very first time that consumers – even high-end consumers – have been exposed to NUMA.

But more importantly, consumer software has been similarly unexposed to NUMA, so almost no software is able to take its idiosyncrasies into account. The good news is that while NUMA changes the rules of the game a bit, it doesn't break software. NUMA-aware OSes do the heavy lifting here, helping unaware software by keeping threads and memory accesses together on the same NUMA node in order to ensure classic performance characteristics. The downside to this is that much like an overprotective parent, the OS is going discourage unaware software from using other NUMA nodes. Or in the case of Threadripper, discouraging applications from using the other die and its 8 cores.


At a hardware level, Threadripper is natively two NUMA nodes

In an ideal world, all software would be NUMA-aware, eliminating any concerns over the matter. From a practical perspective however, software is slow to change and it seems unlikely that NUMA-style CPUs are going to become common in the future. Furthermore NUMA can be tricky to program for, especially in the case of workloads/algorithms that inherently struggle with "far" cores and memory. So the quirks of NUMA are never going to completely go away, and instead AMD has taken it upon themselves to manage the matter.

AMD has implemented BIOS switches and software switches in order to better support and control the NUMAness of Threadripper. By default, Threadripper actually hides its NUMA architecture. AMD instead runs Threadripper in a UMA configuration: a uniform memory access system where memory is sent to any DRAM and the return is variable in latency (e.g. ~100ns averaging between 78ns and 133ns) but focusing for a high peak bandwidth. By presenting the CPU to the OS as a monolithic, single-domain design, memory bandwidth is maximized and all applications (NUMA-aware and not) see all 16 cores as part of the same CPU. So for applications that are not NUMA-aware – and consequently would have been discouraged by the OS in NUMA mode – this maximizes the number of cores/threads they can use and the memory bandwidth they can use.


All 32 threads are exposed as part of a single monolithic CPU

The drawback to UMA mode is that because it's hiding how Threadripper really works, it doesn't allow the OS and applications to make fully informed decisions for themselves, and consequently they may not make the best decisions. Latency-sensitive NUMA-unaware applications that fare poorly with high core/memory latencies can struggle here if they use cores and memory attached to the other die. Which is why AMD also allows Threadripper to be configured for NUMA mode, exposing its full design to the OS and resulting in separate NUMA domains for the two dies. This informs the OS to keep applications pinned to one die when possible as previously discussed, and this mode is vital for some software and some games, and we’ve tested it in this review.

Overall, using a multi-silicon design has positives and negatives. The negatives end up being variable memory latency, variable core-to-core latency, and often redundancy in on-die units that don’t need to be repeated. As a result, AMD uses 400mm2+ of silicon to achieve this, which can increase costs at the manufacturing level. By contrast, the positives are in silicon design and overall yeilds: being able to design a single piece of silicon and repeat it, rather than design several different floor plans which multiplies up the design costs, and having the (largely) fixed number of wafer defects spread out over many more smaller dies.

By contrast, Intel uses a single monolithic die for its Skylake-X processors: the LCC die up to 10-core and HCC die from 12-core up to 18-core. These use a rectangular grid of cores (3x4 and 5x4 respectively), with two of the segments reserved for the memory controllers. In order to communicate between the cores, Intel uses a networking mesh, which determines which direction the data needs to travel (up, down, left, right, or accepted into the core). We covered Intel’s MOdular Decoupled Crossbar (MoDe-X) methodology in our Skylake-X review, but the underlying concept is consistency. This mesh runs at 2.4 GHz nominally. Prior to Skylake-X, Intel implemented a ring topology, such that data would have to travel around the ring of cores to get to where it needed to go.

With reference to glue, or glue-logic, we’re referring to the fabric of each processor. For AMD that’s the Infinity Fabric, which has to travel within the silicon die or out to the other silicon die, and for Intel that’s the internal MoDe-X mesh. Elmer’s never looked so complicated.

Feeding the Beast and CPU Top Trumps AMD’s Solution to Dual Dies: Creator Mode and Game Mode
POST A COMMENT

349 Comments

View All Comments

  • verl - Thursday, August 10, 2017 - link

    "well above the Ryzen CPUs, and batching the 10C/8C parts from Broadwell-E and Haswell-E respectively"

    ??? From the Power Consumption page.
    Reply
  • bongey - Thursday, August 10, 2017 - link

    Yep if you use AVX-512 it will down clock to 1.8Ghz and draw 400w just for the CPU alone and 600w from the wall. See der8auer's video title "The X299 VRM Disaster (en)", all x299 motherboards VRMs can be ran into thermal shutdown under avx 512 loads, with just a small overclock, not to mention avx512 crazy power consumption. That is why AMD didn't put avx 512 in Zen, it is power consumption monster. Reply
  • TidalWaveOne - Thursday, August 10, 2017 - link

    Glad I went with the 7820X for software development (compiling). Reply
  • raddude9 - Thursday, August 10, 2017 - link

    In ars' review they have TR-1950X ahead of the i9-7900X for compilation:
    https://arstechnica.co.uk/gadgets/2017/08/amd-thre...

    In short it's very difficult to test compilation, every project you build has different properties.
    Reply
  • emn13 - Thursday, August 10, 2017 - link

    Yeah, the discrepency is huge - converted to anandtech's compile's per day the arstechnica benchmark maxes out at a little less than 20, which is a far cry from the we see here.

    Clearly, the details of the compiler, settings and codebase (and perhaps other things!) matter hugely.

    That's unfortunate, because compilation is annoyingly slow, and it would be a boon to know what to buy to ameliorate that.
    Reply
  • prisonerX - Thursday, August 10, 2017 - link

    This is very compiler dependent. My compiler is blazingly fast on my wimpy hardware becuase it's blazingly clever. Most compilers seem to crawl no matter what they run on. Reply
  • bongey - Thursday, August 10, 2017 - link

    Looks like anandtech's benchmark for compiling is bunk, it's just way off from all the other benchmarks out there. Not only that, no other test shows a 20% improvement over the 6950x which is also a 10 core/20 thread cpu. Something tells me the 7900x is completely wrong or has something faster like a different pcie ssd. Reply
  • Chad - Thursday, August 10, 2017 - link

    All I know is, for those of us running Plex, SABnzbd, Sonarr, Radarr servers simultaneously (and others), while encoding and gaming all simultaneously, our day has arrived!

    :)
    Reply
  • Ian Cutress - Thursday, August 10, 2017 - link

    We checked with Ars as to their method.

    We use a fixed late March build around v56 under MSVC
    Ars use a fixed newer build around v62 via clang-cl using VC++ linking

    Same software, different compilers, different methods. Our results are faster than Ars, although Ars' results seem to scale better.
    Reply
  • ddriver - Friday, August 11, 2017 - link

    Of every review out there, only your "superior testing methodology" presents a picture where TR is slower than SX. Reply

Log in

Don't have an account? Sign up now