Memory Subsystem: Bandwidth

As we have reported before, measuring the full bandwidth potential with John McCalpin's Stream bandwidth benchmark has become a matter of extreme tuning, requiring a very deep understanding of the platform. 

If we used our previous binaries, both the first and second generation EPYC could not get past 200-210 GB/s. It gave the impression of running into a "bandwidth wall", despite the fact that we now had 8-channel DDR4-3200. So we used the results that Intel and AMD best binaries produce using AVX-512 (Intel) and AVX-2 (AMD). 

The results are expressed in gigabytes per second.

Stream Triad

AMD can reach even higher numbers with the setting "number of nodes per socket" (NPS) set to 4. With 4 nodes per socket, AMD reports up to 353 GB/s. NPS4 will cause the CCX to only access the memory controllers with the lowest latency at the central IO Hub chip.

Those numbers only matter to a small niche of carefully AVX(-256/512) optimized HPC applications. AMD claims a 45% advantage compared to the best (28-core) Intel SKUs. We have every reason to believe them but it is only relevant to a niche. 

For the rest of the enterprise world (probably 95+%), memory latency has much larger impact than peak bandwidth. 

Benchmark Configuration and Methodology Memory Subsystem: Latency
Comments Locked

180 Comments

View All Comments

  • MDD1963 - Friday, August 9, 2019 - link

    of the '1800 servers', how many of those are virtual, just out of curiosity? ('1800 servers' is not quite as impressive if there were, for example, 10 hosts w/ 180 Windows VMs each, for example) U.S.A.F offices are still mostly Windows 10...I'd suspect the are datacenters at each base having a large Windows Server presence as well.. (But, we used Redhat onboard assorted recce aircraft for many years now....; which seems stupid in light of the fact they could easily use CentOS for free; presumably, a Senator's family members work at Redhat, and enjoy the large income from support contracts)
  • eek2121 - Friday, August 9, 2019 - link

    Employers use Windows, data centers use Linux. All the major cloud providers, including Microsoft, have reported that Linux has the highest market share.
  • gylgamesh - Friday, August 9, 2019 - link

    Could you please specify what kind of servers those are and what tasks do they perform, and also which MS Windows OS are they using? Thanks.
  • Slickest - Thursday, August 8, 2019 - link

    I work for one of the largest colleges in the nation, and 90% of our servers are Windows.
  • 69369369 - Thursday, August 8, 2019 - link

    "LUL"

    Go back to Twitch kiddo.
  • azfacea - Thursday, August 8, 2019 - link

    and what would u do if i dont?
  • Oliseo - Thursday, August 8, 2019 - link

    Tell your mum you're up late again and she will force you to go outside, and we all know how much your dislike having to do that.
  • prophet001 - Monday, August 12, 2019 - link

    I don't get the hate for windows server? How you gonna run a domain and active directory in linux?
  • CaedenV - Wednesday, August 21, 2019 - link

    I use to work in schools, and a few that could not afford Windows Server would run AD through some Linux application. It was not exactly full-featured, but it worked well enough for 'free' solution.
  • deltaFx2 - Wednesday, August 7, 2019 - link

    @Pancakes: Are you kidding? 1T perf is at par with Skylake. Windows licenses per core. Why would anyone buy a SKU with more cores than they need? And if they did do that, why would they not run them on a VM? Do these people also buy more racks than they need and run windows just for the fun of it?

Log in

Don't have an account? Sign up now