Compiling Performance / LLVM

As we’re trying to rebuild our server test suite piece by piece – and there’s still a lot of work go ahead to get a good representative “real world” set of workloads, one more highly desired benchmark amongst readers was a more realistic compilation suite. Chrome and LLVM codebases being the most requested, I landed on LLVM as it’s fairly easy to set up and straightforward.

git clone
cd llvm-project
git checkout release/11.x
mkdir ./build
cd ..
mkdir llvm-project-tmpfs
sudo mount -t tmpfs -o size=10G,mode=1777 tmpfs ./llvm-project-tmpfs
cp -r llvm-project/* llvm-project-tmpfs
cd ./llvm-project-tmpfs/build
cmake -G Ninja \
  -DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi;lldb;compiler-rt;lld" \
  -DCMAKE_BUILD_TYPE=Release ../llvm
time cmake --build .

We’re using the LLVM 11.0.0 release as the build target version, and we’re compiling Clang, libc++abi, LLDB, Compiler-RT and LLD using GCC 10.2 (self-compiled). To avoid any concerns about I/O we’re building things on a ramdisk – on a 4KB page system 5GB should be sufficient but on the Altra’s 64KB system it used up to 9.5GB, including the source directory. We’re measuring the actual build time and don’t include the configuration phase as usually in the real world that doesn’t happen repeatedly.

LLVM Suite Compile Time

The LLVM compile test results here are quite more special and demand more attention that what meets the eye at first.

Inherently, the biggest work slice of the test is massively parallel, able to take advantage of all cores in a system, 256 cores in the 2-socket results of the M128-30, however as it’s also a real-world test, the compilation also incurs linking phases where the chip is inherently just under a single-core load and all other cores are just sitting idle.

This behaviour results in some more complex behaviour in the different test scenarios of the M128-30, as the ratio between the parallel/MT and ST phases of the test changes.

In the single-socket results, the chip showcases a +14% performance boost over the Q80-33, while in the 2S results under quadrant mode, this actually transforms into a 16% performance regression. What’s happening here is that while the increased core count of the chip massively helps in improving the actual compilation of objects, the linking phase of the test is significantly slower and takes up a larger percentage of total test time than on the Q80-33, due to the lower CPU frequencies and smaller SLC of the new chip.

Running the M128-30 in monolithic mode actually results in a 24% reduction in compile time, mostly through a large speedup of the linking phase of the compilation as we’re giving that one active core access to the whole 16MB SLC rather than just a 4MB slice.

AMD’s EPYC 7763, even though it has only half the core count, still manages to outperform the M128-30 in the total test time because the linking phase is much sped up thanks to the much superior single-threaded performance of the cores when few threads are active on the SoC. The 34% advantage of the ST SPEC scores here comes more into play than the MT throughput of the chip.

These results are very interesting, and showcase that even in a more real-world scenario like this, the flock-of-chickens approach doesn’t work out as well even in what would consider a massively parallel workload, as some things just cannot be spread out over multiple cores well. It reminds me very much of the eMAG chip, which also suffered in real world code compilations due to the very same reasons.

SPECjbb MultiJVM - Java Performance Conclusion & End Remarks


View All Comments

  • mode_13h - Thursday, October 7, 2021 - link

    > x86 still commands 99% of the server market.

    Depends on what you consider the "server market", but AWS is very rapidly switching over. Others will follow.

    Lots of cloud compute just depends on density and power-efficiency. And here's where ARM has a real advantage.
  • Wilco1 - Thursday, October 7, 2021 - link

    According to Arm server revenue has been 4-5% over the last few quarters. Reply
  • schujj07 - Friday, October 8, 2021 - link

    Anything under 10% market share in the server world is basically considered a niche player. Right now AMD is over 10% so they are finally seen as an actual player in the market. Reply
  • Spunjji - Friday, October 8, 2021 - link

    Pointing at current market share that resulted from a lack of viable ARM competition isn't a great argument for your prediction that ARM will not gain market share, especially when you're being presented with evidence of viable ARM competition. Reply
  • mode_13h - Thursday, October 7, 2021 - link

    > Before AMD can disrupt Intel in the server,

    *before* ? This is already happening! You can clearly see it in AMD's server marketshare, as well as the price structure of Ice Lake.

    > And now Intel is coming back with Saphire Rapids. Doesn't look good for AMD.

    AMD has Genoa, V-Cache, and who knows what else in the pipeline. Oh, and they can also build an ARM core just as good as anyone (with the possible exceptions of Apple and Nuvia/Qualcomm).
  • yetanotherhuman - Friday, October 8, 2021 - link

    Not even in slight agreement. Different architecture. Reply
  • eastcoast_pete - Thursday, October 7, 2021 - link

    Thanks Andrei, great analysis! IMO, the biggest problem Ampere and other firms that develop server CPUs based on ARM designs is that their natural customers - large, cloud-type providers - pretty much all have their own, in-house designed ARM-based CPUs, and won't buy thousands of third party CPUs unless they do something their own can't do, or nowhere near as well. AWS, Google, MS, and Apple still buy x86 CPUs from Intel or AMD because there is a customer demand for those instances, but also try to shift as much as they can to their own, home-grown ARM server systems. In this regard, has anyone heard any updates about the ARM designs supposedly in development at MS? Maybe Ampere can get themselves bought out by them? Reply
  • name99 - Friday, October 8, 2021 - link

    “own house-designed ARM-based CPU’s”?
    We obviously have Graviton. Apple seem a reasonable bet at some point. Maybe a large Chinese player.

    Do we have any evidence (as opposed to hypotheses and rumors) of Google, Facebook, Microsoft, or most of China? Or other smaller but still large players like Yandex or Cloudflare?
  • Sivar - Thursday, October 7, 2021 - link

    This is a proper old-school deep CPU review. Reply
  • vegemeister - Thursday, October 7, 2021 - link

    Text says Intel Xeon 8380 is running at 205 W power limit, but the table says 270 W. Which is it? I assume 270 W like ARK says? Reply

Log in

Don't have an account? Sign up now