CPU Tests: Office and Science

Our previous set of ‘office’ benchmarks have often been a mix of science and synthetics, so this time we wanted to keep our office section purely on real world performance.

Agisoft Photoscan 1.3.3: link

The concept of Photoscan is about translating many 2D images into a 3D model - so the more detailed the images, and the more you have, the better the final 3D model in both spatial accuracy and texturing accuracy. The algorithm has four stages, with some parts of the stages being single-threaded and others multi-threaded, along with some cache/memory dependency in there as well. For some of the more variable threaded workload, features such as Speed Shift and XFR will be able to take advantage of CPU stalls or downtime, giving sizeable speedups on newer microarchitectures.

For the update to version 1.3.3, the Agisoft software now supports command line operation. Agisoft provided us with a set of new images for this version of the test, and a python script to run it. We’ve modified the script slightly by changing some quality settings for the sake of the benchmark suite length, as well as adjusting how the final timing data is recorded. The python script dumps the results file in the format of our choosing. For our test we obtain the time for each stage of the benchmark, as well as the overall time.

(1-1) Agisoft Photoscan 1.3, Complex Test

For a variable threaded load, the i9-10900K sits above the Rocket Lake parts.

RISC-V Toolchain Compile

Our latest test in our suite is the RISCV Toolchain compile from the Github source. This set of tools enables users to build software for a RISCV platform, however the tools themselves have to be built. For our test, we're running a complete fresh build of the toolchain, including from-scratch linking. This makes the test not a straightforward test of an updated compile on its own, but does form the basis of an ab initio analysis of system performance given its range of single-thread and multi-threaded workload sections. More details can be found here.

(1-4) Compile RISCV Toolchain

One place where Intel is winning in absolute terms in our compile-from-scratch test. We re-ran the numbers on Intel with the latest microcode due to a critical issue, but we can see here that AMD's best are single chiplet designs but Intel ekes out a small lead.

 

Science

In this version of our test suite, all the science focused tests that aren’t ‘simulation’ work are now in our science section. This includes Brownian Motion, calculating digits of Pi, molecular dynamics, and for the first time, we’re trialing an artificial intelligence benchmark, both inference and training, that works under Windows using python and TensorFlow.  Where possible these benchmarks have been optimized with the latest in vector instructions, except for the AI test – we were told that while it uses Intel’s Math Kernel Libraries, they’re optimized more for Linux than for Windows, and so it gives an interesting result when unoptimized software is used.

3D Particle Movement v2.1: Non-AVX and AVX2/AVX512

This is the latest version of this benchmark designed to simulate semi-optimized scientific algorithms taken directly from my doctorate thesis. This involves randomly moving particles in a 3D space using a set of algorithms that define random movement. Version 2.1 improves over 2.0 by passing the main particle structs by reference rather than by value, and decreasing the amount of double->float->double recasts the compiler was adding in.

The initial version of v2.1 is a custom C++ binary of my own code, and flags are in place to allow for multiple loops of the code with a custom benchmark length. By default this version runs six times and outputs the average score to the console, which we capture with a redirection operator that writes to file.

For v2.1, we also have a fully optimized AVX2/AVX512 version, which uses intrinsics to get the best performance out of the software. This was done by a former Intel AVX-512 engineer who now works elsewhere. According to Jim Keller, there are only a couple dozen or so people who understand how to extract the best performance out of a CPU, and this guy is one of them. To keep things honest, AMD also has a copy of the code, but has not proposed any changes.

The 3DPM test is set to output millions of movements per second, rather than time to complete a fixed number of movements.

(2-1) 3D Particle Movement v2.1 (non-AVX)(2-2) 3D Particle Movement v2.1 (Peak AVX)

When AVX-512 comes to play, every-one else goes home. Easiest and clearest win for Intel.

y-Cruncher 0.78.9506: www.numberworld.org/y-cruncher

If you ask anyone what sort of computer holds the world record for calculating the most digits of pi, I can guarantee that a good portion of those answers might point to some colossus super computer built into a mountain by a super-villain. Fortunately nothing could be further from the truth – the computer with the record is a quad socket Ivy Bridge server with 300 TB of storage. The software that was run to get that was y-cruncher.

Built by Alex Yee over the last part of a decade and some more, y-Cruncher is the software of choice for calculating billions and trillions of digits of the most popular mathematical constants. The software has held the world record for Pi since August 2010, and has broken the record a total of 7 times since. It also holds records for e, the Golden Ratio, and others. According to Alex, the program runs around 500,000 lines of code, and he has multiple binaries each optimized for different families of processors, such as Zen, Ice Lake, Sky Lake, all the way back to Nehalem, using the latest SSE/AVX2/AVX512 instructions where they fit in, and then further optimized for how each core is built.

For our purposes, we’re calculating Pi, as it is more compute bound than memory bound. In ST and MT mode we calculate 250 million digits.

(2-3) yCruncher 0.78.9506 ST (250m Pi)(2-4) yCruncher 0.78.9506 MT (2.5b Pi)

In ST mode, we are more dominated by the AVX-512 instructions, whereas in MT it becomes a mix of memory as well.

NAMD 2.13 (ApoA1): Molecular Dynamics

One of the popular science fields is modeling the dynamics of proteins. By looking at how the energy of active sites within a large protein structure over time, scientists behind the research can calculate required activation energies for potential interactions. This becomes very important in drug discovery. Molecular dynamics also plays a large role in protein folding, and in understanding what happens when proteins misfold, and what can be done to prevent it. Two of the most popular molecular dynamics packages in use today are NAMD and GROMACS.

NAMD, or Nanoscale Molecular Dynamics, has already been used in extensive Coronavirus research on the Frontier supercomputer. Typical simulations using the package are measured in how many nanoseconds per day can be calculated with the given hardware, and the ApoA1 protein (92,224 atoms) has been the standard model for molecular dynamics simulation.

Luckily the compute can home in on a typical ‘nanoseconds-per-day’ rate after only 60 seconds of simulation, however we stretch that out to 10 minutes to take a more sustained value, as by that time most turbo limits should be surpassed. The simulation itself works with 2 femtosecond timesteps. We use version 2.13 as this was the recommended version at the time of integrating this benchmark into our suite. The latest nightly builds we’re aware have started to enable support for AVX-512, however due to consistency in our benchmark suite, we are retaining with 2.13. Other software that we test with has AVX-512 acceleration.

(2-5) NAMD ApoA1 Simulation

The Intel parts shows some improvement over the previous generations of Intel, however the 10-core Comet Lake still wins ahead of Rocket Lake.

AI Benchmark 0.1.2 using TensorFlow: Link

Finding an appropriate artificial intelligence benchmark for Windows has been a holy grail of mine for quite a while. The problem is that AI is such a fast moving, fast paced word that whatever I compute this quarter will no longer be relevant in the next, and one of the key metrics in this benchmarking suite is being able to keep data over a long period of time. We’ve had AI benchmarks on smartphones for a while, given that smartphones are a better target for AI workloads, but it also makes some sense that everything on PC is geared towards Linux as well.

Thankfully however, the good folks over at ETH Zurich in Switzerland have converted their smartphone AI benchmark into something that’s useable in Windows. It uses TensorFlow, and for our benchmark purposes we’ve locked our testing down to TensorFlow 2.10, AI Benchmark 0.1.2, while using Python 3.7.6.

The benchmark runs through 19 different networks including MobileNet-V2, ResNet-V2, VGG-19 Super-Res, NVIDIA-SPADE, PSPNet, DeepLab, Pixel-RNN, and GNMT-Translation. All the tests probe both the inference and the training at various input sizes and batch sizes, except the translation that only does inference. It measures the time taken to do a given amount of work, and spits out a value at the end.

There is one big caveat for all of this, however. Speaking with the folks over at ETH, they use Intel’s Math Kernel Libraries (MKL) for Windows, and they’re seeing some incredible drawbacks. I was told that MKL for Windows doesn’t play well with multiple threads, and as a result any Windows results are going to perform a lot worse than Linux results. On top of that, after a given number of threads (~16), MKL kind of gives up and performance drops of quite substantially.

So why test it at all? Firstly, because we need an AI benchmark, and a bad one is still better than not having one at all. Secondly, if MKL on Windows is the problem, then by publicizing the test, it might just put a boot somewhere for MKL to get fixed. To that end, we’ll stay with the benchmark as long as it remains feasible.

(2-6) AI Benchmark 0.1.2 Total

Every generation of Intel seems to regress with AI Benchmark, most likely due to MKL issues. I have previously identified the issue for Intel, however I have not heard of any progress to date.

CPU Tests: Microbenchmarks CPU Tests: Simulation and Rendering
POST A COMMENT

275 Comments

View All Comments

  • Hulk - Tuesday, March 30, 2021 - link

    I have also been looking for iGPU tests?

    It's strange. It's like it doesn't exist.
    Reply
  • eastcoast_pete - Tuesday, March 30, 2021 - link

    Predictable results. I don't believe Intel back-ported to 14 nm because their 10 nm couldn't reach high frequencies, they back-ported because their yields at 10 nm aren't high enough and they had manufacturing capacity available at 14 nm. That made the expense of back-porting the design worthwhile.
    Regarding Rocket Lake, the most interesting CPUs in this lineup are the non-K i5, especially the ones that still have the 32 EUs enabled. Any chance you (Ian) can put one or two 11500 or 11600 through their paces. I would really like to also see how "35 W" the T models are. One of those, plus a decent, WiFi-enabled 560 MoBo for $100-130 could serve HTPC and office duties.
    Reply
  • Otritus - Tuesday, March 30, 2021 - link

    Rocket lake is an 8 core cpu based on the cypress (sunny) cove microarchitecture. Tigerlake H is an 8 core cpu based on the willow (sunny+) cove microarchitecture. Both have 32 Xe EUs. 10SF and 10ESF yields well (Intel is shipping much larger server processors just fine). The problem is 10SF seems to max out around 5Ghz which is the upper bounds of the 11700K. The slight clock bump of the 11900K lets Intel claim the fastest gaming cpu, which would not have been possible on Tigerlake H. 14nm having excess capacity was simply the cherry on top. Reply
  • goatfajitas - Tuesday, March 30, 2021 - link

    I dont th9ink anything Intel has done here can be called "cherry on top" if anything we will look on this as a hot mess (pun intended). :P Reply
  • AntonErtl - Wednesday, March 31, 2021 - link

    Intel shipping larger chips in their current 10nm processes does not disprove yield problems. If there is a flaw in a core on a 40-core die, jut disable the core (and another one) and sell it as 38-core CPU. If there are flaws in three cores, sell it as 36-core CPU, etc. Of course that's also possible and done for Tiger Lake, but there is also parts of the CPU where you have no such redundancy, but the area for these parts is not necessarily larger for the bigger dies, and the huge price for the big dies may make it economically more viable there than on the desktop.

    What makes me believe that either 10nm yield or 10nm capacity is not so great (or capcity is not reat because yield is not great) is that the announced Xeon W-13xx CPUs are going to be Rocket, not Tiger Lake; at 80W TDP, I expect that Tiger Lake would outperform Rocket Lake for most multi-threaded and (thangs to larger cache) some single-threaded workloads, yet they give us Rocket Lake,
    Reply
  • Spunjji - Friday, April 9, 2021 - link

    Intel arriving nearly 3 years late with Ice Lake SP and only managing "over 200,000" in the first 3 months isn't "just fine", it's pretty indicative that they're still struggling.

    We have no indication of ESF yields yet as there are no ESF products shipping yet.

    Rocket Lake was ported when Intel couldn't get clock speeds *or* yields out of their 10nm process. If their yields and capacity for 10SF were as good as you're implying, we wouldn't still be waiting for Tiger Lake H to actually hit the market so long after Tiger Lake launched.
    Reply
  • YazX_ - Tuesday, March 30, 2021 - link

    4 years ago, AMD was broke, fighting for survival, targeting the poor and intel was the top dog and spit the same s.hit for the 10 years, fast forward now, AMD is the top dog and intel cannot even catch up.

    Regarding the review, yah as usual s.hit intel CPU that draws alot of power and still priced higher than AMD while offering less.
    Reply
  • haukionkannel - Tuesday, March 30, 2021 - link

    And Intel will be selling these much, much more than amd can their own...
    Intel just need to exist, to win amd in market share...
    Reply
  • Qasar - Tuesday, March 30, 2021 - link

    haukionkannel yep, only because its on the shelf, if/when ryzen 5000 supply gets better expect that to change. once that happens intel wont be selling that well. no one i know is even looking at intel right now, all waiting for zen 3. Reply
  • SkyBill40 - Tuesday, March 30, 2021 - link

    You mean Zen 4? Reply

Log in

Don't have an account? Sign up now