CPU Tests: Office and Science

Our previous set of ‘office’ benchmarks have often been a mix of science and synthetics, so this time we wanted to keep our office section purely on real world performance.

Agisoft Photoscan 1.3.3: link

The concept of Photoscan is about translating many 2D images into a 3D model - so the more detailed the images, and the more you have, the better the final 3D model in both spatial accuracy and texturing accuracy. The algorithm has four stages, with some parts of the stages being single-threaded and others multi-threaded, along with some cache/memory dependency in there as well. For some of the more variable threaded workload, features such as Speed Shift and XFR will be able to take advantage of CPU stalls or downtime, giving sizeable speedups on newer microarchitectures.

For the update to version 1.3.3, the Agisoft software now supports command line operation. Agisoft provided us with a set of new images for this version of the test, and a python script to run it. We’ve modified the script slightly by changing some quality settings for the sake of the benchmark suite length, as well as adjusting how the final timing data is recorded. The python script dumps the results file in the format of our choosing. For our test we obtain the time for each stage of the benchmark, as well as the overall time.

(1-1) Agisoft Photoscan 1.3, Complex Test

There is a small performance gain here in the real world test across the three generations of Intel processors, however it is still a step away from AMD.

Application Opening: GIMP 2.10.18

First up is a test using a monstrous multi-layered xcf file to load GIMP. While the file is only a single ‘image’, it has so many high-quality layers embedded it was taking north of 15 seconds to open and to gain control on the mid-range notebook I was using at the time.

What we test here is the first run - normally on the first time a user loads the GIMP package from a fresh install, the system has to configure a few dozen files that remain optimized on subsequent opening. For our test we delete those configured optimized files in order to force a ‘fresh load’ each time the software in run. As it turns out, GIMP does optimizations for every CPU thread in the system, which requires that higher thread-count processors take a lot longer to run.

We measure the time taken from calling the software to be opened, and until the software hands itself back over to the OS for user control. The test is repeated for a minimum of ten minutes or at least 15 loops, whichever comes first, with the first three results discarded.

(1-2) AppTimer: GIMP 2.10.18

The app initialization test here favors single core performance, and AMD wins despite the lower single thread frequency. The 9900KS has a slight advantage, being a guaranteed 5.0 GHz, but none of the improved IPC from the Cypress Cove seems to come into play here.

 

Science

In this version of our test suite, all the science focused tests that aren’t ‘simulation’ work are now in our science section. This includes Brownian Motion, calculating digits of Pi, molecular dynamics, and for the first time, we’re trialing an artificial intelligence benchmark, both inference and training, that works under Windows using python and TensorFlow.  Where possible these benchmarks have been optimized with the latest in vector instructions, except for the AI test – we were told that while it uses Intel’s Math Kernel Libraries, they’re optimized more for Linux than for Windows, and so it gives an interesting result when unoptimized software is used.

3D Particle Movement v2.1: Non-AVX and AVX2/AVX512

This is the latest version of this benchmark designed to simulate semi-optimized scientific algorithms taken directly from my doctorate thesis. This involves randomly moving particles in a 3D space using a set of algorithms that define random movement. Version 2.1 improves over 2.0 by passing the main particle structs by reference rather than by value, and decreasing the amount of double->float->double recasts the compiler was adding in.

The initial version of v2.1 is a custom C++ binary of my own code, and flags are in place to allow for multiple loops of the code with a custom benchmark length. By default this version runs six times and outputs the average score to the console, which we capture with a redirection operator that writes to file.

For v2.1, we also have a fully optimized AVX2/AVX512 version, which uses intrinsics to get the best performance out of the software. This was done by a former Intel AVX-512 engineer who now works elsewhere. According to Jim Keller, there are only a couple dozen or so people who understand how to extract the best performance out of a CPU, and this guy is one of them. To keep things honest, AMD also has a copy of the code, but has not proposed any changes.

The 3DPM test is set to output millions of movements per second, rather than time to complete a fixed number of movements.

(2-1) 3D Particle Movement v2.1 (non-AVX)(2-2) 3D Particle Movement v2.1 (Peak AVX)

When AVX-512 comes to play, every-one else goes home. Easiest and clearest win for Intel.

y-Cruncher 0.78.9506: www.numberworld.org/y-cruncher

If you ask anyone what sort of computer holds the world record for calculating the most digits of pi, I can guarantee that a good portion of those answers might point to some colossus super computer built into a mountain by a super-villain. Fortunately nothing could be further from the truth – the computer with the record is a quad socket Ivy Bridge server with 300 TB of storage. The software that was run to get that was y-cruncher.

Built by Alex Yee over the last part of a decade and some more, y-Cruncher is the software of choice for calculating billions and trillions of digits of the most popular mathematical constants. The software has held the world record for Pi since August 2010, and has broken the record a total of 7 times since. It also holds records for e, the Golden Ratio, and others. According to Alex, the program runs around 500,000 lines of code, and he has multiple binaries each optimized for different families of processors, such as Zen, Ice Lake, Sky Lake, all the way back to Nehalem, using the latest SSE/AVX2/AVX512 instructions where they fit in, and then further optimized for how each core is built.

For our purposes, we’re calculating Pi, as it is more compute bound than memory bound. In ST and MT mode we calculate 250 million digits.

(2-3) yCruncher 0.78.9506 ST (250m Pi)(2-4b) yCruncher 0.78.9506 MT (250m Pi)

In ST mode, we are more dominated by the AVX-512 instructions, whereas in MT it becomes a mix of memory as well.

NAMD 2.13 (ApoA1): Molecular Dynamics

One of the popular science fields is modeling the dynamics of proteins. By looking at how the energy of active sites within a large protein structure over time, scientists behind the research can calculate required activation energies for potential interactions. This becomes very important in drug discovery. Molecular dynamics also plays a large role in protein folding, and in understanding what happens when proteins misfold, and what can be done to prevent it. Two of the most popular molecular dynamics packages in use today are NAMD and GROMACS.

NAMD, or Nanoscale Molecular Dynamics, has already been used in extensive Coronavirus research on the Frontier supercomputer. Typical simulations using the package are measured in how many nanoseconds per day can be calculated with the given hardware, and the ApoA1 protein (92,224 atoms) has been the standard model for molecular dynamics simulation.

Luckily the compute can home in on a typical ‘nanoseconds-per-day’ rate after only 60 seconds of simulation, however we stretch that out to 10 minutes to take a more sustained value, as by that time most turbo limits should be surpassed. The simulation itself works with 2 femtosecond timesteps. We use version 2.13 as this was the recommended version at the time of integrating this benchmark into our suite. The latest nightly builds we’re aware have started to enable support for AVX-512, however due to consistency in our benchmark suite, we are retaining with 2.13. Other software that we test with has AVX-512 acceleration.

(2-5) NAMD ApoA1 Simulation

The 11700K shows some improvement over the previous generations of Intel, however it does sit much in the middle of the APU and the Zen 3.

AI Benchmark 0.1.2 using TensorFlow: Link

Finding an appropriate artificial intelligence benchmark for Windows has been a holy grail of mine for quite a while. The problem is that AI is such a fast moving, fast paced word that whatever I compute this quarter will no longer be relevant in the next, and one of the key metrics in this benchmarking suite is being able to keep data over a long period of time. We’ve had AI benchmarks on smartphones for a while, given that smartphones are a better target for AI workloads, but it also makes some sense that everything on PC is geared towards Linux as well.

Thankfully however, the good folks over at ETH Zurich in Switzerland have converted their smartphone AI benchmark into something that’s useable in Windows. It uses TensorFlow, and for our benchmark purposes we’ve locked our testing down to TensorFlow 2.10, AI Benchmark 0.1.2, while using Python 3.7.6.

The benchmark runs through 19 different networks including MobileNet-V2, ResNet-V2, VGG-19 Super-Res, NVIDIA-SPADE, PSPNet, DeepLab, Pixel-RNN, and GNMT-Translation. All the tests probe both the inference and the training at various input sizes and batch sizes, except the translation that only does inference. It measures the time taken to do a given amount of work, and spits out a value at the end.

There is one big caveat for all of this, however. Speaking with the folks over at ETH, they use Intel’s Math Kernel Libraries (MKL) for Windows, and they’re seeing some incredible drawbacks. I was told that MKL for Windows doesn’t play well with multiple threads, and as a result any Windows results are going to perform a lot worse than Linux results. On top of that, after a given number of threads (~16), MKL kind of gives up and performance drops of quite substantially.

So why test it at all? Firstly, because we need an AI benchmark, and a bad one is still better than not having one at all. Secondly, if MKL on Windows is the problem, then by publicizing the test, it might just put a boot somewhere for MKL to get fixed. To that end, we’ll stay with the benchmark as long as it remains feasible.

(2-6) AI Benchmark 0.1.2 Total

Every generation of Intel seems to regress with AI Benchmark, most likely due to MKL issues. I have previously identified the issue for Intel, however I have not heard of any progress to date.

CPU Tests: Microbenchmarks CPU Tests: Simulation and Rendering
Comments Locked

541 Comments

View All Comments

  • ottonis - Saturday, March 6, 2021 - link

    How is Ryzen 5800x AMD's "best of the best"?
    It clearly is not but just about somewhere in the middle. If you want AMD's best consumer CPU, you gonna look at the 5950x.
    From the perspective of AMD, completely outsourcing manufacturing was the only way to reap the benefits of latest and greatest process nodes.
    Now, that everybody else, incl. the automobile industry, the consoles, Apple, and even Intel are booking production capacities at TMSC, has certainly contributed to Reaching capacity limits and thus to AMD CPU shortages.
    It is predicted that shortages will be mostly (hopefully) sorted out by this summer.
    But yes, now that AMD are earning quite a lot of money they should buy some TSMC stock and try to partner up, getting more production capacity in the future.
  • Fulljack - Saturday, March 6, 2021 - link

    did you just forgetting why AMD spin-off it's fab on the first place?

    we don't know exactly how much AMD put out their desktop chips from TSMC plant, and as far as we know, the market are still growing in size. while AMD market share are still a long way from reaching 50%, they still have their processor sold out all over the place.

    honestly I don't get what you're trying to say here.
  • shady28 - Saturday, March 6, 2021 - link

    Actually we do know, they made ~1M Zen 3 chips in Q4. 140M PCs were shipped. Based on their market share, about 3% of the chips AMD shipped in Q4 were Zen 3.

    Source:https://wccftech.com/amd-shipped-nearly-1-million-...
  • inighthawki - Monday, March 8, 2021 - link

    5800X = "Best of the best"
    11700K = "medium level SKU"

    Your bias is showing.
  • Bluetooth - Saturday, March 6, 2021 - link

    Does any one know the details of Intels 10 nm node problems. Any article discussing that in technical details?
  • dihartnell - Thursday, March 11, 2021 - link

    I think adored tv did a few articles, videos on this. As I understand it thier main issues are they use a monolithic die (everything on on a single die) that gets harder to make as the process shrinks, IE more dies have defects... AMD got around this by going chiplet, lots of small dies which meant more of them are good. Until Intel chnages to chiplet or they find a way to improve the manufacturing process to lower the defect rate then they will struggle.
  • Santoval - Saturday, March 6, 2021 - link

    Class action for what, excessive power bills? :) Imagine using this during a heatwave in an airless room. Since it doubles as a heater you would need to have the AC on all the time. If you have no AC there will be a competition between who dies first, you or the processor? ^.^
  • sabot00 - Friday, March 5, 2021 - link

    Indeed! I have been reading nothing but wccftech and Tom's leaks. Absolutely amazing surprise this Friday night while searching Rocket Lake
  • Beaver M. - Saturday, March 6, 2021 - link

    I would be very cautious testing or believing results with the Z590 platforms long before Rocket Lakes official release.
    Ive tested 3 of those boards with my Comet Lake (Asus, MSI and ASRock) and they all had pretty bad BIOS versions still, with PCIe/IO performance being low in SSD 4K benchmarks and getting weird frame time stutters from time to time (only noticeable when actually playing or looking at a realtime graph). Not to mention Intels drivers are still bad as well.
    On none of them even very basic features like the sleep state worked!
    Comparing this review with user benchmarks in German forums shows huge differences, so theres not much to add to this.

    That said, I have to laugh when Americans or people from other countries with cheap power complain about the power draw.
    And seeing fanboys downplay the performance of AVX and ignoring that it was always power hungry, even 6 years ago, is another obvious thing. Without it RKL actually runs pretty cool for being ancient 14nm.

    And of course I love the geniuses who still think that you cant use the iGPU without it being connected to a monitor, or dont know about the new power saving/GPU switching feature. Not that this article didnt fail at pretty much everything, incl. explaining things like that.
  • Gigaplex - Saturday, March 6, 2021 - link

    "That said, I have to laugh when Americans or people from other countries with cheap power complain about the power draw."
    Electricity is expensive where I live, but that's not why I want low power consumption. The more power it consumes, the louder the cooling solution will be. That's why my last system was Intel (Ivy Bridge) and my current system is AMD (Zen 3).

Log in

Don't have an account? Sign up now