CPU Tests: Office and Science

Our previous set of ‘office’ benchmarks have often been a mix of science and synthetics, so this time we wanted to keep our office section purely on real-world performance.

Agisoft Photoscan 1.3.3: link

The concept of Photoscan is about translating many 2D images into a 3D model - so the more detailed the images, and the more you have, the better the final 3D model in both spatial accuracy and texturing accuracy. The algorithm has four stages, with some parts of the stages being single-threaded and others multi-threaded, along with some cache/memory dependency in there as well. For some of the more variable threaded workload, features such as Speed Shift and XFR will be able to take advantage of CPU stalls or downtime, giving sizeable speedups on newer microarchitectures.

For the update to version 1.3.3, the Agisoft software now supports command line operation. Agisoft provided us with a set of new images for this version of the test, and a python script to run it. We’ve modified the script slightly by changing some quality settings for the sake of the benchmark suite length, as well as adjusting how the final timing data is recorded. The python script dumps the results file in the format of our choosing. For our test we obtain the time for each stage of the benchmark, as well as the overall time.

(1-1) Agisoft Photoscan 1.3, Complex Test

Photoscan shows good gen-on-gen performance uplifts, with the 5700G on par with the 11700K despite being lower powered. Both the R7 and R5 APUs are within touching distance of their X counterparts, and we see a good performance jump from the 4000G series.

Application Opening: GIMP 2.10.18

First up is a test using a monstrous multi-layered xcf file to load GIMP. While the file is only a single ‘image’, it has so many high-quality layers embedded it was taking north of 15 seconds to open and to gain control on the mid-range notebook I was using at the time.

What we test here is the first run - normally on the first time a user loads the GIMP package from a fresh install, the system has to configure a few dozen files that remain optimized on subsequent opening. For our test we delete those configured optimized files in order to force a ‘fresh load’ each time the software in run. As it turns out, GIMP does optimizations for every CPU thread in the system, which requires that higher thread-count processors take a lot longer to run.

We measure the time taken from calling the software to be opened, and until the software hands itself back over to the OS for user control. The test is repeated for a minimum of ten minutes or at least 15 loops, whichever comes first, with the first three results discarded.

(1-2) AppTimer: GIMP 2.10.18

GIMP is a funny test where it gets harder the more cores you have - that's why the quad cores win here. However slow quad cores (like the 2400G still let you down. There seems to be minor gains here for the R5000 series with Zen 3 under the hood.

 

Science

In this version of our test suite, all the science focused tests that aren’t ‘simulation’ work are now in our science section. This includes Brownian Motion, calculating digits of Pi, molecular dynamics, and for the first time, we’re trialing an artificial intelligence benchmark, both inference and training, that works under Windows using python and TensorFlow.  Where possible these benchmarks have been optimized with the latest in vector instructions, except for the AI test – we were told that while it uses Intel’s Math Kernel Libraries, they’re optimized more for Linux than for Windows, and so it gives an interesting result when unoptimized software is used.

3D Particle Movement v2.1: Non-AVX and AVX2/AVX512

This is the latest version of this benchmark designed to simulate semi-optimized scientific algorithms taken directly from my doctorate thesis. This involves randomly moving particles in a 3D space using a set of algorithms that define random movement. Version 2.1 improves over 2.0 by passing the main particle structs by reference rather than by value, and decreasing the amount of double->float->double recasts the compiler was adding in.

The initial version of v2.1 is a custom C++ binary of my own code, and flags are in place to allow for multiple loops of the code with a custom benchmark length. By default this version runs six times and outputs the average score to the console, which we capture with a redirection operator that writes to file.

For v2.1, we also have a fully optimized AVX2/AVX512 version, which uses intrinsics to get the best performance out of the software. This was done by a former Intel AVX-512 engineer who now works elsewhere. According to Jim Keller, there are only a couple dozen or so people who understand how to extract the best performance out of a CPU, and this guy is one of them. To keep things honest, AMD also has a copy of the code, but has not proposed any changes.

The 3DPM test is set to output millions of movements per second, rather than time to complete a fixed number of movements.

(2-1) 3D Particle Movement v2.1 (non-AVX)(2-2) 3D Particle Movement v2.1 (Peak AVX)

Ignoring the Peak AVX results which are heavily weighted in favor of AVX-512 enabled CPUs, with the non-AVX code we're seeing about a 5% performance gain on R5000G over R4000G.

y-Cruncher 0.78.9506: www.numberworld.org/y-cruncher

If you ask anyone what sort of computer holds the world record for calculating the most digits of pi, I can guarantee that a good portion of those answers might point to some colossus super computer built into a mountain by a super-villain. Fortunately nothing could be further from the truth – the computer with the record is a quad socket Ivy Bridge server with 300 TB of storage. The software that was run to get that was y-cruncher.

Built by Alex Yee over the last part of a decade and some more, y-Cruncher is the software of choice for calculating billions and trillions of digits of the most popular mathematical constants. The software has held the world record for Pi since August 2010, and has broken the record a total of 7 times since. It also holds records for e, the Golden Ratio, and others. According to Alex, the program runs around 500,000 lines of code, and he has multiple binaries each optimized for different families of processors, such as Zen, Ice Lake, Sky Lake, all the way back to Nehalem, using the latest SSE/AVX2/AVX512 instructions where they fit in, and then further optimized for how each core is built.

For our purposes, we’re calculating Pi, as it is more compute bound than memory bound. In single thread mode we calculate 250 million digits, while in multithreaded mode we go for 2.5 billion digits. That 2.5 billion digit value requires ~12 GB of DRAM, and so is limited to systems with at least 16 GB.

(2-3) yCruncher 0.78.9506 ST (250m Pi)(2-4b) yCruncher 0.78.9506 MT (250m Pi)(2-4) yCruncher 0.78.9506 MT (2.5b Pi)

The R3 5300G is crushing the R4000 series here, which is likely down to the unified L3 cache strcuture. 

NAMD 2.13 (ApoA1): Molecular Dynamics

One of the popular science fields is modeling the dynamics of proteins. By looking at how the energy of active sites within a large protein structure over time, scientists behind the research can calculate required activation energies for potential interactions. This becomes very important in drug discovery. Molecular dynamics also plays a large role in protein folding, and in understanding what happens when proteins misfold, and what can be done to prevent it. Two of the most popular molecular dynamics packages in use today are NAMD and GROMACS.

NAMD, or Nanoscale Molecular Dynamics, has already been used in extensive Coronavirus research on the Frontier supercomputer. Typical simulations using the package are measured in how many nanoseconds per day can be calculated with the given hardware, and the ApoA1 protein (92,224 atoms) has been the standard model for molecular dynamics simulation.

Luckily the compute can home in on a typical ‘nanoseconds-per-day’ rate after only 60 seconds of simulation, however we stretch that out to 10 minutes to take a more sustained value, as by that time most turbo limits should be surpassed. The simulation itself works with 2 femtosecond timesteps. We use version 2.13 as this was the recommended version at the time of integrating this benchmark into our suite. The latest nightly builds we’re aware have started to enable support for AVX-512, however due to consistency in our benchmark suite, we are retaining with 2.13. Other software that we test with has AVX-512 acceleration.

(2-5) NAMD ApoA1 Simulation

The slower 5000G processors are +10% faster generationally, while the R7 is about 4% faster. They all sit behind the desktop counterparts though.

AI Benchmark 0.1.2 using TensorFlow: Link

Finding an appropriate artificial intelligence benchmark for Windows has been a holy grail of mine for quite a while. The problem is that AI is such a fast moving, fast paced word that whatever I compute this quarter will no longer be relevant in the next, and one of the key metrics in this benchmarking suite is being able to keep data over a long period of time. We’ve had AI benchmarks on smartphones for a while, given that smartphones are a better target for AI workloads, but it also makes some sense that everything on PC is geared towards Linux as well.

Thankfully however, the good folks over at ETH Zurich in Switzerland have converted their smartphone AI benchmark into something that’s useable in Windows. It uses TensorFlow, and for our benchmark purposes we’ve locked our testing down to TensorFlow 2.10, AI Benchmark 0.1.2, while using Python 3.7.6.

The benchmark runs through 19 different networks including MobileNet-V2, ResNet-V2, VGG-19 Super-Res, NVIDIA-SPADE, PSPNet, DeepLab, Pixel-RNN, and GNMT-Translation. All the tests probe both the inference and the training at various input sizes and batch sizes, except the translation that only does inference. It measures the time taken to do a given amount of work, and spits out a value at the end.

There is one big caveat for all of this, however. Speaking with the folks over at ETH, they use Intel’s Math Kernel Libraries (MKL) for Windows, and they’re seeing some incredible drawbacks. I was told that MKL for Windows doesn’t play well with multiple threads, and as a result any Windows results are going to perform a lot worse than Linux results. On top of that, after a given number of threads (~16), MKL kind of gives up and performance drops of quite substantially.

So why test it at all? Firstly, because we need an AI benchmark, and a bad one is still better than not having one at all. Secondly, if MKL on Windows is the problem, then by publicizing the test, it might just put a boot somewhere for MKL to get fixed. To that end, we’ll stay with the benchmark as long as it remains feasible.

(2-6) AI Benchmark 0.1.2 Total

Microbenchmarks CPU Tests: Simulation and Rendering
Comments Locked

135 Comments

View All Comments

  • GeoffreyA - Friday, August 6, 2021 - link

    Okay, that makes sense.
  • 19434949 - Friday, August 6, 2021 - link

    Do you know if 5600G or 5700G can output 4k 120fps movie/video playback?
  • GeoffreyA - Saturday, August 7, 2021 - link

    Tested this now on a 2200G. Taking the Elysium trailer, I encoded a 10-second clip in H.264, H.265, and AV1 using FFmpeg. The original frame rate was 23.976, so using the -r switch, got it up to 120. Also, scaled the video from 3840x1606 to 3840x2160, and kept the audio (DTS-MA, 8 ch). On MPC-HC, they all ran smoothly. Rough CPU/GPU usage:

    H.264: 25% | 20%
    H.265: 6% | 21%
    AV1: 60% | 20%

    So if Raven Ridge can hold up at 4K/120, Cezanne should have no problem. Note that the video was downscaled during playback, owing to the screen not being 4K. Not sure if that made it easier. 8K, choppy. And VLC, lower GPU usage but got stuck on the AV1 clip.
  • GeoffreyA - Sunday, August 8, 2021 - link

    Something else I found. 10-bit H.264 seems to be going through software/CPU decoding, whereas 8-bit is running through the hardware, dropping CPU usage down to 6/8%. H.265 hasn't got the same problem. It could be the driver, the hardware, the player, or my computer.
  • mode_13h - Monday, August 9, 2021 - link

    > 10-bit H.264 seems to be going through software/CPU decoding
    ...
    > It could be the driver, the hardware, the player

    It's quite likely the driver or a GPU hardware limitation. There's some nonzero chance it's the player, but I'd bet the player tries to use acceleration and then only falls back on software when that fails.
  • GeoffreyA - Tuesday, August 10, 2021 - link

    Yes, quite likely.
  • mode_13h - Monday, August 9, 2021 - link

    > the video was downscaled during playback, owing to the screen not being 4K.
    > Not sure if that made it easier.

    Not usually. I only have detailed knowledge of H.264, where hierarchical encoding is mostly aimed at adapting to different bitrates. That would never be enabled by default, because not only is it more work for the encoder, but it's also poorly supported by decoders and generates slightly less efficient bitstreams.
  • GeoffreyA - Tuesday, August 10, 2021 - link

    "hierarchical encoding"

    That could be x264's b-pyramid, which, I think, is enabled most of the time. Apparently, allows B-frames to be used as references, saving bits. The stricter setting limits it to the Bluray spec.
  • GeoffreyA - Tuesday, August 10, 2021 - link

    By the way, looking forward to x266 coming out. AV1 is excellent, but VVC appears to be slightly ahead of it.
  • mode_13h - Wednesday, August 11, 2021 - link

    > That could be x264's b-pyramid

    No, I meant specifically Scalable Video Coding, which is what I thought, but I didn't want to cite the wrong term.

    https://en.wikipedia.org/wiki/Advanced_Video_Codin...

    The only way the decoder can opt to do less work is by discarding some layers of a SVC-encoded H.264 stream. Under any other circumstance, the decoder can't take any shortcuts without introducing a cascade of errors in all of the frames which reference the one being decoded.

    > The stricter setting limits it to the Bluray spec.

    I think blu-ray mainly limits which profiles (which are a collection of compression techniques) and levels (i.e. bitrates & resolutions) can be used, so that the bitstream can be decoded by all players and can be streamed off the disc fast enough.

    I once tried authoring valid blu-ray video dics, but the consumer-grade tools were too limiting and the free tools were a mess to figure out and use. In the end, I found simply copying MPEG-2.ts files on a BD-R would play it every player I tested. I was mainly interested in using it for a few videos shot on a phone, plus a few things I recorded from broadcast TV.

Log in

Don't have an account? Sign up now