Simulation

Simulation and Science have a lot of overlap in the benchmarking world, however for this distinction we’re separating into two segments mostly based on the utility of the resulting data. The benchmarks that fall under Science have a distinct use for the data they output – in our Simulation section, these act more like synthetics but at some level are still trying to simulate a given environment.

DigiCortex v1.35: link

DigiCortex is a pet project for the visualization of neuron and synapse activity in the brain. The software comes with a variety of benchmark modes, and we take the small benchmark which runs a 32k neuron/1.8B synapse simulation, similar to a small slug.

The results on the output are given as a fraction of whether the system can simulate in real-time, so anything above a value of one is suitable for real-time work. The benchmark offers a 'no firing synapse' mode, which in essence detects DRAM and bus speed, however we take the firing mode which adds CPU work with every firing.

The software originally shipped with a benchmark that recorded the first few cycles and output a result. So while fast multi-threaded processors this made the benchmark last less than a few seconds, slow dual-core processors could be running for almost an hour. There is also the issue of DigiCortex starting with a base neuron/synapse map in ‘off mode’, giving a high result in the first few cycles as none of the nodes are currently active. We found that the performance settles down into a steady state after a while (when the model is actively in use), so we asked the author to allow for a ‘warm-up’ phase and for the benchmark to be the average over a second sample time.

For our test, we give the benchmark 20000 cycles to warm up and then take the data over the next 10000 cycles seconds for the test – on a modern processor this takes 30 seconds and 150 seconds respectively. This is then repeated a minimum of 10 times, with the first three results rejected. Results are shown as a multiple of real-time calculation.

(3-1) DigiCortex 1.35 (32k Neuron, 1.8B Synapse)

Dwarf Fortress 0.44.12: Link

Another long standing request for our benchmark suite has been Dwarf Fortress, a popular management/roguelike indie video game, first launched in 2006 and still being regularly updated today, aiming for a Steam launch sometime in the future.

Emulating the ASCII interfaces of old, this title is a rather complex beast, which can generate environments subject to millennia of rule, famous faces, peasants, and key historical figures and events. The further you get into the game, depending on the size of the world, the slower it becomes as it has to simulate more famous people, more world events, and the natural way that humanoid creatures take over an environment. Like some kind of virus.

For our test we’re using DFMark. DFMark is a benchmark built by vorsgren on the Bay12Forums that gives two different modes built on DFHack: world generation and embark. These tests can be configured, but range anywhere from 3 minutes to several hours. After analyzing the test, we ended up going for

  • A Medium World, 127x127, with 550 years, 10 civilizations and 4 megabeasts

DFMark outputs the time to run any given test, so this is what we use for the output. We loop the test for as many times possible in 30 minutes.

(3-2b) Dwarf Fortress 0.44.12 World Gen 129x129, 550 Yr

We saw a few instances where the 35W/45W results were almost identical, with the margin that the 35W would come out ahead in single threaded tasks. This may be because 35W was a fixed setting in the software options, whereas 45W was the power management framework in action.

Dolphin v5.0 Emulation: Link

Many emulators are often bound by single thread CPU performance, and general reports tended to suggest that Haswell provided a significant boost to emulator performance. This benchmark runs a Wii program that ray traces a complex 3D scene inside the Dolphin Wii emulator. Performance on this benchmark is a good proxy of the speed of Dolphin CPU emulation, which is an intensive single core task using most aspects of a CPU. Results are given in seconds, where the Wii itself scores 1051 seconds.

(3-3) Dolphin 5.0 Render Test

 

Rendering

Rendering tests, compared to others, are often a little more simple to digest and automate. All the tests put out some sort of score or time, usually in an obtainable way that makes it fairly easy to extract. These tests are some of the most strenuous in our list, due to the highly threaded nature of rendering and ray-tracing, and can draw a lot of power. If a system is not properly configured to deal with the thermal requirements of the processor, the rendering benchmarks is where it would show most easily as the frequency drops over a sustained period of time. Most benchmarks in this case are re-run several times, and the key to this is having an appropriate idle/wait time between benchmarks to allow for temperatures to normalize from the last test.

Blender 2.83 LTS: Link

One of the popular tools for rendering is Blender, with it being a public open source project that anyone in the animation industry can get involved in. This extends to conferences, use in films and VR, with a dedicated Blender Institute, and everything you might expect from a professional software package (except perhaps a professional grade support package). With it being open-source, studios can customize it in as many ways as they need to get the results they require. It ends up being a big optimization target for both Intel and AMD in this regard.

For benchmarking purposes, we fell back to one rendering a frame from a detailed project. Most reviews, as we have done in the past, focus on one of the classic Blender renders, known as BMW_27. It can take anywhere from a few minutes to almost an hour on a regular system. However now that Blender has moved onto a Long Term Support model (LTS) with the latest 2.83 release, we decided to go for something different.

We use this scene, called PartyTug at 6AM by Ian Hubert, which is the official image of Blender 2.83. It is 44.3 MB in size, and uses some of the more modern compute properties of Blender. As it is more complex than the BMW scene, but uses different aspects of the compute model, time to process is roughly similar to before. We loop the scene for at least 10 minutes, taking the average time of the completions taken. Blender offers a command-line tool for batch commands, and we redirect the output into a text file.

(4-1) Blender 2.83 Custom Render Test

Corona 1.3: Link

Corona is billed as a popular high-performance photorealistic rendering engine for 3ds Max, with development for Cinema 4D support as well. In order to promote the software, the developers produced a downloadable benchmark on the 1.3 version of the software, with a ray-traced scene involving a military vehicle and a lot of foliage. The software does multiple passes, calculating the scene, geometry, preconditioning and rendering, with performance measured in the time to finish the benchmark (the official metric used on their website) or in rays per second (the metric we use to offer a more linear scale).

The standard benchmark provided by Corona is interface driven: the scene is calculated and displayed in front of the user, with the ability to upload the result to their online database. We got in contact with the developers, who provided us with a non-interface version that allowed for command-line entry and retrieval of the results very easily.  We loop around the benchmark five times, waiting 60 seconds between each, and taking an overall average. The time to run this benchmark can be around 10 minutes on a Core i9, up to over an hour on a quad-core 2014 AMD processor or dual-core Pentium.

(4-2) Corona 1.3 Benchmark

POV-Ray 3.7.1: Link

A long time benchmark staple, POV-Ray is another rendering program that is well known to load up every single thread in a system, regardless of cache and memory levels. After a long period of POV-Ray 3.7 being the latest official release, when AMD launched Ryzen the POV-Ray codebase suddenly saw a range of activity from both AMD and Intel, knowing that the software (with the built-in benchmark) would be an optimization tool for the hardware.

We had to stick a flag in the sand when it came to selecting the version that was fair to both AMD and Intel, and still relevant to end-users. Version 3.7.1 fixes a significant bug in the early 2017 code that was advised against in both Intel and AMD manuals regarding to write-after-read, leading to a nice performance boost.

The benchmark can take over 20 minutes on a slow system with few cores, or around a minute or two on a fast system, or seconds with a dual high-core count EPYC. Because POV-Ray draws a large amount of power and current, it is important to make sure the cooling is sufficient here and the system stays in its high-power state. Using a motherboard with a poor power-delivery and low airflow could create an issue that won’t be obvious in some CPU positioning if the power limit only causes a 100 MHz drop as it changes P-states.

(4-4) POV-Ray 3.7.1

V-Ray: Link

We have a couple of renderers and ray tracers in our suite already, however V-Ray’s benchmark came through for a requested benchmark enough for us to roll it into our suite. Built by ChaosGroup, V-Ray is a 3D rendering package compatible with a number of popular commercial imaging applications, such as 3ds Max, Maya, Undreal, Cinema 4D, and Blender.

We run the standard standalone benchmark application, but in an automated fashion to pull out the result in the form of kilosamples/second. We run the test six times and take an average of the valid results.

(4-5) V-Ray Renderer

Cinebench R20: Link

Another common stable of a benchmark suite is Cinebench. Based on Cinema4D, Cinebench is a purpose built benchmark machine that renders a scene with both single and multi-threaded options. The scene is identical in both cases. The R20 version means that it targets Cinema 4D R20, a slightly older version of the software which is currently on version R21. Cinebench R20 was launched given that the R15 version had been out a long time, and despite the difference between the benchmark and the latest version of the software on which it is based, Cinebench results are often quoted a lot in marketing materials.

Results for Cinebench R20 are not comparable to R15 or older, because both the scene being used is different, but also the updates in the code bath. The results are output as a score from the software, which is directly proportional to the time taken. Using the benchmark flags for single CPU and multi-CPU workloads, we run the software from the command line which opens the test, runs it, and dumps the result into the console which is redirected to a text file. The test is repeated for a minimum of 10 minutes for both ST and MT, and then the runs averaged.

(4-6a) CineBench R20 Single Thread(4-6b) CineBench R20 Multi-Thread

CPU Tests: Office and Science CPU Tests: Encoding
Comments Locked

92 Comments

View All Comments

  • mode_13h - Friday, March 18, 2022 - link

    > I haven't seen most of the anime you mentioned

    The Patlabor OAVs and movies are good for an 80's/90's nostalgia hit, IMO. It's that sort of old timey mecha anime that attracted many of us to the anime multiverse, in the first place. When I watched the OAVs on blu ray, the restoration was fantastic. Very crisp HD.

    > the new Dune was a big disappointment to me.

    Yeah, I read the trilogy before watching it. I'm so glad I did, because I knew exactly what was going on. Otherwise, I might've been lost.

    Yeah, they tried too hard to follow the narrative of the book. I think the only way to do it, and still end up with a good movie, is to focus on a particular story arc. If they'd nailed it, that would've set the stage for more to follow.

    I'd imagine they shot so much footage that it could conceivably be re-edited. I know they didn't shoot beyond the movie's ending, because I saw an interview with Zendaya, where she said she was only on location for the desert scenes for a couple days.
  • mode_13h - Friday, March 18, 2022 - link

    I mean the original Patlabor OAVs. There are about 7 of them, I think. Near movie-quality animation, for the time.

    BTW, some aspects of the Ghost in the Shell franchise definitely make more sense, as we move towards the future it predicted.

    Oh, and Planetes is a nice series about a crew working to collect orbital debris. It aired about 2 decades ago, but I think it was based on manga that was older, still. Some aspects of it were a bit anachronistic even for the time, but other aspects about space physics and orbital living clearly received a lot of thought and attention. The story arc is a lot more interesting than it sounds, with lots of commentary about life, love, the privatization of space, corporate politics, geopolitics, personal ambition, and the ultimate path and personal costs of space exploration. If you don't mind a bit of slapstick and are willing to look past some of the more anachronistic aspects, it's worth a watch.
  • GeoffreyA - Saturday, March 19, 2022 - link

    I haven't read the books but hope to do so before going into the coffin. Well, my view is that the excessive realism somehow harms the movie. If you go back to Lynch's 1984 version, despite the outlandish visuals, it is pretty alien, as Dune should be; and from a storytelling point of view, does that pretty well, going forward rapidly. Also, the princess's summary in the beginning got the viewer up to speed with this strange universe.

    The new movie took "show, don't tell" a bit too far, and the story didn't feel cohesive or unified, especially towards the end. It was tedious. The visions seemed forced and overdone. And for an epic, the cinematography was poor in my opinion. A key problem, I feel, is that it didn't bring out the true spirit of the desert. Coupled to this issue is Zimmer's music, in my view, missing the mark. It was too loud and vulgar, and seemed to view the desert from a commercialised, Hollywood lens, rather than feeling its power and reflecting that desolation. Then the CGI, I say no more.

    On the plus side, two sequences were outstanding: when Paul first steps onto the desert and picks up the sand/spice; and Paul and his mother's flight through the dust storm. That was world class.
  • GeoffreyA - Saturday, March 19, 2022 - link

    When I saw the Patlabor poster some time ago, I was intrigued. That was part two I believe. Yes, as time goes by, I prefer to look back at older anime. I think it's fair to say the industry has gone downhill these days.

    And thanks for that great description of Planetes. I won't mind giving one or two episodes a go and seeing what it's like. It reminds me that I've still got to watch Cowboy Bepop.
  • mode_13h - Sunday, March 20, 2022 - link

    Planetes is one of those series that takes a while to get going. The further you go, the deeper it gets. If you really don't like the first couple episodes, maybe it's not for you. However, you do get rewarded the longer you stick with it.
  • mode_13h - Monday, March 21, 2022 - link

    > I haven't read the books but hope to do so before going into the coffin.

    There's a lot you can read into it about the corrupting tendencies of empires and exploitation of peoples and their natural resources. It feels like it might've tapped into the decolonization zeitgeist, or at least what I presume it should've been, as the former colonial powers of Europe unwound their foreign holdings. I could do without so much of the psychedelic stuff, but I know Heinlein also went there. So, maybe that was just another trend in 1960's sci fi.

    > If you go back to Lynch's 1984 version

    It's funny this came up, because I just started watching it last weekend and finished it mere hours ago. It did seem a bit overwrought. I remember how he seemed rather too fascinated with the perversions and excesses of the Harkkonen. I thought the exposition was a bit too much for the naive movie-goer, but probably a helpful reminder for those who'd read the books years earlier.

    Since I came to it with low expectations, I really wasn't disappointed. Since the movie had many shots in low light, I wonder just how much I benefited from seeing a clean, HD presentation. Overall, I guess my main complaints would be that some of the acting seemed sub-par (Sting, for one, definitely should've stayed focused on the music business) and I just wouldn't have tried to cover so much plot. It felt busy and probably hard for people to follow, without having read it. I wasn't too bothered by the dated special effects, but they do kind of jump out at you. Some of the sets were quite impressive.

    > The new movie took "show, don't tell" a bit too far

    Probably a reaction to Lynch's version. I also wonder if his 1990 TV series, Twin Peaks, was also sort of a reaction to the criticism he got for too much exposition in Dune.

    Anyway, the last I'll say about it is that I'm finding the Wikipedia page on Dune to be a good resource on the author and his influences.
  • GeoffreyA - Monday, March 21, 2022 - link

    The visuals prevented me from watching it for a long time. It was only after I became a fan of David Lynch that I was able to see past that and appreciate what he had done. I think for a two-hour adaptation of such a vast novel, it is a commendable attempt, and I prefer it. As for the excesses, etc., well, that's Lynch as always. He always tends to bring out the darker side of things.

    I actually love Twin Peaks, and the recent season 3 was spectacular, if strange. But strange is this man's domain. Did too much exposition have an effect on his later work? I would say that Dune was an exception. Generally, his films are pretty obscure, nothing much being spelled out, and one often has to piece together a puzzle. He started off with that note in Eraserhead and hasn't really changed in four decades.
  • GeoffreyA - Monday, March 21, 2022 - link

    "Sting, for one, definitely should've stayed focused on the music business"

    The best line!
  • mode_13h - Tuesday, March 22, 2022 - link

    > I actually love Twin Peaks

    I never really watched it. My older sister watched the original TV airing. All I remember of it was the general strangeness and a recollection that even its conclusion left much unanswered.

    I haven't seen much of Lynch's work, but I did enjoy Mulholland Dr. All I remember from it is that I decided it's a fool's errand to make complete sense of the plot, since there were paradoxes inserted seemingly with the intent to break any strict interpretation.

    > Did too much exposition have an effect on his later work?

    That's not really what I meant. I was suggesting he got too much negative feedback on all the exposition in Dune, and therefore went too far in the other direction of being too obscure.

    From what I've heard, Kubrick would sometimes indulge in excessive obscurity to create a false sense of depth. The prime example being 2001: A Space Odyssey. If you actually read the book, you can supposedly see what kind of shenanigans he got up to, which I've heard he even admitted in an interview.

    > He started off with that note in Eraserhead and hasn't really changed in four decades.

    Ah, right. I never got round to watching that one.
  • GeoffreyA - Wednesday, March 23, 2022 - link

    Mulholland Drive is perhaps my favourite film of all films. Again, it's the tragic note that speaks to me, and Naomi Watts, brilliant. I've racked my head over this story a great deal, and my tentative answer is that even the latter part, Diane's tale, is a dream, various pieces of evidence pointing there, particularly the blue box and the smoke. The question is, whose dream is *that*! Perhaps it's the director's dream after all.

    I haven't read 2001, but you're right, the film keeps things pretty bare and mysterious, and that creates the feeling of a deep, even terrible, mystery. Our age could actually learn something from that and stop filling in all the details. The human mind does a far better job at piecing together the monster in the shadows.

Log in

Don't have an account? Sign up now