CPU Tests: Legacy and Web

In order to gather data to compare with older benchmarks, we are still keeping a number of tests under our ‘legacy’ section. This includes all the former major versions of CineBench (R15, R11.5, R10) as well as x264 HD 3.0 and the first very naïve version of 3DPM v2.1. We won’t be transferring the data over from the old testing into Bench, otherwise it would be populated with 200 CPUs with only one data point, so it will fill up as we test more CPUs like the others.

The other section here is our web tests.

Web Tests: Kraken, Octane, and Speedometer

Benchmarking using web tools is always a bit difficult. Browsers change almost daily, and the way the web is used changes even quicker. While there is some scope for advanced computational based benchmarks, most users care about responsiveness, which requires a strong back-end to work quickly to provide on the front-end. The benchmarks we chose for our web tests are essentially industry standards – at least once upon a time.

It should be noted that for each test, the browser is closed and re-opened a new with a fresh cache. We use a fixed Chromium version for our tests with the update capabilities removed to ensure consistency.

Mozilla Kraken 1.1

Kraken is a 2010 benchmark from Mozilla and does a series of JavaScript tests. These tests are a little more involved than previous tests, looking at artificial intelligence, audio manipulation, image manipulation, json parsing, and cryptographic functions. The benchmark starts with an initial download of data for the audio and imaging, and then runs through 10 times giving a timed result.

We loop through the 10-run test four times (so that’s a total of 40 runs), and average the four end-results. The result is given as time to complete the test, and we’re reaching a slow asymptotic limit with regards the highest IPC processors.

(7-1) Kraken 1.1 Web Test

Google Octane 2.0

Our second test is also JavaScript based, but uses a lot more variation of newer JS techniques, such as object-oriented programming, kernel simulation, object creation/destruction, garbage collection, array manipulations, compiler latency and code execution.

Octane was developed after the discontinuation of other tests, with the goal of being more web-like than previous tests. It has been a popular benchmark, making it an obvious target for optimizations in the JavaScript engines. Ultimately it was retired in early 2017 due to this, although it is still widely used as a tool to determine general CPU performance in a number of web tasks.

(7-2) Google Octane 2.0 Web Test

Speedometer 2: JavaScript Frameworks

Our newest web test is Speedometer 2, which is a test over a series of JavaScript frameworks to do three simple things: built a list, enable each item in the list, and remove the list. All the frameworks implement the same visual cues, but obviously apply them from different coding angles.

Our test goes through the list of frameworks, and produces a final score indicative of ‘rpm’, one of the benchmarks internal metrics.

We repeat over the benchmark for a dozen loops, taking the average of the last five.

(7-3) Speedometer 2.0 Web Test

Legacy Tests

(6-5a) x264 HD 3.0 Pass 1(6-5b) x264 HD 3.0 Pass 2

(6-4a) 3DPM v1 ST(6-4b) 3DPM v1 MT

(6-3a) CineBench R15 ST(6-3b) CineBench R15 MT

CPU Tests: Simulation CPU Tests: Synthetic
Comments Locked

98 Comments

View All Comments

  • DesireeTR - Wednesday, July 14, 2021 - link

    OK, found the news too. IDK if I can link any other website here other than Anandtech, but look for "Lenovo is Using AMD PSB to Vendor Lock AMD CPUs" from servethehome, dated April 5th 2021. Lenovo P620 with Threadripper Pro was tested and found that they used the strict PSB lock-in like Dell do on their PowerEdge servers.
  • Threska - Wednesday, July 14, 2021 - link

    I think "permanently" is the biggest concern, otherwise it could be a great feature as part of a "root of trust" if the user could control it, especially via hardware modification. e.g. jumper.
  • DesireeTR - Wednesday, July 14, 2021 - link

    Yeah, and if this trend continues, the Ryzen PRO definitely is next on the line getting this PSB. Laptops might be OK, since they use soldered BGA processor anyway, but definitely a big no no for prebuild towers.
  • arashi - Saturday, July 17, 2021 - link

    If it can be overridden like that then it isn't a root of trust anymore.
  • Threska - Saturday, July 17, 2021 - link

    There's the presumption you trust yourself.
  • Mikewind Dale - Wednesday, July 14, 2021 - link

    Dear Anandtech: If you ever review the motherboards, I'll relate something a few things I discovered about the Supermicro M12SWA-TF:

    First, it cannot use sleep mode. If you put the computer to sleep, then when you wake it up, the fans will all spin at low RPM, and they will fail to adjust to temperature. HWiNFO64 reports two sets of sensors: one set is direct, and the other is indirect, via the IPMI. After waking from sleep, the direct sensor readings were still reported, but the indirect-via-IPMI sensors were all null. When I logged into the BMC/IPMI, all the sensors were null there too. And when I ran a CPU burn-in after waking from sleep, my CPU temperature quickly climbed higher than normal, and the fans did NOT ramp up their RPM. (I was prepared for this, so I was running only a single-threaded CPU benchmark.)

    Not only did rebooting the computer fix the problem, but so did Windows hibernate. The fact that Windows hibernate fixed the problem told me that the problem was hardware, not OS.

    I contacted Supermicro, and they said this behavior is normal (!!!!!!). They explained that the IPMI controls the fan RPM, but it only connects to the sensors during POST. If you put the computer to sleep, the IPMI loses its connection to the sensors, and it cannot resume that connection until the computer POSTs again.

    So if you review the motherboards, make sure to test the sleep behavior.

    Second, the Supermicro board is programmed with critical low fan RPM threshholds that are lower than Noctua's RPM. If you Google, you'll see a lot of people have problems with using Noctua fans with Supermicro boards. What happens is, the the Noctua fan's RPM will drop below the critical low RPM threshholds, so the Supermicro board will think the fan is failing, and it will quickly ramp the fan up to 100% PWM. Once the fan exceeds the critical low RPM threshold, the alert will end, and the fan will drop its RPM back down again, starting the cycle over. So the fans cycle back and forth between high and low RPM. When I logged into the IPMI, I saw that I every single fan was triggering the low RPM alert every few seconds.

    The solution is to reprogram the IPMI with new critical low RPM thresholds. Supermicro's own IPMI software does NOT allow this, because Supermicro explained to me that some people have overheated and fried their motherboards using insufficient cooling. So I had to use a third-party tool called "ipmitool".

    Usually, ipmitool is obtained via "sudo apt-get install ipmitool". However, I found that the Linux version was unable to establish a connection with my BMC, even though other IPMI tools had no problem with establishing that connection. But other IPMI tools did not have the ability to reprogram the fan thresholds.

    Luckily, the Windows version of ipmitool was able to establish a connection and alter my fan thresholds just fine. The Windows version is available at https://www.dannynieuwenhuis.nl/download-windows-i...

    If you Google, you'll find many, many different websites offering instructions for how to use ipmitool to modify your Supermicro board to be compatible with Noctua fans. I'll just give a few sample lines of code here, in case anyone needs them:

    ipmitool -I lanplus -H <ipaddress> -U <username> -P <password> sensor thresh FAN1 lower 40 140 240
    ipmitool -I lanplus -H <ipaddress> -U <username> -P <password> sensor thresh FAN1 upper 1650 1750 1850

    Where:
    --- FAN1 is the name of the fan header, as labeled in the motherboard manual. Options are FAN1-FAN6 and FANA-FAND.
    --- "lower" numbers are lower non-recoverable, lower critical, and lower non-critical, in that order.
    --- "upper" numbers are upper non-critical, upper critical, and upper non-recoverable, in that order.

    To calculate the thresholds, I did the following:
    First, I looked up Noctua's specs. FAN1 is my Noctua NH-U14S TR4-SP3. According to Noctua, its fan's RPM are 300 +/-20% to 1500 +/- 10% RPM.
    Second, I set the lower non-critical to 300*0.8 (i.e. -20%) and the upper non-critical to 1500*1.1 (i.e. +10%).
    Third, for the critical and non-recoverable thresholds, I just added or subtracted 100%.

    Do the same for every other fan in every other header. I wrote about every line in a .BAT file in Windows, which read like this:

    REM **************************************************************************************************
    REM **********
    REM FAN1 is Noctua NH-U14S TR4-SP3: 300 +/-20% to 1500 +/- 10% RPM
    REM **********

    ipmitool -I lanplus -H <ipaddress> -U <username> -P <password> sensor thresh FAN1 lower 40 140 240
    ipmitool -I lanplus -H <ipaddress> -U <username> -P <password> sensor thresh FAN1 upper 1650 1750 1850
    REM **************************************************************************************************

    REM **************************************************************************************************
    REM **********
    REM FAN2 is Noctua NF-A15: 300 +/- 20% to 1200 +/- 10%
    REM **********

    ipmitool -I lanplus -H <ipaddress> -U <username> -P <password> sensor thresh FAN2 lower 40 140 240
    ipmitool -I lanplus -H <ipaddress> -U <username> -P <password> sensor thresh FAN2 upper 1320 1420 1520
    REM **************************************************************************************************

    and so forth, for every fan header. This successfully solved the problem of the fans triggering the threshold alerts and cycling up and down.
  • Mikewind Dale - Wednesday, July 14, 2021 - link

    "Second, the Supermicro board is programmed with critical low fan RPM threshholds that are lower than Noctua's RPM."

    I meant *higher*. The Supermicro default critical low fan RPM thresholds are *higher* than Noctua's.
  • Mikewind Dale - Wednesday, July 14, 2021 - link

    Oh, and because sleep mode is dangerous, threatening to potentially fry your CPU (since the fans no longer respond to temperature), I not only set my computer never to sleep, but I removed sleep from the power options in the start menu. That way, I cannot accidentally put the computer to sleep.

    If you do ever put your Supermicro M12SWA-TF to sleep, you will not receive any alerts that every sensor is null. If you log into the BMC, you'll see every sensor is null, but there are no alerts. And the fans all spin at minimum RPM regardless of your fan setting, and regardless of temperature. So sleep mode appears to have the potential to fry your CPU.
  • Threska - Wednesday, July 14, 2021 - link

    You keep saying "fry" but haven't CPUs had thermal protection for ages at this point?
  • Mikewind Dale - Wednesday, July 14, 2021 - link

    Threska, possibly. But I didn't want to find out.

    At best, sleep mode would cause the computer to constantly downclock or shut down without any clear cause (unless the user realized it was because sleep mode deactivated the IPMI's reporting of the sensors while the sensors themselves were still reporting values to software such as HWiNFO).

Log in

Don't have an account? Sign up now