Concluding Remarks

While the primary purpose of this exercise was just to update our datasets for future system reviews, it none the less proved to be an enlightening one, and something worth sharing. We already had an idea of what to expect going into refreshing our benchmark data for Meltdown and Spectre, and in some ways we still managed to find a surprise or two while looking at Intel's NUC7i7BNH NUC. The table below summarizes the extent of performance loss in various benchmarks.

Meltdown & Spectre Patches - Impact on the Intel NUC7i7BNH Benchmarks
Benchmark Performance Notes (Fully Patched vs. Unpatched)
BAPCo SYSmark 2014 SE - Overall -5.47%
BAPCo SYSmark 2014 SE - Office -5.17%
BAPCo SYSmark 2014 SE - Media -4.11%
BAPCo SYSmark 2014 SE - Data & Financial Analysis -2.05%
BAPCo SYSmark 2014 SE - Responsiveness -10.48%
   
Futuremark PCMark 10 Extended -2.31%
Futuremark PCMark 10 Essentials -6.56%
Futuremark PCMark 10 Productivity -8.03%
Futuremark PCMark 10 Gaming +5.56%
Futuremark PCMark 10 Digital Content Creation -0.33%
   
Futuremark PCMark 8 - Home -1.9%
Futuremark PCMark 8 - Creative -2.32%
Futuremark PCMark 8 - Work -0.83%
Futuremark PCMark 8 - Storage -1.34%
Futuremark PCMark 8 - Storage Bandwidth -29.15%
   
Futuremark PCMark 7 - PCMark Suite Score -4.03%
   
Futuremark 3DMark 11- Entry Preset +2.44%
   
Futuremark 3DMark 13 - Cloud Gate +1.14%
Futuremark 3DMark 13 - Ice Storm -13.73%
   
Agisoft Photoscan - Stage 1 -2.09%
Agisoft Photoscan - Stage 2 -12.82%
Agisoft Photoscan - Stage 3 -6.70%
Agisoft Photoscan - Stage 4 -2.84%
Agisoft Photoscan - Stage 1 (with GPU) +1.1%
Agisoft Photoscan - Stage 2 (with GPU) +1.46%
   
Cinebench R15 - Single Threaded +3.58%
Cinebench R15 - Multi-Threaded -0.32%
Cinebench R15 - Open GL +3.78%
   
x264 v5.0 - Pass I -1.1%
x264 v5.0 - Pass II -0.75%
   
7z - Compression -0.16%
7z - Decompression -0.38%

Looking at the NUC – and really this should be on the mark for most SSD-equipped Haswell+ systems – there isn't a significant universal trend. The standard for system tests such as these is +/- 3% performance variability, which covers a good chunk of the sub-benchmarks. What's left then are more meaningful performance impacts in select workloads of the BAPCo SYSmark 2014 SE and Futuremark PCMark 10 benchmarks, particularly storage-centric benchmarks. Other than those, we see certain compute workloads (such as the 2nd stage of the Agisoft Photoscan benchmark) experience a loss in performance of more than 10%.

On the whole, we see that the patches for Meltdown and Spectre affect real-world application benchmarks, but, synthetic ones are largely unaffected. The common factor among most of these benchmarks in turn is storage and I/O; the greater the number of operations, the more likely a program will feel the impact of the patches. Conversely, a compute-intensive workload that does little in the way of I/O is more or less unfazed by the changes. Though there is a certain irony to the fact that taken to its logical conclusion, patching a CPU instead renders storage performance slower, with the most impacted systems having the fastest storage.

As for what this means for future system reviews, the studies done as part of this article give us a way forward without completely invalidating all the benchmarks that we have processed in the last few years. While we can't reevaluate every last system – and so old data will need to stick around for a while longer still – these results mean that the data from unimpacted benchmarks is still valid and relevant even after the release of the Meltdown and Spectre patches. To be sure, we will be marking these results with an asterisk to denote this, but ultimately this will allow us to continue comparing new systems to older systems in at least a subset of our traditional benchmarks. Which combined with back-filling benchmarks for those older systems that we do have, lets us retain a good degree of review and benchmark continuity going forward.

Miscellaneous Benchmarks
POST A COMMENT

84 Comments

View All Comments

  • Klimax - Sunday, March 25, 2018 - link

    Just well chosen set of likely badly written projects to get wanted conclusion... Reply
  • ZolaIII - Sunday, March 25, 2018 - link

    What isn't correct there? Why don't you write better one's as source is there? The general tests as SQL, web serving or Github init create & compile times are among many real use & not synthetic benchmarks. You may not find those crucial on standard desktop, on workstation and small server they are & OS choose dictated upon them & system purpose on large server space you won't find anything called Windows there even M$ embraced Linux there (Azure platform).
    Now try to explain what is your so called "wanted conclusion"? While Linux makes his small talks how he still isn't satisfied with fixes person mostly struck with the regressions & solely most responsible one for the getting Linux to performance leading platform & on much more architectures than any other OS ever supported is Peter Zijlstra.
    Now buzz off!
    Reply
  • ZolaIII - Sunday, March 25, 2018 - link

    Linus* Reply
  • nismotigerwvu - Friday, March 23, 2018 - link

    Wow, it looks like this puts Intel behind AMD on IPC for the first time since 2006, at least until new SKUs come to market. Reply
  • mkaibear - Saturday, March 24, 2018 - link

    Nah, Zen was about 7% behind on IPC on average, this gets AMD closer but not quite there yet.

    Means picking a system based on use case has got more interesting though.
    Reply
  • Klimax - Sunday, March 25, 2018 - link

    Since AMD doesn't have patch for Specters ready, such conclusion is not warranted yet. Reply
  • phoenix_rizzen - Friday, March 23, 2018 - link

    Are the OS patch levels listed correct?

    .125 is unpatched.
    .309 is Meltdown-only patched.
    .214 is both patched.

    Should that last one be .314?
    Reply
  • ganeshts - Saturday, March 24, 2018 - link

    Initially, I wanted to present only the unpatched and fully patched results.

    Unfortunately, I have a number of PCs pending review that don't seem to be receiving BIOS updates anytime soon (despite Intel having released the final microcode for its patching). Hence, I had to add the OS-patch only scenario at the last minute after rolling back the BIOS.

    So, the order of testing was :

    1. Unpatched
    2. Both Patched
    3. Only OS patched
    Reply
  • Drazick - Saturday, March 24, 2018 - link

    How did you make your system unpatched? Reply
  • Ryan Smith - Saturday, March 24, 2018 - link

    There are registry settings available in Windows to turn off the OS patches. Steve Gibson's InSpectre can twiddle the necessary bits rather easily: https://www.grc.com/inspectre.htm Reply

Log in

Don't have an account? Sign up now