Meltdown & Spectre: Analyzing Performance Impacts on Intel's NUC7i7BNH
by Ganesh T S on March 23, 2018 4:15 PM EST- Posted in
- Systems
- Spectre
- Benchmarks
- Meltdown
Concluding Remarks
While the primary purpose of this exercise was just to update our datasets for future system reviews, it none the less proved to be an enlightening one, and something worth sharing. We already had an idea of what to expect going into refreshing our benchmark data for Meltdown and Spectre, and in some ways we still managed to find a surprise or two while looking at Intel's NUC7i7BNH NUC. The table below summarizes the extent of performance loss in various benchmarks.
| Meltdown & Spectre Patches - Impact on the Intel NUC7i7BNH Benchmarks | |
| Benchmark | Performance Notes (Fully Patched vs. Unpatched) |
| BAPCo SYSmark 2014 SE - Overall | -5.47% |
| BAPCo SYSmark 2014 SE - Office | -5.17% |
| BAPCo SYSmark 2014 SE - Media | -4.11% |
| BAPCo SYSmark 2014 SE - Data & Financial Analysis | -2.05% |
| BAPCo SYSmark 2014 SE - Responsiveness | -10.48% |
| Futuremark PCMark 10 Extended | -2.31% |
| Futuremark PCMark 10 Essentials | -6.56% |
| Futuremark PCMark 10 Productivity | -8.03% |
| Futuremark PCMark 10 Gaming | +5.56% |
| Futuremark PCMark 10 Digital Content Creation | -0.33% |
| Futuremark PCMark 8 - Home | -1.9% |
| Futuremark PCMark 8 - Creative | -2.32% |
| Futuremark PCMark 8 - Work | -0.83% |
| Futuremark PCMark 8 - Storage | -1.34% |
| Futuremark PCMark 8 - Storage Bandwidth | -29.15% |
| Futuremark PCMark 7 - PCMark Suite Score | -4.03% |
| Futuremark 3DMark 11- Entry Preset | +2.44% |
| Futuremark 3DMark 13 - Cloud Gate | +1.14% |
| Futuremark 3DMark 13 - Ice Storm | -13.73% |
| Agisoft Photoscan - Stage 1 | -2.09% |
| Agisoft Photoscan - Stage 2 | -12.82% |
| Agisoft Photoscan - Stage 3 | -6.70% |
| Agisoft Photoscan - Stage 4 | -2.84% |
| Agisoft Photoscan - Stage 1 (with GPU) | +1.1% |
| Agisoft Photoscan - Stage 2 (with GPU) | +1.46% |
| Cinebench R15 - Single Threaded | +3.58% |
| Cinebench R15 - Multi-Threaded | -0.32% |
| Cinebench R15 - Open GL | +3.78% |
| x264 v5.0 - Pass I | -1.1% |
| x264 v5.0 - Pass II | -0.75% |
| 7z - Compression | -0.16% |
| 7z - Decompression | -0.38% |
Looking at the NUC – and really this should be on the mark for most SSD-equipped Haswell+ systems – there isn't a significant universal trend. The standard for system tests such as these is +/- 3% performance variability, which covers a good chunk of the sub-benchmarks. What's left then are more meaningful performance impacts in select workloads of the BAPCo SYSmark 2014 SE and Futuremark PCMark 10 benchmarks, particularly storage-centric benchmarks. Other than those, we see certain compute workloads (such as the 2nd stage of the Agisoft Photoscan benchmark) experience a loss in performance of more than 10%.
On the whole, we see that the patches for Meltdown and Spectre affect real-world application benchmarks, but, synthetic ones are largely unaffected. The common factor among most of these benchmarks in turn is storage and I/O; the greater the number of operations, the more likely a program will feel the impact of the patches. Conversely, a compute-intensive workload that does little in the way of I/O is more or less unfazed by the changes. Though there is a certain irony to the fact that taken to its logical conclusion, patching a CPU instead renders storage performance slower, with the most impacted systems having the fastest storage.
As for what this means for future system reviews, the studies done as part of this article give us a way forward without completely invalidating all the benchmarks that we have processed in the last few years. While we can't reevaluate every last system – and so old data will need to stick around for a while longer still – these results mean that the data from unimpacted benchmarks is still valid and relevant even after the release of the Meltdown and Spectre patches. To be sure, we will be marking these results with an asterisk to denote this, but ultimately this will allow us to continue comparing new systems to older systems in at least a subset of our traditional benchmarks. Which combined with back-filling benchmarks for those older systems that we do have, lets us retain a good degree of review and benchmark continuity going forward.
Facebook
Twitter
RSS
83 Comments
View All Comments
boeush - Friday, March 23, 2018 - link
Speculative execution uses up compute cycles and can cause excessive memory loads and cache thrashing - which amount to wasted power and in some cache-sensitive cases, possibly even a drop in performance - when the speculation is frequently-enough incorrect (i.e. when the actual branch taken doesn't match the CPU's guess.)I'd expect that disabling speculative execution under high load (e.g. benchmarking scenarios) should normally result in improved power efficiency (avoiding wasted computation and I/O) - but at the cost of raw compute performance. In less intense, more 'bursty' scenarios, where the CPU spends a lot of time in an idle state, the "hurry up and rest" dynamic might strongly reduce the overall power waste of speculative execution, as the CPU would spend less time in an active-but-stalled state while spending more time in a sleep state...
Cravenmor - Friday, March 23, 2018 - link
The thing that caught my eye was the reduction in power from the patches. I wonder what to deduct from speculative function and whether it's inefficient.Lord of the Bored - Saturday, March 24, 2018 - link
Speculative execution does add somewhat to the power load. That's why Atom parts were in-order for a long time, and many ARM parts still are.Cravenmor - Friday, March 23, 2018 - link
willis936 beat to it by a noseeva02langley - Friday, March 23, 2018 - link
It is interesting nonetheless. The storage data is absolutely devastating. Can we make conclusions to the server world from Intel? I don't know since servers are still using hard drives. However, it might force companies to switch to Epyc or to upgrade to Canon Lake. It would be interesting.boeush - Friday, March 23, 2018 - link
The CPU used in these tests was a low-power 2-core - pretty weak to begin with. Knock some performance off the top, and you have detectable impact on I/O.Probably the impact would be much less severe with a more powerful CPU: where the test scenario would again 'flip' from CPU-bound to bus/storage device performance- limited.
Reflex - Friday, March 23, 2018 - link
Also, servers are usually only using NVMe drives as cache, SAS is less likely to have significant impact.Drazick - Friday, March 23, 2018 - link
This is a great analysis.We'd be happy to have more like this (On various performance impacting situations).
I'd be happy to have a guide how to prevent the patching for each OS (Windows, macOS, Linux) as the private user mostly has no reason to be afraid of those.
Thank You!
ZolaIII - Friday, March 23, 2018 - link
I found this comparation much more interesting.https://www.phoronix.com/scan.php?page=article&...
It's done on much more capable system which whose more hit in the first place & at least some benchmarks are representative in real usage workloads. Seams M$ again did a bad job & chubby Linus is still not satisfied with up to date results so future work still carries on.
Klimax - Sunday, March 25, 2018 - link
Not really correct...