Meltdown & Spectre: Analyzing Performance Impacts on Intel's NUC7i7BNH
by Ganesh T S on March 23, 2018 4:15 PM EST- Posted in
- Systems
- Spectre
- Benchmarks
- Meltdown
Concluding Remarks
While the primary purpose of this exercise was just to update our datasets for future system reviews, it none the less proved to be an enlightening one, and something worth sharing. We already had an idea of what to expect going into refreshing our benchmark data for Meltdown and Spectre, and in some ways we still managed to find a surprise or two while looking at Intel's NUC7i7BNH NUC. The table below summarizes the extent of performance loss in various benchmarks.
| Meltdown & Spectre Patches - Impact on the Intel NUC7i7BNH Benchmarks | |
| Benchmark | Performance Notes (Fully Patched vs. Unpatched) |
| BAPCo SYSmark 2014 SE - Overall | -5.47% |
| BAPCo SYSmark 2014 SE - Office | -5.17% |
| BAPCo SYSmark 2014 SE - Media | -4.11% |
| BAPCo SYSmark 2014 SE - Data & Financial Analysis | -2.05% |
| BAPCo SYSmark 2014 SE - Responsiveness | -10.48% |
| Futuremark PCMark 10 Extended | -2.31% |
| Futuremark PCMark 10 Essentials | -6.56% |
| Futuremark PCMark 10 Productivity | -8.03% |
| Futuremark PCMark 10 Gaming | +5.56% |
| Futuremark PCMark 10 Digital Content Creation | -0.33% |
| Futuremark PCMark 8 - Home | -1.9% |
| Futuremark PCMark 8 - Creative | -2.32% |
| Futuremark PCMark 8 - Work | -0.83% |
| Futuremark PCMark 8 - Storage | -1.34% |
| Futuremark PCMark 8 - Storage Bandwidth | -29.15% |
| Futuremark PCMark 7 - PCMark Suite Score | -4.03% |
| Futuremark 3DMark 11- Entry Preset | +2.44% |
| Futuremark 3DMark 13 - Cloud Gate | +1.14% |
| Futuremark 3DMark 13 - Ice Storm | -13.73% |
| Agisoft Photoscan - Stage 1 | -2.09% |
| Agisoft Photoscan - Stage 2 | -12.82% |
| Agisoft Photoscan - Stage 3 | -6.70% |
| Agisoft Photoscan - Stage 4 | -2.84% |
| Agisoft Photoscan - Stage 1 (with GPU) | +1.1% |
| Agisoft Photoscan - Stage 2 (with GPU) | +1.46% |
| Cinebench R15 - Single Threaded | +3.58% |
| Cinebench R15 - Multi-Threaded | -0.32% |
| Cinebench R15 - Open GL | +3.78% |
| x264 v5.0 - Pass I | -1.1% |
| x264 v5.0 - Pass II | -0.75% |
| 7z - Compression | -0.16% |
| 7z - Decompression | -0.38% |
Looking at the NUC – and really this should be on the mark for most SSD-equipped Haswell+ systems – there isn't a significant universal trend. The standard for system tests such as these is +/- 3% performance variability, which covers a good chunk of the sub-benchmarks. What's left then are more meaningful performance impacts in select workloads of the BAPCo SYSmark 2014 SE and Futuremark PCMark 10 benchmarks, particularly storage-centric benchmarks. Other than those, we see certain compute workloads (such as the 2nd stage of the Agisoft Photoscan benchmark) experience a loss in performance of more than 10%.
On the whole, we see that the patches for Meltdown and Spectre affect real-world application benchmarks, but, synthetic ones are largely unaffected. The common factor among most of these benchmarks in turn is storage and I/O; the greater the number of operations, the more likely a program will feel the impact of the patches. Conversely, a compute-intensive workload that does little in the way of I/O is more or less unfazed by the changes. Though there is a certain irony to the fact that taken to its logical conclusion, patching a CPU instead renders storage performance slower, with the most impacted systems having the fastest storage.
As for what this means for future system reviews, the studies done as part of this article give us a way forward without completely invalidating all the benchmarks that we have processed in the last few years. While we can't reevaluate every last system – and so old data will need to stick around for a while longer still – these results mean that the data from unimpacted benchmarks is still valid and relevant even after the release of the Meltdown and Spectre patches. To be sure, we will be marking these results with an asterisk to denote this, but ultimately this will allow us to continue comparing new systems to older systems in at least a subset of our traditional benchmarks. Which combined with back-filling benchmarks for those older systems that we do have, lets us retain a good degree of review and benchmark continuity going forward.
Facebook
Twitter
RSS
83 Comments
View All Comments
Samus - Saturday, March 24, 2018 - link
It isn't a bug, or a design flaw. It's just an exploit of the architecture.Saying otherwise is like saying houses not built for category 3 hurricanes in New York have a design flaw when the area has never needed construction to that spec. But with climate change, the need is becoming necessary as the architecture is no longer fit for the climate.
Not a great analogy, but in the same example, neither Intel nor construction designers anticipated the architecture would become flawed due to unforeseen circumstances.
Alexvrb - Saturday, March 24, 2018 - link
It's definitely a flaw. Unintentional security flaws are still flaws. It's an exploit of a security flaw in the architecture. They'll release a CPU in the future which still have speculative execution and isn't vulnerable to this flaw, at an architecture level. It might have other flaws, however. CPU flaws, bugs, errata are very common.HStewart - Saturday, March 24, 2018 - link
"It isn't a bug, or a design flaw. It's just an exploit of the architecture."It is attempt to distract on Intel productions, that also back fire because it also effects ARM and AMD.
bji - Sunday, March 25, 2018 - link
That's not really true. There is a class of very hard to exploit design flaws in most implementations of speculative execution that seem to be systemic to almost all chips, that is true. These are so hard to exploit as to be nearly unexploitable in my opinion. These are called Spectre.But there is also a much more significant and easy to exploit design flaw. This affects only Intel chips (and I guess some ARM chips too -- but not AMD). This is called Meltdown.
So there is no distracting going on here. Almost all chips are affected by spectre, but it's so hard to exploit as to almost be irrelevant. Meltdown is serious and it's Intel only.
Manch - Monday, March 26, 2018 - link
HStewart is an Intel shill/fan boy. In his mind Intel can do no wrong. You're wasting your breath arguing with him.bcronce - Sunday, March 25, 2018 - link
A bug is when something does not work to spec.boozed - Sunday, March 25, 2018 - link
I bet you $10 the spec doesn't say "our branch prediction should have this massive security flaw".linuxgeex - Monday, March 26, 2018 - link
Correct, it's a design flaw not a bug. When something operates as designed it isn't a bug, but that doesn't make it correct either. Intel is rightly getting sued by the people who are significantly affected and who can afford to battle Intel in court... that will be people who forked over millions to obtain small performance improvements based on Intel's claims of security, performance, and fitness for the purpose for which it was sold, which later was found to be false. They are being sued exactly the same way that Honda would be sued if they supplied a Formula One team with an engine that performed 30% slower than advertised.Alexvrb - Saturday, March 24, 2018 - link
It's a CPU vulnerability.willis936 - Friday, March 23, 2018 - link
The power consumption drop with performance is interesting. I'd be interested in seeing a comparison of efficiency pre and post patch. At a glance t's difficult to tell if efficiency has gone up, down, or stayed the same. I'm under the impression that this patch disables speculative execution entirely which, unless speculation takes a lot more power than I think, efficiency should go down.