Meltdown & Spectre: Analyzing Performance Impacts on Intel's NUC7i7BNH
by Ganesh T S on March 23, 2018 4:15 PM EST- Posted in
- Systems
- Spectre
- Benchmarks
- Meltdown
Concluding Remarks
While the primary purpose of this exercise was just to update our datasets for future system reviews, it none the less proved to be an enlightening one, and something worth sharing. We already had an idea of what to expect going into refreshing our benchmark data for Meltdown and Spectre, and in some ways we still managed to find a surprise or two while looking at Intel's NUC7i7BNH NUC. The table below summarizes the extent of performance loss in various benchmarks.
| Meltdown & Spectre Patches - Impact on the Intel NUC7i7BNH Benchmarks | |
| Benchmark | Performance Notes (Fully Patched vs. Unpatched) |
| BAPCo SYSmark 2014 SE - Overall | -5.47% |
| BAPCo SYSmark 2014 SE - Office | -5.17% |
| BAPCo SYSmark 2014 SE - Media | -4.11% |
| BAPCo SYSmark 2014 SE - Data & Financial Analysis | -2.05% |
| BAPCo SYSmark 2014 SE - Responsiveness | -10.48% |
| Futuremark PCMark 10 Extended | -2.31% |
| Futuremark PCMark 10 Essentials | -6.56% |
| Futuremark PCMark 10 Productivity | -8.03% |
| Futuremark PCMark 10 Gaming | +5.56% |
| Futuremark PCMark 10 Digital Content Creation | -0.33% |
| Futuremark PCMark 8 - Home | -1.9% |
| Futuremark PCMark 8 - Creative | -2.32% |
| Futuremark PCMark 8 - Work | -0.83% |
| Futuremark PCMark 8 - Storage | -1.34% |
| Futuremark PCMark 8 - Storage Bandwidth | -29.15% |
| Futuremark PCMark 7 - PCMark Suite Score | -4.03% |
| Futuremark 3DMark 11- Entry Preset | +2.44% |
| Futuremark 3DMark 13 - Cloud Gate | +1.14% |
| Futuremark 3DMark 13 - Ice Storm | -13.73% |
| Agisoft Photoscan - Stage 1 | -2.09% |
| Agisoft Photoscan - Stage 2 | -12.82% |
| Agisoft Photoscan - Stage 3 | -6.70% |
| Agisoft Photoscan - Stage 4 | -2.84% |
| Agisoft Photoscan - Stage 1 (with GPU) | +1.1% |
| Agisoft Photoscan - Stage 2 (with GPU) | +1.46% |
| Cinebench R15 - Single Threaded | +3.58% |
| Cinebench R15 - Multi-Threaded | -0.32% |
| Cinebench R15 - Open GL | +3.78% |
| x264 v5.0 - Pass I | -1.1% |
| x264 v5.0 - Pass II | -0.75% |
| 7z - Compression | -0.16% |
| 7z - Decompression | -0.38% |
Looking at the NUC – and really this should be on the mark for most SSD-equipped Haswell+ systems – there isn't a significant universal trend. The standard for system tests such as these is +/- 3% performance variability, which covers a good chunk of the sub-benchmarks. What's left then are more meaningful performance impacts in select workloads of the BAPCo SYSmark 2014 SE and Futuremark PCMark 10 benchmarks, particularly storage-centric benchmarks. Other than those, we see certain compute workloads (such as the 2nd stage of the Agisoft Photoscan benchmark) experience a loss in performance of more than 10%.
On the whole, we see that the patches for Meltdown and Spectre affect real-world application benchmarks, but, synthetic ones are largely unaffected. The common factor among most of these benchmarks in turn is storage and I/O; the greater the number of operations, the more likely a program will feel the impact of the patches. Conversely, a compute-intensive workload that does little in the way of I/O is more or less unfazed by the changes. Though there is a certain irony to the fact that taken to its logical conclusion, patching a CPU instead renders storage performance slower, with the most impacted systems having the fastest storage.
As for what this means for future system reviews, the studies done as part of this article give us a way forward without completely invalidating all the benchmarks that we have processed in the last few years. While we can't reevaluate every last system – and so old data will need to stick around for a while longer still – these results mean that the data from unimpacted benchmarks is still valid and relevant even after the release of the Meltdown and Spectre patches. To be sure, we will be marking these results with an asterisk to denote this, but ultimately this will allow us to continue comparing new systems to older systems in at least a subset of our traditional benchmarks. Which combined with back-filling benchmarks for those older systems that we do have, lets us retain a good degree of review and benchmark continuity going forward.
Facebook
Twitter
RSS
83 Comments
View All Comments
iter - Monday, March 26, 2018 - link
According to whom? You, the workstation all-seer? Or perhaps some statistics done over the internet?iter - Monday, March 26, 2018 - link
Also, if a "standalone system" is for you the opposite of "connected to the internet" that is quite indicative... You know there exists this thing called a network, on top of which the internet runs. You can have a load of workstations and servers in a network that is not connected to the outside world.Most places that do important work do it this way. Eliminates 99.99% of threats from the outside and the from the inside. Just one of many other common sense things, such as disabled usb storage devices, unauthorized network clients and whatnot. Machines that do connect to the internet are physically isolated from the secure network. They use secure proprietary interfaces for explicit data transfer between the two networks under tight scrutiny.
rhoades-brown - Tuesday, March 27, 2018 - link
Eh? So, your saying that you would put your workstations unpatched and completely unprotected on a network where other devices can connect to it?Did you hear about WannaCrypt? Your network connected workstation would have been easy prey.
Would you allow these unprotected workstations to share files with other workstations and what about the cheaper machines? I assume that you are either creating or processing content/data of some description. Have a look at MS16-120 - 'The most serious of these vulnerabilities could allow remote code execution if a user either visits a specially crafted website or opens a specially crafted document.'
What about USB sticks? Something on one machine could easily be spread to another, and people are stupid enough to plug in a USB stick that they found in a car park, etc.
There are exceptions- air-gaped networks to make things highly secure, but that seems unlikely, and if your workstations are in that rare scenario, have a look at xLED which uses a compromised switch to flash it's status LEDs to share data- crazy, I know; scary, absolutely.
Gasaraki88 - Monday, March 26, 2018 - link
Wow, that's a big exaggeration...Bulat Ziganshin - Friday, March 23, 2018 - link
>Though there is a certain irony to the fact that taken to its logical conclusion, patching a CPU instead renders storage performance slower, with the most impacted systems having the fastest storage.It looks ironic because it was incorrectly attributed as CPU bug. But the point is that it allows to discover information when OS allows it, and thus it's an OS bug of not preventing it. As far as you run pure CPU computations, it doesn't need any mitigations.
The only thing that need to be patched is communication between OS and application, and therefore you got larger hit when these communications are more intensive - on higher-IOPS operations. So f.e. I/O in large blocks (1 MB or so) is unaffected, but 4K I/O is affected, especially with higher-performance drives and higher QD scenarios.
jordanclock - Friday, March 23, 2018 - link
It is a CPU bug. The speculative execution is faulty and that is a CPU feature. The OS patches are simply workarounds to prevent certain kinds of speculative execution.Reflex - Friday, March 23, 2018 - link
It is not a bug at all at either level. It is a feature that was found to be able to be abused. That happens all the time. Once found, it was mitigated, in this case by disabling the feature (Meltdown) or mitigating the impact (Spectre). In future designs it will be mitigated or eliminated.There are all sorts of features your CPU is capable of utilizing that can compromise your data or stability (hey, you can still run in unprotected mode for memory!), when it is found to be a problem it is typically disabled at the appropriate level (microcode/firmware/OS).
bji - Friday, March 23, 2018 - link
Uh, no. It's a feature that comes with an unintended side effect of allowing data reads that should be disallowed. That part of it is a bug, plain and simple. I guess you are the kind of person that would call a bug that crashes the computer a "feature" because "it saves you power when your PC is off because it crashed".PixyMisa - Friday, March 23, 2018 - link
So it's a bug.yeeeeman - Saturday, March 24, 2018 - link
Bug is something that doesn't work as designed. I am pretty sure that they designed and verified it this way. These vulnerabilities are not bugs, they are just security loopholes.