Testing Spectre and Meltdown: SYSMark

As we were performing this testing, the issue of Spectre and Meltdown reared its ugly head. After 40 hours of testing, we realised that the motherboard was not BIOS patched for the latest issues, and we reached out to get the latest update, and had to retest all over again.

It was around this time that Intel also reached out to us to give us the results of their own performance testing relating to the patches. The long and short of the discussions about Intel results were that the patches affected systems with older the most, and systems that had fast storage (SSD vs HDD) also took the brunt of the performance hit.

For our testing, we took the SYSMark benchmark and did a before and after comparison. We confirmed the patches were applied by using the Inspectre tool before running in patched mode. You can read our analysis of the Spectre and Meltdown issues in the following articles:

SYSMark 2014 SE

For the overall score, every processor lost some performance:

SYSMark 2014 SE: Overall Score

The biggest overall loser in real terms was the W-2155, which mixes single core performance with many threads. This is interesting - the processor with the most threads, the W-2195, did not have such a percentage dip. This might be related to how each of these processors is laid out differently: the W-2195 uses Intel's HCC 18-core die, whereas the W-2155 uses the LCC 10-core die. The HCC die has extra core-to-core latency because of the larger floorplan, which might hide some of the deficiencies here.

If we compare the percentage decrease across all of the SYSMark sub-tests:

We can see that the biggest decreases are seen in the Response sub-test, which contributes a lot to the overall score decreases. The response sub-test uses a fair amount of storage, which we know is likely to be the biggest loser from the patches. However, our overall decreases in performance range from 2.0% on the small slow core to 5.6% on the 10-core and back down to 3.5% on the largest 18-core part. The hardest hit tests were down 12%.

Benchmarking Performance: CPU Legacy Tests Conclusion: Is Intel Serious About Xeon W?
Comments Locked

74 Comments

View All Comments

  • Ian Cutress - Monday, July 30, 2018 - link

    If we ever get a Xeon Scalable system up and running for proper benchmarking, I'll run through some tests.
  • JoJ - Saturday, August 4, 2018 - link

    Hi Ian,

    I beg you to consider the futility of approaching HPE, to obtain a Z station and 56 Scalable cores, max RAM etc, may not may be wholly as anticipated.

    I have a very high profile media production job coming, that could warrant a room of fully loaded such machines. Truly high profile. I have been always able to establish nearly immediate rapport with the marketing folk who are not apparently thought about by the UK online tech media, as warmly as I think it fair, but then I deal in printed worlds of very controlled readerships, and possibly by reflex alone I'm treated like I have more to say than I often do. But approaching the twenty five years mark, I probably ran out of lines some nineteen years ago anyway...

    I care about hurling around heavily laden InDesign files, often without any media placeholders to include 409MB Hassleblad captures straight from the photographer. This project is going to be filled with custom adverts not run in any other media ever. (It took longer to agree copy for Rolls Royce than to sell the deal, in our first encounter with this project. VWAG owned both marques including Bentley, then.) Advertisers may visit, our job to produce for them their first display copy ever, because of interest from services who are in the word of mouth referral ecosystem this publication represents. We are effectively building a advertising studio, to accommodate the needs presented by, eg a long established security company, beyond public profile not by necessity or plan but rather not seeing the circle squared to deliver satisfactory result, without establishing unwanted capacity or staffing. So founder directors shall descend, for interview and portrait, the deal being approved by them signing a proof of the pages, in person, ideally we are able to complete 60% of the interview and layout remotely in advance.

    I want to suggest to you that HPE could genuinely be intrigued by the client base of our project, and want to be exposed to them... mostly boutique banks and funds. A typical user is running a dozen Bloomberg Professional windows, Excel, with live updates and a lot of libraries for giving cell results, will have a terabyte or so of core textual references, between academic papers, email threads, online discussion (usually scraped) , and the job is to fill very specialist magazines with content and advertisers who don't play in the spray and pray brand game, but care everything about how they're found by a prospective new customer. So atop the trappings of a financial trader's desktop, will be layered Word, sure, but a TeX compositor, Visual studio, InDesign, PS...WHY? Because if we can get the message right, we can trade quickly to fill a matching campaign of the same redundancy as big ad buyers get, at even sharper prices. They come for creative, stay for flow execution.

    It's realistic, even commonplace, for a Xeon Scalable workstation to find itself in front of board member powers, in this way.

    Looking at the decisive commitment required to purchase current generation high end workstations, I assume that HPE, Dell et.al. just lean on their direct sales forces.

    but strip away our Corporate veneer, and we're indistinguishable from any of your readers.

    Individual buyers of workstation class computers, commonly inhabit the front line trenches of high end media production. London seethes with freelancer artists, driven by insufficient investment by employers, to fly solo simply to stay current with the tools and turnaround of their art. I can't win any argument about how far from the public imagination of the game, is the typical MB Pro user, sat elegantly in a WeWork chair, I thought my first sight of the Barbican WeWork office, was a film set... (whereas we're actually building a film set where to conduct our FOH business, because I can't vouch for the appearance of our actual digs. (Mr CIO left of his own accord, incapable of tolerating our disinterested views of fancy workspaces... the fact that our dev team is 80% founder level partner, actually never came into play, but the non attendance at his"corporate development" meetings, was protest against the idea of competing for hires with luxuries. Pity, in this one respect alone, the one character aspect we didn't believe necessary to investigate in interview, so patent is our third world architectural infrastructure, relations truly soured only because the guy was so good in every way otherwise. We just couldn't please with such capital commitments, not looking at the impending A grade glut just completing in the City. Such dangerous words; "absolutely, in the future, if you take us there, we will want a statement head office for sales etc". Just not on any immediate plans..

    sorry, Ian, I am avoiding saying that I think I certainly could sell a HPE on the facts we know very well, where they'll be getting sales, in this sector at least, more from physical exposure to buyers and influencers, directly demonstrated the new generation capability by entirely non conveniently situated independent professionals and outside service companies, but absolutely nothing to look at in the kind of performance review I can't really understand is so homogeneous online, because this sort of evaluation hardly relates to the real use at all.

    I am adamant that the entire presentation of high end desktop bixes is hostile and increasingly damaging to sales of performance workstations to corporate office buyers of all kinds.

    Wall to wall gamer imagery has diluted the public perception of utilitarian power of computers beyond homeopathic insignificance.

    Nothing, not a single word for twenty years- it seems to me - has conveyed any part of this crucial message: "With this new Intel computer, and the latest developments in the most advanced quantitative tools for generations, I WHOOP MY COMPETITIONS REAR!'

    By all means, running a renderer matters in the industry that we inhabit. But the reason why we'll shell out for a 56 core beast, is so we can quickly previs a scene, while still working through the latest circulation data dump for the 200,000 or so publications we trade advertising on. Most days, we're rendering visuals of the data we don't want to take our concentration aeay from, again the reason why we're looking at the new HPE Z station Scalable workstations.

    i have to be distracted but I'll conclude with guaranteed brevity upon my short return. There's plenty of angles here to get a vendor to cough up a couple of fully loaded machines for you, and I am entirely serious I'll invest the necessary time to get you a shot if you think I'm not batshii crazy. bfn
  • Elstar - Monday, July 30, 2018 - link

    To be fair, the top end Core X and Xeon W have different TDP values.
  • HStewart - Monday, July 30, 2018 - link

    So does TDP values make bigger impact than actual Frequency - on same chip of course.
  • BurntMyBacon - Tuesday, July 31, 2018 - link

    It is more that TDP and frequency can no longer be separated. If you completely ignore TDP and fix the frequencies to their maximum, then TDP only matters in as much as you have the proper cooling solution to manage it. However, the actual running frequency of Intel's processors are tied into processor temperature, power, number of active cores, etc. On the same chip, a larger TDP means the processor can spend more time at higher frequencies or have more cores active at the same frequency.
  • mode_13h - Monday, July 30, 2018 - link

    The difference relative to X-series is probably due to thermal throttling due to dual AVX-512 and heat buildup under the non-soldered IHS.
  • HStewart - Monday, July 30, 2018 - link

    That would also be interesting test - I believe AVX-512 can be turned off in bios - not sure since I don't have one.
  • JlHADJOE - Tuesday, July 31, 2018 - link

    Lack of Turbo Boost 3.0 maybe?
  • stanleyipkiss - Monday, July 30, 2018 - link

    Besides the VM situation on Threadripper, why would anyone spend $2500+ for this?
  • duploxxx - Monday, July 30, 2018 - link

    what VM situation?

Log in

Don't have an account? Sign up now