The VMmark Scoring Chaos

When the new Xeon based on the "Nehalem" architecture launched, there was a lot - and I am being polite - of confusion about the VMmark scores. Scores ranged from 14.22 to 23.55 for dual CPU servers based on the same Xeon X5570 2.93GHz. Look at the published and "non published" VMmark scores we found from various sources:

VMware VMmark (ESX 3.5 update 3 if no remark)

The new Xeon 5570 is between 55% and 157% faster, depending on your source. Let's try to make sense out of this. First, take the lowest score, 14.22.

Intel's own benchmarking department was very courageous to publish this score of 14.22. Intel allowed us to talk to Tom Adelmeyer and Doshi A. Kshitij, both experienced engineers. Tom is one of the creators of vConsolidate and Doshi is a principal engineer who wrote some excellent academic papers; his specialty is hardware virtualization. The answer I got was surprising: the 14.22 score was a simple "out-of-the box" VMmark run that did not make use of Hardware Assisted Paging (or EPT if you like) running on top of an ESX 3.5 update 4 beta. Remember only the full ESX 3.5 update 4 contains full support for Intel's EPT technology. Update: My mistake. ESX 3.5 update 4 has full support for Intel's X55xx CPU, but not for EPT technology. That is only available in ESX 4.0 and later.  My thanks goes to Scott Drummonds (VMware) to point this out.

So the 14.22 score is not comparable to the 11.28 score of the AMD Opteron "Shanghai" as the latter is a fully optimized result. Intel's Xeon X5570 can obviously do better, but how much better? A score obtained in February in the same Intel labs, which was never published, was 17.9. It is in the right ballpark for a VMmark run that is clearly better optimized and most likely running with EPT enabled. Update: We believe this was run on early version of ESX 4.0.

On Intel's own site you can find this PDF, which in the very small print mentions a score of 19.51, obtained somewhere in march on a VMware ESX Build 140815 with DDR3-800 (Update: This seems to be a sort of "Release Candidate" of ESX 4.0). As running 13 tiles requires a lot of memory, Intel outfitted the Nehalem server with 18x4GB DDR3 DIMMs. Since this means there are three DIMMs per channel, the clock speed of the DDR3 is throttled back to 800MHz.

At the launch date of the Xeon X5570, scores above 23 were reached with VMware builds 148592, 148783, and 150817. Since the launch of VMware's vSphere 4.0, these numbers have been replaced by ESX 4.0. One of the reasons that these configurations obtain higher scores is the fact they run with two DIMMs per channel, so the DDR3 DIMMs run at 1066MHz. That is good for a boost of 5-6%, which has been confirmed by both Intel's and AMD's VMmark experts. The second and most important reason is that ESX 4.0 is used. As VMware states in some whitepapers, CPU scheduling has improved in the new ESX 4.0, especially for the Nehalem architecture with its SMT (Hyper-Threading).

Intel's server benchmarking department did a favor to the IT community by releasing the 14.22 "out-of-the-box, unoptimized, and no EPT" score. This gave us a realistic worst-case score. The benchmark showed that the newest Nehalem Xeon could outperform its competitor in even the worst conditions, adding credibility to Intel's claims. In contrast, the benchmark claims at Intel's product page are pretty shady.

A 161% performance boost over the previous generation is called an "exceptional gain", but the claim is completely overshadowed by the flawed comparison. It is simply a bad practice to compare a score obtained on - at that time - unreleased brand-new software (VMware ESX build 148592, similar to ESX 4.0) with a benchmark run on older but available software (ESX 3.5 Update 3). While there is little doubt that any server based on the Xeon X55xx is a superior consolidation platform, it is unfortunate that Intel did not inform its customers properly. Especially if you consider that a fair comparison with the Xeon 54xx and Xeon 55xx both on ESX 4.0 would probably also deliver "exceptional gains".

Understanding the VMmark Score ESX 4.0: Nehalem Enhanced
POST A COMMENT

23 Comments

View All Comments

  • ktwebb - Saturday, May 16, 2009 - link

    "How many of us are running more than 50 to 100 VMs, which need on average only 1GB per VM?"

    600+ VM's. Less than 10% run with more than a gig of memory. 90% of the VM's runnning windows 2003
    Reply
  • JohanAnandtech - Saturday, May 16, 2009 - link

    On one physical server? that would probably be a new record :-). Reply
  • joekraska - Thursday, May 14, 2009 - link

    I run a virtual data center with over 800 virtual machines in it and a sustained 25+% growth rate. I found it curious that one poster wrote that "no one pays attention to this." We do pay attention to it, it's the first (and only) benchmark that I look at in deciding things on our side, however not for the reason that might first appear. To wit, we do not use it to decide between vendors: we already have the 'best' vendor, and in truth given two different vendor systems based on the same underlying intel equipment, the results are insufficiently different to compel a vendor-switch, generally speaking. Besides that, we're homogenous. No, that's not why we're interested at all.

    However, we /are/ interested in using VMMark data to help us decide which of our current vendors many platforms we will switch to, when we switch, as well as slightly influencing WHEN our recap occurs. You have to keep in mind that VMWare licenses are quite costly. And the maintenance is costly! There is a cold calculus to ongoing consolidation involving server costs, memory footprint (e.g., can the new server hold many more dimms than the last), and the price one pays for vmware. VMMark data and the ratio of memory per CPU in a proposed new server type both play a key deciding role here.

    C//

    Reply
  • duploxxx - Sunday, May 10, 2009 - link

    I'll give some ideas on reality and this is based several months perf testing and many installations.

    lets say we use 4 types of VM's: dbs + file server + compression server + iis server.

    If I run these Vm's in a 2cpu 4GB config at 60-70% of load.
    All these combined on a 2s quad machine we run 3VM if we would use 4vm we see a perf drop of approx 10% in total netw and i/o.

    that is reality, quite a difference then these benchmarking scores. but the vmmark still gives a good idea about cpu performance. For example compare barcelona against harpert and the harpert will require about 10-15% more clock for equal perf, against shanghai this raised even to 20-25% as you would clearly see in vmmark. The 3,3harper was not able to perform like a 2,7 shangh.
    Now with nehalem the stakes are changed, I didn't check a full build yet but from what i've seen it will be 10-15% more shanghai speed required against nehal. HT ain't doing nothing here when you have some heavy Vm's, when you run 15-20/1 server farm you might have some added value here.

    The solution for AMD will be Istanbul, although not much faster clock/clock then shangh it will add more real cores, with current cache system and ht link the istanbul will scale good enough. For high load VM this will be a much better choice then a HT core. As long as VM off course stays in the same price/socket system :)

    That will be for sure my choice of building rig.
    Reply
  • has407 - Saturday, May 09, 2009 - link

    "The easiest way to see that VMmark is showing its age is in the consolidation ratio of the VMmark runs. Dual CPU machines are consolidating 8 to 17 tiles. That means a dual CPU system is running 102 virtual machines, of which 85 are actively stressed! How many dual CPU machines have you seen that even operate half that many virtual machines?"

    True. OTOH, the more VM's, the greater the stress on the hypervisor, which is essentially a measure of how well the hypervisor multi-tasks (or more accurately "multi-VM's"). While obviously not indicative of real-world workloads, as a synthetic measure it would seem a reasonable first-order indication of the hypervisor's efficiency dealing with disparate VM's.

    That said, I'd agree that there is no substitute for measuring and analyzing real-world workloads. Anyone who depends on a VMark score for determining their virtualization strategy or how many VM's they can shove into a box is in for a rude surprise. Not to mention that VMware has quite a few papers on specific apps that provide good clues and a good basis for analysis independent of VMark.

    Then again, I don't know of anyone with more than a room temperature IQ that depends on VMark or uses it for anything but bragging rights. The typical and appropriate steps are to (1) look at the current workload; (2) do an estimate of what that will look like when virtualized; (2) try it with a subset of the worload; (3) correct estimates and calculations as needed; (4) move forward.

    In short, VMark may be flawed, but the flaw has less to do with VMark and more with the way people use it. VMark scores provide one potentially useful data point--but only one data point.
    Reply
  • JohanAnandtech - Saturday, May 09, 2009 - link

    "the greater the stress on the hypervisor, which is essentially a measure of how well the hypervisor multi-tasks (or more accurately "multi-VM's").

    Excellent comment... VMmark is indeed overemphasizing relatively simple world switches.

    "While obviously not indicative of real-world workloads, as a synthetic measure it would seem a reasonable first-order indication of the hypervisor's efficiency dealing with disparate VM's."

    Indeed. But IMHO, VMmark spends too much time in the scheduler, which makes the percentage it's spends in the exception handling (Hardware virtualization works with exceptions: Memory management, Interrupts etc.) uncharacteristically low. Further analysis needed of course.
    Reply
  • has407 - Saturday, May 09, 2009 - link

    Just as with any OS, the effectiveness of the scheduler is a function of how fairly and efficiently it manages the available resources. For a CPU-intensive workload it's generally pretty simple. It's when you start throwing in IO (or QoS) that really tests the mettle of a scheduler.

    The reason why those VMark reports tend to show such a ludicrously high number of tiles is because that is typically the number of tiles needed to saturate the CPU. Which is another way of saying that either: (1) the workload is too heavily IO-bound; or (2) the IO subsystem is too slow relative to the workload.

    Which is the reason the VMark tests typically ramp up the number of tiles until CPU saturation, otherwise you're testing--and constrained by--the IO subsystem's performance, (which is typically the limiting factor), and not the hypervisor's performance. Obviously the hypervisor's IO efficiency is a factor, but if you've still got 20% CPU idle, you can push it further.

    That's not to say VMark is "correct"; at the risk of oversimplification...

    1. Would a smaller number of tiles/VM's produce less time in the hypervisor? Yes, as there are fewer VM's with timers, etc. to be emulated. That could be measured by running an increasing number of idle VM's and measuring the hypervisor cost. E.g., if basic emulation takes 1% of the CPU per idle VM, then 50 VMs should consume 50% of the CPU.

    2. Would a smaller number of VM's incur less scheduling overhead? Maybe. It largely depends on what the VM's are doing:

    a) For a CPU-bound workload, the difference is probably nominal for any but an extreme number of VM's. That could be easily determined by running an increasing number of CPU-intensive VM's and calculating the additional pure scheduling overhead/VM. E.g., if the scheduling and context switch overhead is 1ms/VM, and the scheduling quantum is 100ms, then pure scheduling and context-switch overhead will consume 50% with 50 VM's

    b) For an IO-bound workload, there will be more events likely to invoke the scheduler, and thus increasing scheduler time. Is that increased scheduler time a function of the number of VM's or the number of IO events? Hard to say, but I'd argue the latter. For the VM doing the IO, that's also more likely to lead to that VM stalled waiting on IO (unless of course that VM also has CPU-bound processes), so the more VM's the more likely the hypervisor can find a non-stalled VM and keep the CPU utlized doing useful work.

    3. Would a smaller number of VM's that produced an aggregate IO load (IO events and thus scheduling events) equivalent to a bunch of VMark tiles show an appreciable difference? Maybe... too many factors to make blanket statements, such as the type of IO, the speed of the hardware IO subsystem, and the hypervisor's efficiency in performing specific types of IO. Which of course is why people really should measure their workloads. (VMWare also has a number of micro-benchmarks available that provide clues independent of VMWark.)

    In short, I think that if VMark has a problem, it's less a matter of how many VM's, and more that the aggregate IO load from all those (small) VM's is higher than what a real-world set of (larger) VM's will produce given the the number of VM's that can fit into the same amount of memory. That would mean lower CPU utilization (more likely all VM's are waiting on IO), or it means your IO subsystem is too slow (and EMC is ready and waiting to help you fix that :).
    Reply
  • tviceman - Friday, May 08, 2009 - link

    Synthetic benchmarks are a worthless waste of time and reading. When I purchase a new CPU, GPU, RAM, HD, etc. for improved performance it is based only on real world applications. Reply
  • has407 - Friday, May 08, 2009 - link

    "One of the reasons that these configurations obtain higher scores is the fact they run with two DIMMs per channel, so the DDR3 DIMMs run at 1066MHz. That is good for a boost of 5-6%, which has been confirmed by both Intel's and AMD's VMmark experts. The second and most important reason..."

    You forgot about EPT support that showed up with Nehalem, which can provide nominal to very significant performance improvements.
    Reply
  • JohanAnandtech - Saturday, May 09, 2009 - link

    I mention it elsewhere in the article, but I agree that I should have talked about EPT in that sentence too :-).

    EPT is probably good for about 20%. I would try it out myself, but VMmark requires at least 72 GB on a Nehalem Xeon, and that is out of the reach of our lab right now.
    Reply

Log in

Don't have an account? Sign up now