vApus Mark I: the choices we made

vApus mark I uses only Windows Guest OS VMs, but we are also preparing a mixed Linux and Windows scenario. vApus Mark I uses four VMs with four server applications:

  • The Nieuws.be OLAP database, based on SQL Server 2008 x64 running on Windows 2008 64-bit, stress tested by our in-house developed vApus test.
  • Two MCS eFMS portals running PHP, IIS on Windows 2003 R2, stress tested by our in house developed vApus test.
  • One OLTP database, based on Oracle 10G Calling Circle benchmark of Dominic Giles.

We took great care to make sure that the benchmarks start, run under full load, and stop at the same moment. vApus is capable of breaking off a test when another is finished, or repeating a stress test until the others have finished.


The OLAP VM is based on the Microsoft SQL Server database of the Flemish/Dutch Nieuws.be site, one of the newest web 2.0 websites launched in 2008. Nieuws.be uses a 64-bit SQL Server 2008 x64 database on top of Windows 2008 Enterprise RTM (64-bit). It is a typical OLAP database, with more than 100GB of data consisting of a few hundred separate tables. 99% of the load on the database consists of selects, and about 5% of these are stored procedures. Network traffic is 6.5MB/s average and 14MB/s peak, so our Gigabit connection still has a lot of headroom. DQL (Disk Queue Length) is at 2.0 in the first round of tests, but we only record the results of the subsequent rounds where the database is in a steady state. We measured a DQL close to 0 during these tests, so there is no tangible impact from the storage system. The database is warmed up with 50 to 150 users. The results are recorded while 250 to 700 users hit the database.

The MCS eFMS portal, a real-world facility management web application, has been discussed in detail here. It is a complex IIS, PHP, and FastCGI site running on top of Windows 2003 R2 32-bit. Note that these two VMs run in a 32-bit guest OS, which impacts the VM monitor mode.

Since OLTP testing with our own flexible stress testing software is still in beta, our fourth VM uses a freely available test: "Calling Circle" of the Oracle Swingbench Suite. Swingbench is a free load generator designed by Dominic Giles to stress test an Oracle database. We tested the same way as we have tested before, with one difference: we use an OLTP database that is only 2.7GB large (instead of 9.5GB). We used a 9.5GB database to make sure that locking contention didn't kill scaling on systems with up to 16 logical CPUs. In this case, 2.7GB is enough as we deploy the database on a 4 vCPU VM. Keeping the database relatively small allows us to shrink the SGA size (Oracle buffer in RAM) to 3GB (normally it's 10GB) and the PGA size to 350MB (normally it's 1.6GB). Shrinking the database ensures that our VM is content with 4GB of RAM. Remember that we want to keep the amount of memory needed low so we can perform these tests without needing the most expensive RAM modules on the market. A calling circle test consists of 83% selects, 7% inserts, and 10% updates. The OLTP test runs on the Oracle 10g Release 2 (10.2) 64-bit on top of Windows 2008 Enterprise RTM (64-bit).

Below is a small table that gives you the "native" characteristics that matter for virtualization in each test. (Page management is still being researched.) With "native" we mean the characteristics measured running on the native OS (Windows 2003 and 2008 server) with perfmon.

Native Performance Characteristics
Native Application / VM Kernel Time Typical CPU Load Interrupt/s Network Disk I/O DQL
Nieuws.be / VM1 0.65% 90-100% 3000 1.6MB/s 0.9MB/s 0.07
MCS eFMS / VM2 & 3 8% 50-100% 4000 3MB/s 0.01MB/s 0
Oracle Calling Circle / VM4 17% 95-100% 11900 1.6MB/s 3.2MB/s 0.07

Our OLAP database ("Nieuws.be") is clearly mostly CPU intensive and performs very little I/O besides a bit of network traffic. In contrast, the OLTP test causes an avalanche of interrupts. How much time an application spends in the native kernel gives a first rough indication of how much the hypervisor will have to work. It is not the only determining factor, as we have noticed that a lot of page activity is going on in the MCS eFMS application, which causes it to be even more "hypervisor intensive" than the OLTP VM. From the data we gathered, we suspect that the Nieuws.be VM will be mostly stressing the hypervisor by demanding "time slices" as the VM can absorb all the CPU power it gets. The same is true for the fourth "OLTP VM", but this one will also cause a lot of extra "world switches" (from the VM to hypervisor and back) due to the number of interrupts.

The two web portal VMs, which sometimes do not demand all available CPU power (4 cores per VM, 8 cores in total), will allow the hypervisor to make room for the other two VMs. However, the web portal (MCS eFMS) will give the hypervisor a lot of work if Hardware Assisted Paging (RVI, NPT, EPT) is not available. If EPT or RVI is available, the TLBs (Translation Lookaside Buffer) of the CPUs will be stressed quite a bit, and TLB misses will be costly.

As the SGA buffer is larger than the database, very little disk activity is measured. It helps of course that the storage system consist of two extremely fast X25-E SSDs. We only measure performance when all VMs are in a "steady" state; there is a warm up time of about 20 minutes before we actually start recording measurements.

Independent Real-World Virtualization Benchmarking vApus: Virtual Stress Testing
Comments Locked

66 Comments

View All Comments

  • has407 - Sunday, May 24, 2009 - link

    Thanks very much for the additional data points, and especially for providing details. Still digesting your data (thanks again!), but a few thoughts...

    1. At the risk of being pedantic... Both VMark and vAplus scores are dimensionless. It would be better to avoid terms such as "faster" to describe them; IMHO, that has lead to distraction (or *cough* in some cases *cough* irrationality). Is a car that can move 5 people at 160KPH "faster" than a bus that can move 20 people at 80KPH? Maybe sticking to terms such as "throughput" or simply "performance" would be better.

    2. While the geometric mean provides a nice single score, I hope you will continue to publish the detailed numbers that contribute to it (as done with VMark disclosures). The individual scores provide important clues as to whether a closer look is warranted, whether of the workload mix, the CPU, or the hypervisor.

    For example, the sum of the workload (or arithmetic mean * 4) provides total overall throughput, which is an important indicator; in an ideal world that should match the geometric mean. A significant difference between those suggests a closer look is warranted.

    E.g., Unless you have a workload mix that can soak up extra CPU cycles, the Xeon 5080 and to a lesser extent the Opteron 2222 don't look like good choices. For the 5080, the CPU-intensive OLAP VM contributes 60% to the result, whereas the others tend to be ~40-45%, and the difference between the geometric and arithmetic mean for the 5080 is 19%, whereas for the rest it's <14%.

    3. Note what happens if you pull the CPU-intensive OLAP VM out of the picture. While I can't empirically test that, and I'm using a bit of a sledgehammer here... Eliminate it from the scoring and see what happens: the difference between the geometric and arithmetic mean drops to ~1% across the board.

    Moreover, the ratio of the scores with and without the OLAP VM is quite constant, with a correlation > 0.999. The outliers again, but not by all that much, being the Xeon 5080 and the Opteron 2222, and to a lesser extent the Xeon L5350.

    4. In short, I'm not sure what the addition of a CPU-intensive VM such as OLAP is adding to the picture, other than soaking up CPU cycles and some memory. A CPU-intensive VM is the easiest (or should be the easiest) for a hypervisor to handle, and appears to tell us little more than what idle time figures would tell us. In the case of the Xeon 5080 and Opteron 2222, it also appears to inflate their overall score (whether due to the processor or hypervisor, or more likely a combination of the two, is unclear).

    5. That said, maybe it would be good to include a CPU-intensive VM in the mix, if for no other reason than to highlight those systems or hypervisors where that VM scores higher or lower than expected (e.g., the Xeon 5080 and Opteron 2222). However, I'd bet you can achieve the same result with a lot less work using a simpler synthetic CPU/memory-intensive test in the VM.

    OTOH, maybe artificially driving CPU utilization towards 100% with such CPU-intensive VM's doesn't really tell us much more than we'd know without them--as IMHO my admittedly crude analysis suggests--and that vAplus might be a better indicator for those looking for clues as to appropriate workload allocation among virtualized systems, rather than those looking for a single magic number to quantify performance.
  • JohanAnandtech - Tuesday, May 26, 2009 - link

    "However, I'd bet you can achieve the same result with a lot less work using a simpler synthetic CPU/memory-intensive test in the VM. "

    That would eliminate the network traffic. While the "native" running database is not making the OS kernel sweat, the hypervisor does get some work from the network, and thus this VM influences the scores of the other VMs. It is not a gigantic effect but it is there. And remember, we want to keep control of what happens in our VMs. Once you start running synthetic benches, you have no idea what kind of instructions are run. SQL server is closed source too, but at least we know that the instructions which will be send to the CPU will be the same as in the real world.

    We will of course continue to publish all the different scores so that our inquisitive readers can make their minds :-). Nothing worse than people who quickly gloss over the graphs and than start ranting ;-).

    Thanks for the elaborate comment, although I am still not sure why you would remove the OLAP database. The fact that the 4 core machines (Dempsey, dualcore opteron) do not have a lot of cycles left for the other VMs illustrates what happens in an oversubscribed system where one VM demands a lot of CPU power.
  • has407 - Wednesday, May 27, 2009 - link

    Johan -- My thought was not so much whether to get rid of the OLAP VM, than whether a simpler CPU-intensive VM would suffice, synthetic or otherwise. However, that's probably an academic question at this point, as you've already got it the mix. (And a question I probably spent too much time thinking-out-loud about in my post. :)

    The other arguably more important questions are whether including CPU-intensive VM's (OLAP or synthetic) in order to drive CPU utilization to 100% easier--especially as it is 25% of the workload--provides significant additional information, and whether is more representative than the VMark approach.

    That's a much harder question to answer, and far more difficult to model. Real-world benchmarks may be desirable and necessary, but they are not sufficient; a representative and real-world workload mix is also needed. What constitutes a "representative and real-world" mix is of course the Big Question.

    I'll spare everyone more thinking-out-loud on that subject :), other than to say that benchmarks should help us understand how to characterize and model to more accurately predict performance. Without that we end up with lots of data (snapshots of workload X on hardware Y), but little better formal or rigorous understanding as to why. (One area where synthetic- or micro-benchmarks can help provide insight, as much as they might be derided. And one reason IMHO why what passes for most benchmarking today contributes more noise than signal. But that's another subject.)

    In any case, it's good to have vApus to provide additional data points and as a counterpoint to VMark. Thanks again. Looking forward to the next round of data.
  • has407 - Monday, May 25, 2009 - link

    Sorry, fourth column in table labeled "B:GM" (duplicate of third column label) should be "B:AM".
  • has407 - Monday, May 25, 2009 - link

    p.s. here's the numbers on which that post was based, calculated using your raw data...

    A - With OLAP VM
    B - Without OLAP VM
    GM - Geometric mean
    AM - Arithmetic mean

    - A:GM -- geometric mean of all four VM's * 4
    - A:AM -- arithmetic mean of all four VM's * 4 (or the sum of the individual scores).
    - B:GM -- geometric mean of the three VM's excluding the OLAP VM * 3.
    - B:AM -- arithmetic mean of the three VM's excluding OLAP * 3 (or the sum of the individual scores excluding the OLAP VM).

    A:GM A:AM B:GM B:GM A:GM/B:GM
    2.03 2.14 1.28 1.29 1.58 Dual Opteron 8389 2.9
    2.45 2.54 1.60 1.60 1.54 Dual Xeon X5570 2.93
    2.08 2.21 1.29 1.29 1.61 Dual Xeon X5570 2.93 HT off
    1.87 1.99 1.16 1.17 1.61 Dual Xeon E5450 3.0
    1.68 1.81 1.02 1.02 1.65 Dual Xeon X5365 3.0
    1.12 1.22 0.68 0.68 1.66 Dual Xeon L5350 1.86
    0.59 0.78 0.30 0.31 1.96 Dual Xeon 5080 3.73
    0.82 0.96 0.45 0.46 1.80 Dual Opteron 2222 3.0

    Correlation( A:GM, B:GM ): 0.9993

    Hope that helps explains my conclusions.
  • solori - Friday, May 22, 2009 - link

    I'm glad to see Johan's team has gone beyond the "closed" VMmark standard with a Windows-based benchmark and I hope this leads to more sanity-checking of results down the line. However, the first step is verifying the process before you get to the results. Here's an example of where you're leaving some issues dangling:

    "However, the web portal (MCS eFMS) will give the hypervisor a lot of work if Hardware Assisted Paging (RVI, NPT, EPT) is not available. If EPT or RVI is available, the TLBs (Translation Lookaside Buffer) of the CPUs will be stressed quite a bit, and TLB misses will be costly."

    This implies RVI is defaulted for 32-bit VM's. VMware's default for 32-bit virtual machines is BT (binary-translation) and not RVI, even though VROOM! tests show a clear advantage for RVI over BT for most 32-bit workloads. While you effectively discuss the affects of disabling RVI in the 64-bit case, you're unclear about "forcing" RVI in the 32-bit case. Are you saying that AMD-v and RVI are enabled for the 32-bit workloads by default? VMware's guidance states otherwise:

    "By default, ESX automatically runs 32bit VMs (Mail, File, and Standby) with BT, and runs 64bit VMS (Database, Web, and Java) with AMD-V + RVI."

    - VROOM! Blog, http://blogs.vmware.com/performance/2009/03/perfor...">http://blogs.vmware.com/performance/200...uation-o...

    This guidance is echoed in the latest VI3.5 Performance Guide Release:

    "RVI is supported beginning with ESX 3.5 Update 1. By default, on AMD processors that support it ESX Update 1 uses RVI for virtual machines running 64-bit guest operating systems and does not use RVI for virtual machines running 32-bit guest operating systems.

    Although RVI is disabled by default for virtual machines running 32-bit guest operating systems, enabling it for certain 32-bit operating systems may achieve performance benefits similar to those achieved for 64-bit operating systems. These 32-bit operating systems include Windows 2003 SP2, Windows Vista, and Linux.

    When RVI is enabled for a virtual machine we recommend you also?when possible?configure that virtual machine?s guest operating system and applications to make use of large memory pages."

    - Performance Best Practices and Benchmarking Guidelines, VMware, Inc. (page 18)

    Your chart on page 9 further indicates "SVM + RVI" for 32-bit hosts, but there is no mention of steps you took to enable RVI. This process is best described by the Best Practices Guide:

    "If desired, however, this can be changed using the VI Client by selecting the virtual machine to be configured, clicking Edit virtual machine settings, choosing the Options tab, selecting Virtualized MMU, and selecting the desired radio button. Force use of these features where available enables RVI, Forbid use of these features disables RVI, and Allow the host to determine automatically results in the default behavior described above."

    - Performance Best Practices and Benchmarking Guidelines, VMware, Inc. (page 18)

    So, which is it: 32-bit without RVI or undocumented changes to the VMM according to VMware guidance? If it is the former, the conclusions are misleading (as stated); if the latter, such modifications should be stated explicitly since they do not represent the "typical" or "default" configuration for 32-bit guests. This oversight does not invalidate the results of the test by any means, it simply makes them more difficult to interpret.

    That said, a good effort! You may as well contrast 32-bit w & w/o RVI - those results might be interesting too. I know you guys probably worked VERY hard to get these results out, and I'd like to see more, despite what "tshen83" thinks :-)

    Collin C. MacMillan -- http://solori.wordpress.com">http://solori.wordpress.com
  • JohanAnandtech - Sunday, May 24, 2009 - link

    Hi Collin,

    I was under the impression that ESX now choses RVI+SVM automatically, but that might have been ESX 4.0. I am going to check again on monday, but I am 99.9% sure we have enabled RVI in most tests (unless indicated otherwise) as it is a best performance practice for the Opterons.
  • alpha754293 - Friday, May 22, 2009 - link

    Another excellent, thorough, well researched article.

    Thanks! :o)
  • JohanAnandtech - Sunday, May 24, 2009 - link

    You are most welcome. Thx for letting us know!
  • knutjb - Monday, May 25, 2009 - link

    Thanks for presenting another point of view. When I read the original article showing the new Xeons so far ahead, I was skeptical. Rarely does a company produce a product that is such a huge leap, not only over their competitors, over their own products too. When there is only one primary benchmark the results can be skewed. Also, the wide variety of software combinations is eyepopping so it is very time consuming to create a resonable balance using real databaeses for a different, but valid benchmark.

    Thanks for the hard work, I look forward to reading more on this subject.

Log in

Don't have an account? Sign up now