vApus Mark II

vApus Mark II is our newest benchmarksuite that tests how well servers cope with virtualizing "heavy duty applications". We explained the benchmark methodology here.

vApus Mark II score - VMware ESX 4.1
* 2 tiles instead of 4 tiles test
** 128GB instead of 64GB

Before we can even start analyzing these numbers, we must elaborate about some benchmark nuances. We had to test several platforms in two different setups to make sure the comparison was as fair as possible. First, let's look at the Xeon 7560.

The Xeon 7560 has two memory controllers, and each controller has two serial memory interfaces. Each SMI connects to two memory buffers, and each buffer needs two DIMMs. Each CPU needs thus eight DIMMs to achieve maximum bandwidth. So our Quad Xeon X7560 needs 32 DIMMs. Now, we also want to do a performance/watt comparison of these servers. So to accomplish this, we decided to test with 16 DIMMs (64GB) in all servers. With 16 channels, bandwidth goes down from 58GB/s to 38GB/s and bandwidth has a tangible impact in a virtualized environment. Therefore, we tested with both 128GB and 64GB. The 128GB number represents the best performance of the quad Xeon 7560; the 64GB number will allow us to determine performance/watt.

Next the dual Opteron and dual Xeon numbers. We tested with both 2- and 4-tile virtualization scenarios. With 2-tiles we demand 36 virtual CPUs, which is more than enough to stress the dual socket servers. As these dual socket servers will be limited by memory space, we feel that the 2-tile numbers are more representative. By comparing the 2-tile numbers with the 4-tile numbers, we take into account that the quad socket systems will be able to leverage their higher number of DIMM slots. So comparing the 2-tile (Dual Socket) with the 4-tile (quad socket) is closest to the real world. However, if you feel that keeping the load the same is more important we added the 4-tile numbers. Four tile numbers result in slightly higher scores for the dual socket systems, and this is similar to how high VMmark scores are achieved. But if you look at the table below, you’ll see that there is another reason why this is not the best way to benchmark:

The four tiles benchmark achieves higher throughput, but the individual tiles perform very badly. If you remember, our reference scores (100%) are based on the quad-core Xeon 5570 2.93. You can see that the 4-tile benchmark runs achieve only 13% (Opteron) or 11% (Xeon) of a quad Xeon 5500 on the Oracle OLTP test. That means the OLTP VM gets less than a 1.5GHz Xeon 5570 (half a Xeon 5570). In the 2-tile test, the OLTP VM gets the performance of a full Xeon 5570 core (in the case of AMD, probably 1.5 Opteron “Istanbul” cores).

In the real world, getting much more throughput at the expense of the response times of individual applications is acceptable for applications such as underutilized file servers and authentication servers (an active directory server might only see a spike at 9 AM). But vApus always had the objective of measuring the performance of virtualized performance critical applications such as important web services, OLAP, and OLTP databases. So since performance matters, we feel that the individual response time of the VMs is more important than pure throughput. For our further performance analysis we will use the 2-tile numbers of the dual Xeon and dual Opteron.

The quad Xeon has a 15% advantage over the quad Magny-cours. In our last article, we noted that the quad Xeon 7560 might make sense even to the people who don’t feel that RAS is their top priority. The reason was that the performance advantage over the dual socket server was compelling enough to consider buying a few quad Xeons instead of 2/3 times more dual Xeons. However, the Dell R815 and the 48 AMD cores inside block the way downwards for the quad Intel platform. The price/performance of the Opteron platform is extremely attractive: you can almost buy two Dell R815 for the price of a quad Xeon server and you get 85% of the performance.

The performance advantage over the Dual Xeon X5670 is almost 80% for a price premium of about 30%. You need about twice as many dual Intel servers, so this is excellent value. Only power can spoil AMD’s value party. We’ll look into this later in this article.

Although the quad Opteron 6136 may not enjoy the same fame as its twelve-core 6174 sibling, it is worth checking out. A Dell R815 equipped with four 6136 Opterons and 128GB costs about $12000. Compared to the dual Xeon 5670 with 128GB, you save about $1000 and get essentially 40% more performance for free. Not bad at all. But won’t that $1000 dissipate in the heat of extra power? Let us find out!

VMmark Power Extremes: Idle and Full Load
Comments Locked

51 Comments

View All Comments

  • Exelius - Thursday, September 9, 2010 - link

    So, the "product differentiators" from HP are because they primarily sell through partner channels. This is the model IBM used for years; and if you buy your ProLiants through an HP partner and not a mere reseller, they will know the product line and can configure the hardware however you want. HP does very well at making their servers extremely modular, but you do have to know how/where to find the part numbers. Often this information is not widely distributed outside of HP's partner sales trainings (which are very good.)

    If you're used to the Dell model of sales, it doesn't make much sense. But because Dell sells directly, their policies for channel partners are stupid (the wholesale price for a channel partner is often higher than the retail price for a direct customer.) But because partners have more pricing and configuration flexibility from HP, the partner can often beat Dell's direct price with HP hardware (which IMO is higher quality than Dell anyway.) Dell doesn't want their partners to compete with their direct sales and HP doesn't want to jeopardize their partner relationships by pushing direct sales too hard.

    There are pros and cons to each approach, and it all depends on how you handle your IT. If most of it is in-house, but you're not quite big enough to have an internal buyer who would take in the HP sales training, Dell makes a lot of sense because, well, it's easy to understand and most HP partners make their money off implementation services, not hardware sales. Dell is willing to work with you a little more. But if you look at companies where IT is not a core competency (regional insurance companies, banks, etc) a lot of them use consultants to do IT projects/maintenance and HP is making an absolute killing in this market.
  • JohanAnandtech - Thursday, September 9, 2010 - link

    insightful and enlightening comment. Still, there is a point where extremely modular increases the complexity and price too much. The result is a slightly higher price (which is still acceptable, but sometimes also small configuration mistakes which cause extra delays. The result is significant higher cost. And this happens regularly as even trained people make mistakes. So my first impression is that HP should lower the complexity a bit.
  • Exelius - Saturday, September 11, 2010 - link

    True; but without a "direct sales" option they have no way to offer flexible configurations without having a different part number for every possible configuration. Most HP partners will simply use a sales quote tool to build server configurations (in fact; this is exactly what Dell does if you order through their sales reps, which is how you get the best prices.) Again though; HP partners are unlikely to give you a sweetheart deal unless you're buying implementation services from them as well. They make 5-10% on the hardware and 80-150% on the labor,

    But I'll tell you now that HP was consistently able to beat Dell on price through channels over the last 5 years. IMO this is because Dell has the same sort of parts system internally; HP cuts costs by not bothering to make sense of it all and just pushing it off onto their resellers. They're pretty much not interested in selling direct to consumers because it's really just a small part of their business.

    What's killed Dell's profits over the last few years has been that the economic troubles have pushed small/midsize companies to outsource their IT. The companies they outsource to are probably HP partners. Thus, when these companies need hardware, it's likely to be HP (used to be IBM as well; but IBM's support is pathetic and their prices are in the stratosphere.) Channel resellers are also used to dealing with complicated product lists (last I checked Symantec's product book had something like 25,000 individual SKUs) so it's probably not going to change. If anything, it's likely to get worse. For all the consumerization in IT, the enterprise side is only getting more complicated. I wouldn't try to spec a server from HP without at least being familiar with their product line and the options it offers.
  • lorribot - Friday, September 10, 2010 - link

    Being a bit weird we buy Dell PCs (and a sprinkling of Macs) and HP servers.

    The Dell PCs are cheap and do the job, initial sales calls are good and they will bust a gut on price but beyond that Dell are pretty much hopeless at support in the UK, with our sales manager changing 3 or 4 times a year and never refering any support calls to some else in the team w have never heard of before.

    Our HP partner, however, is much more stable and they are generally knowledgeable and help configure servers accordingly though for the most part they are actually straight forward until some nutty developer wants 16 disks locally in an 580 g5. And we have a single point of contact for everything.

    You pays your money and you takes your choice.

    Did mention the next day or two (HP) versus 1 to 2 weeks (Dell) delivery options?

    The draw full of 2 GB DDR3 RAM I have from our HP blades is very irritating, I wish HP would supply with out any RAM installed, it is such a waste.
  • AllYourBaseAreBelong2Us - Thursday, September 9, 2010 - link

    Nice article but HP either sells DL380 (Intel) or DL385 (AMD) servers. Please correct all the DL387 references.
  • JohanAnandtech - Thursday, September 9, 2010 - link

    Yes, fixed that one. I always get in trouble with these number codes.
  • Stuka87 - Thursday, September 9, 2010 - link

    Great review, I love it when different platforms are compared to each other. Also happy to see AMD hold their own to the much lauded 7500 series Xeons in a market that I feel AMD is better suited for (VM Servers)

    However, its possible I missed it, but was the price of the SGI system listed anywhere? It would have been nice to see the price of each system as configured side by side.
  • vol7ron - Thursday, September 9, 2010 - link

    Pictures on Page 5-6 look delicious.

    Nice article
  • duploxxx - Thursday, September 9, 2010 - link

    "Comparing the dual with the quad Opteron 6174 power numbers, we notice a relatively high increase in power: 244 Watt. So for each Opteron that we add, we measure 122 W at the wall. This 122 W includes a few Watts of PSU losses, VRM and DIMM wiring losses. So the real power consumed by the processor is probably somewhere between 100 and 110W. Which is much closer to the TDP (115W) than the ACP (80W) of this CPU."

    when the power draw test was done between 2 socket and 4 socket dell 815 did you remain with the same amount of dimms? so you divided the 2 socket amount in the 4 socket?

    On the power draw calculation don't forget that you also have an additional SR5690 to account for which is 18W TDP, electronics etc, so I don't think it will be operating close to TDP but neither to ACP :)

    btw a lot of mistakes with the HP 387G7 which should be 380G7
  • eanazag - Thursday, September 9, 2010 - link

    This is a strong article. Very helpful and most of us basically need to decide which customer we are and what matches our apps and usage requirements.

Log in

Don't have an account? Sign up now