10Gbit Ethernet: Killing Another Bottleneck?
by Johan De Gelas on March 8, 2010 12:00 PM EST- Posted in
- IT Computing
Delving Deeper
Let us take a closer look at the Neterion and Intel 10G chips configuration on VMware’s vSphere/ESX platform. First, we checked what the S2IO driver of Neterion did when ESX was booting.
If you look closely, you can see that eight Rx queues are recognized, but only one Tx queue. Compare this to the Intel ixgbe driver:
Eight Tx and Rx queues are recognized, one for each VM. This is also confirmed when we start up the VMs. Each VM gets its own Rx and Tx queue. The Xframe-E has eight transmit and eight receive paths, but it seems that for some reason the driver is not able to use the full potential of the card on ESX 4.0.
Conclusion
The goal of this short test was to discover the possibilities of 10 Gigabit Ethernet in a virtualized server. If you have suggestion for more real world testing, let us know.
CX4 is still the only affordable option that comes with reasonable power consumption. Our one-year-old dual-port CX4 card consumes only 6.5W; a similar 10GBase-T solution would probably need twice as much. The latest 10GBase-T (4W instead of >10W per port) advancements are very promising, as we might see power efficient 10G cards with CAT-6 UTP cables this year.
The Neterion Xframe-E could not fulfill the promise of near 10Gbit speeds at low CPU utilization, but our test can only give a limited indication. It is rather weird, as the card we tested was announced as one of the first to support NetQueue in ESX 3.5. We can only guess that driver support for ESX 4.0 is not optimal (yet). The Xframe X3100 is Neterion’s most advanced product and the spec sheet emphasizes its VMware NetQueue support. Neterion ships mostly to OEMs, so it is hard to get an idea of the pricing. When you spec your HP, Dell or IBM server for ESX 4.0 virtualization purposes, it is probably a good idea to check if the 10G Ethernet card is not an older Neterion card.
At a price of about $450-$550, the Supermicro AOC-STG-I2 dual-port with the Intel 82598EB chip is a very attractive solution. Typically, a quad-port gigabit Ethernet solution will cost you half as much, but it delivers only half the bandwidth at twice the CPU load in a virtualized environment.
Facebook
Twitter
RSS
49 Comments
View All Comments
radimf - Wednesday, March 10, 2010 - link
HI,thanks for article!
Btw I am reading your site because of your virtualization articles.
I planned almost 3 years ago for IT project with only a 1/5 of complete budget for small virtualization scenario.
If you want redundancy, It can´t get much simplier than that:
- 2 ESX servers
- one SAN + one NFS/iSCSI/potentially FC storage for D2D backup
- 2 TCP switches, 2 FC switches
world moved, IT changed, EU dotation took too long to process - we finished last summer what was planned years ago...
My 2 cents from small company finishing small IT virtualization project?
FC saved my ass.
iSCSI was on my list (DELL gear), but went FC instead(HP) for lower total price (thanks crisis :-)
HP hardware looked sweet on specs sheets, and actual HW is superb, BUT.... FW sucked BIG TIME.
IT took HP half year to fix it.
HP 2910al switches do have option for up to 4 10gbit ports - that was the reason I bought them last summer.
Coupled with DA cables - very cheap solution how to get 10gbit to your small VMware cluster. (viable 100% now)
But unfortunatelly FW (that time) sucked so much, that 3 out of 4 supplied DA cables did not work at all (out of the box).
Thanks to HP - they changed our DA for 10gbit SFP+ SR optics! :-)
After installation we had several issues with "dead ESX cluster".
Not even ping worked!
FC worked flawlessly through these nightmares.
Swithces again...
Spanning tree protocol bug ate our cluster.
Now we are happy finally. Everything works as advertised.
10gbit primary links are backed up by 1gbit stand-by.
Insane backup speeds of whole VMs compared to our legacy SMB solution to nexenta storage appliance.
JohanAnandtech - Monday, March 8, 2010 - link
Thank you. Very nice suggestion especially since we already started to test this out :-). Will have to wait until April though, as we got a lot of server CPU launches this month;Lord 666 - Monday, March 8, 2010 - link
Aren't the new 32nm Intel server platforms coming with standard 10gbe nics? After my SAN project, going to phase in the new 32nm cpu servers mainly for AES-NI. The 10gbe nics would be an added bonus.hescominsoon - Monday, March 8, 2010 - link
It's called xsigo(pronounced zee-go) and solves the i/o issue you are tying to solve here for vm i/o bandwidth.JohanAnandtech - Monday, March 8, 2010 - link
Basically, it seems like using infiniband to connect each server to an infinibandswitch. And that infiniband connection is then used by a software which offers both a virtual HBA and a virtual NIC. Right? Innovative, but starting at $100k, looks expensive to me.vmdude - Monday, March 8, 2010 - link
"Typically, we’ll probably see something like 20 to 50 VMs on such machines."That would be a low vm per core count in my environment. I typically have 40 vms or more running on a 16 core host that is populated with 96 GB of Ram.
ktwebb - Sunday, March 21, 2010 - link
Agreed. With Nahalems it's about a 2 VM's per core ratio in our environment. And that's conservative. At least with vSphere and overcommit capabilities.duploxxx - Monday, March 8, 2010 - link
All depends on design and application type, we typically have 5-6 VM's on a 12 core 32GB machine and about 350 of those, running in a constant 60-70% CPU utilization range.switcher - Thursday, July 29, 2010 - link
Great article and comments.Sorry I'm so late to this thread, but I was curious to know what the vSwitch is doing during the benchmark? How is it configured? @emuslin notes that SR-IOV is more than just VMDq, and AFAIK the Intel 82598EB doesn't support SR-IOV so what we're seeing it the boost from NetQueue. What support for SR-IOV is there in ESX these days?
I'd be nice to see SR-IOV data too.