10G Ethernet: More Than a Big Pipe
by Johan De Gelas on November 24, 2010 2:34 PM EST- Posted in
- IT Computing
- Networking
- 10G Ethernet
Exar's Neterion Solution
SR-IOV will be supported in ESX 5.0 and the successor of Windows Server 2008. Since VMware’s ESX is the dominant hypervisor in most datacenters, that means that a large part of the already virtualized servers will have to wait a year or more before they can get the benefits of SR-IOV.
Exar, the pioneer of multiple devices queues, saw a window of opportunity. Besides the standard SR-IOV and VMware NetQueue support, the X3100 NICs also have a proprietary SR-IOV implementation.
Poprietary solutions only make sense if they offer enough advantages. Neterion claims that the NIC chip has extensive hardware support for network prioritization and quality of service. That hardware support should be superior to hypervisor traffic shaping, especially on the receive side. After all, if bursty traffic causes the NIC to drop packets on the receive side, there is nothing the hypervisor can do: it never saw those packets pass. To underline this, Neterion equips the X3100 with a massive 64MB receive buffer; for comparison, the competitors have in the best case a 512KB receive buffer. This huge receive buffer should ensure that the QoS is guaranteed even if relatively long bursts of network traffic occur.
Neterion NICs can be found in IBM, HP, Dell, Fujitsu, and Hitachi machines. Neterion is part of Exar and has also access to a world distributor channel. The typical price of this NIC is around $745.
The Competition: Solarflare
Solarflare is a relatively young company, founded in 2001. The main philosophy of Solarflare has been been “make Ethernet [over copper] better” (we added “over copper”). Solarflare NICs support optical media too, but Solarflare made headlines with their 10G Ethernet copper products. In 2006, Solarflare was the first with 10GBase-T PHY. 10GBase-T allows 10Gigabit over the very common and cheap CAT5E and the reasonably priced CAT6 and CAT6A UTP cables. Solarflare is also strongly advocating the use of 10GbE in the HPC world with the claim that the latency of 10GbE can be as low as 4 µs. In June of this year, Solarflare launched the SFN5121T, a dual-ported 10Gbase-T NIC which featured a very reasonable 12.9W power consumption, especially for a UTP based 10GbE product. In January of this year, the company decided to start selling NIC adapters directly to the end-user.
As we got an SFP+ Neterion X3120, we took a look at the optical brother of the SFN5121T, the SFN5122F. Both Solarflare NICs support SR-IOV and and make use of PCIe 2.0. The SFP+ SFN5122F should only consume a very low 4.9W for the complete card. Solarflare pulled this off by designing their own PHYs and reducing the chip count on the NIC. Although our power measurement methods (measured at the wall) are too crude to measure the exact power consumption, we can confirm that the Solarflare NIC consumed the least of the three NICs we tested.
The Solarflare chips are only slightly more expensive than the other NICs. Prices on the web were typically around $815.
The oldies
It is always interesting to get some “historical perspective”. Do these new cards outperform the older ones by a large margin? We included the Neterion XFrame-E for one test, and used the multi-queue pioneer and the Intel 82598, the 10GbE price breaker, as the historical reference for every benchmark. We'll try to add the Intel 82599 which also support SR-IOV, has a larger receive buffer, more queues and is priced around $700. We plugged those NICs in our Supermicro Twin² for testing.
Facebook
Twitter
RSS
38 Comments
View All Comments
fr500 - Wednesday, November 24, 2010 - link
I guess there is LACP or PAGP and some propietary solution.A quick google told me it's called cross-module trunking.
mlambert - Wednesday, November 24, 2010 - link
FCoE, iSCSI (*not that you would, but you could), FC, and IP all across the same link. Cisco offers VCP LACP with CNA as well. 2 links per server, 2 links per storage controller, thats not many cables.mlambert - Wednesday, November 24, 2010 - link
I meant VPC and Cisco is the only one that offers it today. I'm sure Brocade will in the near future.Zok - Friday, November 26, 2010 - link
Brocade's been doing this for a while with the Brocade 8000 (similar to the Nexus 5000), but their new new VDX series takes it a step further for FCoE.Havor - Wednesday, November 24, 2010 - link
Do these network adapters are real nice for servers, don't need a manged NIC, i just really want affordable 10Gbit over UTP ore STP.Even if its only 30~40M / 100ft because just like whit 100Mbit network in the old days my HDs are more then a little out preforming my network.
Wondering when 10Gbit will become common on mobos.
Krobar - Thursday, November 25, 2010 - link
Hi Johan,Wanted to say nice article first of all, you pretty much make the IT/Pro section what it is.
In the descriptions of the cards and conclusion you didnt mention Solarflares "Legacy" Xen netfront support. This only works for paravirt Linux VMs and requires a couple of extra options at kernal compile time but it run like a train and requires no special hardware support from the motherboard at all. None of the other brands support this.
marraco - Thursday, November 25, 2010 - link
I once made a resume of total cost of the network on the building where I work.Total cost of network cables was far larger than the cost of the equipment (at least with my country prices). Also, solving any cable related problem was a complete hell. The cables were hundreds, all entangled over the false roof.
I would happily replace all that for 2 of tree cables with cheap switches at the end. Selling the cables would pay for new equipment and even give a profit.
Each computer has his own cable to the central switch. A crazy design.
mino - Thursday, November 25, 2010 - link
IF you go 10G for cable consolidation, you better forget about cheap switches.The real saving are in the manpower, not the cables themselves.
myxiplx - Thursday, November 25, 2010 - link
If you're using a Supermicro Twin2, why don't you use the option for the on board Mellanox ConnectX-2? Supermicro have informed me that with a firmware update these will act as 10G Ethernet cards, and Mellanox's 10G Ethernet range has full support for SR-IOV:Main product page:
http://www.mellanox.com/content/pages.php?pg=produ...
Native support in XenServer 5:
http://www.mellanox.com/content/pages.php?pg=produ...
AeroWB - Thursday, November 25, 2010 - link
Nice Article,It is great to see more test around virtual environments. What surprises me a little bit is that at the start of the article you say that ESXi and Hyper-V do not support SR-IOV yet. So I was kind of expecting a test with Citrix Xenserver to show the advantages of that. Unfortunately it's not there. I hope you can do that in the near future.
I work with both Vmware ESX and Citrix XenServer we have a live setup of both. We started with ESX and later added a XenServer system, but as XenServer is getting more mature and gets more and more features we probably replace the ESX setup with XenServer (as it is much much cheaper) when maintenance runs out in about one year so I'm really interested in tests on that platform.