10G Ethernet: More Than a Big Pipe
by Johan De Gelas on November 24, 2010 2:34 PM EST- Posted in
- IT Computing
- Networking
- 10G Ethernet
Exar's Neterion Solution
SR-IOV will be supported in ESX 5.0 and the successor of Windows Server 2008. Since VMware’s ESX is the dominant hypervisor in most datacenters, that means that a large part of the already virtualized servers will have to wait a year or more before they can get the benefits of SR-IOV.
Exar, the pioneer of multiple devices queues, saw a window of opportunity. Besides the standard SR-IOV and VMware NetQueue support, the X3100 NICs also have a proprietary SR-IOV implementation.
Poprietary solutions only make sense if they offer enough advantages. Neterion claims that the NIC chip has extensive hardware support for network prioritization and quality of service. That hardware support should be superior to hypervisor traffic shaping, especially on the receive side. After all, if bursty traffic causes the NIC to drop packets on the receive side, there is nothing the hypervisor can do: it never saw those packets pass. To underline this, Neterion equips the X3100 with a massive 64MB receive buffer; for comparison, the competitors have in the best case a 512KB receive buffer. This huge receive buffer should ensure that the QoS is guaranteed even if relatively long bursts of network traffic occur.
Neterion NICs can be found in IBM, HP, Dell, Fujitsu, and Hitachi machines. Neterion is part of Exar and has also access to a world distributor channel. The typical price of this NIC is around $745.
The Competition: Solarflare
Solarflare is a relatively young company, founded in 2001. The main philosophy of Solarflare has been been “make Ethernet [over copper] better” (we added “over copper”). Solarflare NICs support optical media too, but Solarflare made headlines with their 10G Ethernet copper products. In 2006, Solarflare was the first with 10GBase-T PHY. 10GBase-T allows 10Gigabit over the very common and cheap CAT5E and the reasonably priced CAT6 and CAT6A UTP cables. Solarflare is also strongly advocating the use of 10GbE in the HPC world with the claim that the latency of 10GbE can be as low as 4 µs. In June of this year, Solarflare launched the SFN5121T, a dual-ported 10Gbase-T NIC which featured a very reasonable 12.9W power consumption, especially for a UTP based 10GbE product. In January of this year, the company decided to start selling NIC adapters directly to the end-user.
As we got an SFP+ Neterion X3120, we took a look at the optical brother of the SFN5121T, the SFN5122F. Both Solarflare NICs support SR-IOV and and make use of PCIe 2.0. The SFP+ SFN5122F should only consume a very low 4.9W for the complete card. Solarflare pulled this off by designing their own PHYs and reducing the chip count on the NIC. Although our power measurement methods (measured at the wall) are too crude to measure the exact power consumption, we can confirm that the Solarflare NIC consumed the least of the three NICs we tested.
The Solarflare chips are only slightly more expensive than the other NICs. Prices on the web were typically around $815.
The oldies
It is always interesting to get some “historical perspective”. Do these new cards outperform the older ones by a large margin? We included the Neterion XFrame-E for one test, and used the multi-queue pioneer and the Intel 82598, the 10GbE price breaker, as the historical reference for every benchmark. We'll try to add the Intel 82599 which also support SR-IOV, has a larger receive buffer, more queues and is priced around $700. We plugged those NICs in our Supermicro Twin² for testing.
Facebook
Twitter
RSS
38 Comments
View All Comments
mino - Thursday, November 25, 2010 - link
Well the main issue with 10G, especially copper, is that the medium frequencies are nowhere near FC.Then there is the protocol overhead for iSCSI, FCoE is musch better in that respect though.
That IOps measure was a reliably achievable peak - meaning generally with <4k IO operations.
10k on Gbit can be done in the lab easily, but it was not sustainable/reliable-enough to consider it for production use.
Those disk arrays have caches, today they even have SSD's etc. etc. then there is the dark fiber DR link one has to wait for ...
But yes, in a typical virtualized web-serving or SMB scenario 10G makes all the sense in the world.
All I ask is that you generally not dismiss FC without discussing its strengths.
It is a real pain explaining to a CIO why 10G will not work out when a generally reputable site AT says it is just "better" than FC.
JohanAnandtech - Thursday, November 25, 2010 - link
I do not dismiss FC. You are right that we should add FC to the mix. I'll do my best to get this is in another article. That way we can see if the total latency (= actual response times) are really worse on iSCSI than on FC.Then again, it is clear that if you (could) benefit from 8 gbit/s FC now, consolidating everything into a 10 Gbit pipe is a bad idea. I really doubt 10 GbE iSCSI is worse than 4 Gb FC, but as always, the only way to check this, is to measure.
gdahlm - Thursday, November 25, 2010 - link
The main issue is that Ethernet is designed to drop packets. This means that to be safe all iSCSI writes need to be synchronous and this means you will be hitting the disks hard or you are going to risk data loss if congestion starts dropping packets or you fill your ring buffer etc..Even with ZFS and a SSD ZIL you will be slower then ram based WriteBack Cache.
As an example here are some filebench oltp results from a linux based host to a zfs array over 4GB FC.
Now this is a pretty cheap array but will show the difference between a SSD backed ZIL and using the memory in writeback mode.
Host: noop elevator with Direct IO to SSD ZIL
6472: 77.657: IO Summary: 486127 ops, 8099.587 ops/s, (4040/4018 r/w), 31.6mb/s, 683us cpu/op, 48.4ms latency
Host: noop elevator with Direct IO with WB cache on the target.
18042: 73.066: IO Summary: 767336 ops, 12778.487 ops/s, (6373/6340 r/w), 50.0mb/s, 481us cpu/op, 29.6ms latency
Basic FC switches are cheap compared to 10gig switches at this point in time too.
JohanAnandtech - Friday, November 26, 2010 - link
"The main issue is that Ethernet is designed to drop packets. This means that to be safe all iSCSI writes need to be synchronous"IMHO, this only means that congestion is a worse thing for iSCSI (so you need a bit more headroom). Why would writes not be async? Once you hit the cache of your iSCSI target, the packets are in the cache, Ethernet is not involved anymore. So a smart controller can perform writes async. As matt showed with his ZFS article.
"Even with ZFS and a SSD ZIL you will be slower then ram based WriteBack Cache."
Why would write back cache not be possible with iSCSI?
"Basic FC switches are cheap compared to 10gig switches at this point in time too. "
We just bought a 10GbE switch from Dell: about $8000 for 24 ports. Lots of 24 port FC Switches are quite a bit more expensive. It is only a matter of time before 10GbE switches are much cheaper.
Also, with VLAN, the 10GbE can be used for both network as SAN traffic.
gdahlm - Friday, November 26, 2010 - link
I may be missing the part where Matt talked about async on ZFS. I only see where he was discussing using SSDs for an external ZIL. However writing to the ZIL is not an async write, it is fully committed to disk even if that commit is happening on an external log device.There are several vendors who do support async iSCSI writes using battery backed cache etc.. But to move up into the 10G performance level puts them at a price point where the costs of switches is fairly trivial.
iSCSI is obviously is a route-able protocol and thus often it is not just traversing one top of rack switch. Due to this lots of targets and initiators tend to be configured a conservative manor. COMSTAR is one such product, all COMSTAR iSCSI writes are synchronous, thus the reason you gain any advantage from an external ZIL. It appears that the developers assumed that FC storage is “reliable” and thus by default (at least in sol 11 express) zvols that are exported through COMSTAR are configured as writeback by default. You need to actually use stmfadm to specifically enable the honoring of sync writes and thus the use of the ZIL on pool or ssd.
I do agree that 10Gig Ethernet will be cheaper soon. I do not agree that it is cheaper at the moment.
Dell does have a 10GbE switch for about 8K but that is without the SFP modules. Qlogic has a 20 port 8GB switch that can be purchased for about 9K with the SFP modules.
If you have a need for policy on your network side or other advanced features the cost per port for 10GbE goes up dramatically.
I do fully expect that this cost difference will change dramatically over the next 12 months.
Ideally had SUN not been sold when they were we would probably have a usable SAS target today, but it appears that work on the driver has stopped.
This would have enabled the use of LSI's SAS switches which are dirt cheap to provide full 4 lane 6Gbs connectivity to hosts through truly inexpensive SAS HBAs.
Photubias - Friday, November 26, 2010 - link
quote "That same vendor has observed the best 1GbE solutions choke at <5k IOps..."This guy achieves 20k IOPS through 1GbE: http://www.anandtech.com/show/3963/zfs-building-te... ?
blowfish - Wednesday, November 24, 2010 - link
Trying to read this article is making my brain hurt! ;(It surprises me to see what looks like pci connectors on the NIC;s though. Are servers slower to adopt new interfaces?
Alroys - Wednesday, November 24, 2010 - link
They are not PCI, they actually are PCIe x8.blowfish - Thursday, November 25, 2010 - link
oh, thanks for the clarification!blandead - Wednesday, November 24, 2010 - link
On that note, does anyone know how to combine ports to aggregate with an extreme switch rather than a fail-over solution like as mentioned above.. combining 1GbE ports. Just a general idea of commands will point me in right direction : )Much appreciated if anyone replies!