10G Ethernet: More Than a Big Pipeby Johan De Gelas on November 24, 2010 2:34 PM EST
Native Bandwidth: Windows Server 2008
Our first test was done with Windows 2008 Server R2 Enterprise installed on both nodes. We used Ixia IxChariot 5.4 to test throughtput. Two subtests were performed; one with standard Ethernet frame length (1.5KB) and one with 9KB jumbo frames. All tests were done with four threads. One of the most interesting enhancements in Windows Server 2008 is Receive-Side Scaling (RSS), a feature that spreads the receive processing out over several CPU cores. RSS was enabled in the driver properties.
All three NICs perform worse with jumbo frames than without. Jumbo frames lowers the CPU load for every NIC from 13-14% to 10%. Still only 5-5.6 Gbit/s is pretty disappointing. The Solarflare and Neterion NIC are clearly made for virtualized environments. CPU load was around 14% for the Neterion and Intel NICs when we tested with standard Ethernet frames, while Solarflare’s NIC needed 10%. When we enabled Jumbo frames, all NICs needed about 10% of the available CPU power (the Xeon E5504 at 2GHz).
The latency of the Solarflare NIC is higher than the other chips. Since Solarflare has put a lot of effort into an optimized opensource network stack for Linux called “OpenOnload”, we assume that the low latency that Solarflare claims are for Linux NIC latency. That makes sense as Linux is dominant in the HPC cluster world. In HPC clusters, latency matters more than bandwidth.