Testing the 10GBase-T

Many thanks to Brett Howse for his help!

As an indication of how far away from mainstream adoption of 10GBase-T we are, our testing setup was not ready to receive 10GBase-T – we have no switches or other PCIe cards in house to test the ports.  Apart from that, one of the downsides of testing network ports as a whole is that any test out one machine and into another, meaning that and any result will always be at the whims of the lowest common denominator between the two systems. The easiest way to test is therefore to essentially loop back on itself. As the X540-BT2 is a dual port solution, this was ideal for the testing scenario.

We set up the system with ESXi and two Windows Server 2012 VMs, each allocated with 8 threads, 16 GB of DRAM and one of the 10GBase-T ports with custom IPs. We then used LAN Speed Test to set up a server on one VM and a client on the other. LAN Speed Test allows us to simulate multiple clients through the same cable, effectively probing our ports similar to a SOHO/SMB environment.

We organized a series of send/receive commands to go through the connections with differing numbers of streams and differing amounts of data per stream. The system was set to repeat for 10 minutes, and the peak transfer rate was recorded. Results shown are in Gbps.

There are several key points to note with these results.

  • Firstly, single client speed never broke 2.2 Gbps. This puts an absolute limit on individual point-to-point communication speed in our system setup.
  • Next, with between 4-9 clients the speed is fairly consistent between 6.7 and 8 Gbps, no matter what the size of the transfer is.
  • Thus in order to get peak throughput, at least 10 concurrent connections need to be made. This is when 8+ Gbps is observed.
  • With a small number of clients, a longer transfer results in higher overall speed. However with a larger amount of clients, faster transfers results in higher peak speed.

As part of the test, we also examined CPU usage when each stream was set for 1GB transfers. Normally CPU usage for a standard 1Gbit Ethernet port is minimal although one of the factors that some companies like to point to for increased gaming rates. Normally this rarely goes above a few percent on a quad core system, but with the X540-BT2 that changed, especially when the system was being hammered.

As the main VM alternated between reading and writing from the server VM, reading CPU usage peaked after 10 concurrent clients but write CPU usage shot up very quickly to 50% and stayed there. This might explain the slow increases in peak performance we observed, if the software simply ran out of threads and was only geared for four threads.

With a 4 core VM we saw 100% usage during writes, and above was the result in terms of CPU performance monitoring.

This marks an interesting juncture, suggesting that a faster single threaded CPU could deliver better performance and that the X540T-BT2 would be better attached to a Haswell/Broadwell platform – at least in our testing scenario.  The truth of the matter is that fast connectivity technology runs on optimized FPGAs because general purpose CPUs cannot keep up with what is needed.

So where does leave the X99 WS-E/10G? The best example I could propose is in that SOHO/SMB environment where the system is connected to a 10Gbit/1Gbit mixed switch that has 24+ clients that all need to access the system, either as a virtualized workspace, some form of storage or a joint streaming venture. As the former, it allows the rest of the office to use very basic machines and rely on the grunt of the virtualized environment to perform tasks.

Additional: As mentioned in the comments, this is almost the complete out-of-the-box scenario where the only thing configured is the RAMDisk transfers and multiple stream application, whereas most users might be limited by the SSD speed. Jammrock in the comments has posted a list in helping to optimize the connection solely for individual point-to-point transfers and is worth a read.

Overview, Visual Inspection, Board Features BIOS
POST A COMMENT

45 Comments

View All Comments

  • DanNeely - Monday, December 15, 2014 - link

    "Users should also note that only one [motherboard molex plug] needs to be connected when 3+ PCIe devices are used to help boost power. I quizzed them on SATA power connectors instead, or a 6-pin PCIe, however the response was not enthusiastic."

    I can understand them not liking the idea of using a PCIe cable if they don't need more than a single 12v pin for extra power because a lot of users wouldn't have an extra available to use without going for a kludgy molex-pcie adapter; but what's the problem with using a sata power plug?
    Reply
  • themeinme75 - Tuesday, December 16, 2014 - link

    I find this interesting the molex spec is 60 watts on the 12v line, even though you probably get 75+ safely and sata would have 54 watts for 12v line. I think for a MB that cost 500+ with user that plan multiple GPU I think you can get a power supply with plenty of connections. Reply
  • themeinme75 - Tuesday, December 16, 2014 - link

    http://www.moddiy.com/pages/Power-Supply-Connector... Reply
  • wolrah - Tuesday, December 16, 2014 - link

    My guess is it's because the Molex connector is significantly more durable and any modern system is sure to have one available.

    A home server would be a pretty logical role for this board, so you might have already devoted all 12 of the SATA power connectors most power supplies ship with to running hard drives. Likewise as you note a workstation could easily use up their PCIe power connections with GPUs.

    That leaves the Molex as the one of the three regularly available power connections which can be most expected to be available.
    Reply
  • Dakosta Le'Marko - Thursday, December 18, 2014 - link

    Stunning! I've started averaging 85 dollars/hourly since i started working online half a year ago... What i do is to sit at home several hours each day and do simple jobs i get from this company that i found over the internet... I am very happy to share this with you... It's an awesome side job to have http://orkan201.tk Reply
  • ddriver - Monday, December 15, 2014 - link

    Price : US (Newegg) ->
    Search Terms: "x99 ws-e 10g"
    We have found 0 items that match "x99 ws-e 10g".
    Reply
  • Ian Cutress - Monday, December 15, 2014 - link

    While the product is officially announced, it doesn't seem to have filtered through yet, hence we don't know the pricing. When it gets to Newegg, hopefully that link will show it. Reply
  • ShieTar - Monday, December 15, 2014 - link

    There are a few offers in Europe already, for >700€ (>900$):

    http://preview.tinyurl.com/mduqtgu
    Reply
  • akula2 - Thursday, December 18, 2014 - link

    Thanks for mentioning the price. I'll settle for Asus X99-E WS boards. Reply
  • Pcgeek21 - Monday, December 15, 2014 - link

    Were jumbo frames used for the 10GBASE-T testing? They would need to be enabled both inside the VMs and in ESXi's virtual switches (if they were used). My recollection is that jumbo frames were created to deal with the problems you encountered with CPU usage on 10Gb links. Reply

Log in

Don't have an account? Sign up now