Investigating Cavium's ThunderX: The First ARM Server SoC With Ambition
by Johan De Gelas on June 15, 2016 8:00 AM EST- Posted in
- SoCs
- IT Computing
- Enterprise
- Enterprise CPUs
- Microserver
- Cavium
Selecting the Competition
In setting up our benchmarks, we chose four different Intel SKUs to compete with the Cavium ThunderX. Our choices are not ideal (as we only have a limited number of SKUs available) but there is still some logic behind the SKU choice.
The Xeon E5-2640 v4 (10 cores @2.4 GHz, $939) has Intel's latest server core (Broadwell EP) and features a price tag in the ballpark of the ThunderX ($800) along with a low 90W TDP.
The Xeon E5-2690 v3 (12 cores @2.6 GHz, $2090) is a less optimal choice, but we wanted an SKU with a higher TDP, in case that the actual power consumption of the Thunder-X is higher than what can be expected from the official 120W TDP. To be frank, it was the only SKU that was faster than the E5-2640 v4 that we had. The Xeon E5-2699v4 ($4115, 145W TDP) did not make much sense to us in this comparison... so we settled for the Xeon E5-2690v3.
And then we added all the Xeon Ds we had available. At first sight it's not fair to compare a 45W TDP SoC to our 120W ThunderX. But the Xeon D-1557 is in the same price range as the Cavium ThunderX, and is targeted more or less at the same market. And although they offer fewer network and SATA interfaces, Cavium has to beat these kind of Xeon Ds performance wise, otherwise Intel's performance per watt advantage will steal Cavium's thunder.
The Xeon D-1581 is the most expensive Xeon D, but it is Intel's current server SoC flagship. But if the ARM Server SoCs start beating competitively priced Xeon Ds, Intel can always throw this one in the fray with a lower price. It is the SoC the ARM server vendors have to watch.
Configuration
Most of our testing was conducted on Ubuntu Server 14.04 LTS. We did upgrade this distribution to the latest release (14.04.4), which gives us more extensive hardware support. However, to ensure support for the ThunderX, the gcc compiler was upgraded to 5.2. In case of the ThunderX, the kernel was also 4.2.0, while the Intel systems still used kernel 3.19.
The reason why we did not upgrade the kernel is simply that we know from experience that this can generate all kinds of problems. In the case of the ThunderX using a newer kernel was necessary, while for the Intel CPUs we simply checked that there were no big differences with the new Ubuntu 16.04. The only difference that we could see there is that some of our software now does not compile on 16.04 (Sysbench, Perlbench). As we already waste a lot of time with debugging all kinds of dependency trouble, we kept it simple.
Gigabyte R120-T30 (1U)
The full specs of the server can be found here.
| CPU | One ThunderX CN8890 |
| RAM | 128GB (4x32GB) DDR4-2133 |
| Internal Disks | 2x SanDisk CloudSpeed Ultra 800GB |
| Motherboard | Gigabyte MT30-GS0 |
| BIOS version | 1/28/2016 |
| PSU | Delta Electronics 400w 80 Plus Gold |
Supermicro X10SDV-7TP8F and X10SDV-12C-TLN4F (2U case)
| CPU | Xeon D-1557 (1.5 GHz, 12 cores, 45 W TDP) Xeon D-1581 (1.8 GHz, 16 cores, 65 W TDP) |
| RAM | 64 GB (4x16 GB) DDR4-2133 |
| Internal Disks | 2x Intel SSD3500 400GB |
| Motherboard | Supermicro X10SDV-7TP8F Supermicro X10SDV-12C-TLN4F |
| BIOS version | 5/5/2016 |
| PSU | Delta Electronics 400w 80 Plus Gold |
Hyperthreading, Turbo Boost, C1 and C6 were enabled in the BIOS.
Intel's Xeon E5 Server – S2600WT (2U Chassis)
This is the same server that we used in our latest Xeon v4 review.
| CPU | Xeon E5-2640 v4 (2.4 GHz, 10 cores, 90 W TDP) Xeon E5-2690 v3 (2.6 GHz, 12 cores, 135 W TDP) |
| RAM | 128GB (8x16GB) Kingston DDR-2400 |
| Internal Disks | 2x Intel SSD3500 400GB |
| Motherboard | Intel Server Board Wildcat Pass |
| BIOS version | 1/28/2016 |
| PSU | Delta Electronics 750W DPS-750XB A (80+ Platinum) |
Hyperthreading, Turboost, C1 and C6 were enabled in the BIOS.
Other Notes
All servers are fed by a standard European 230V (16 Amps max.) power line. The room temperature is monitored and kept at 23°C by our Airwell CRACs in our Sizing Servers Lab.
Facebook
Twitter
RSS
82 Comments
View All Comments
JohanAnandtech - Wednesday, June 15, 2016 - link
Good suggestion. I have been using an ipmi client to manage several other servers, like the IBM servers. However, such a GUI client is still a bit more userfriendly, ipmi commands can get complicated if you don't use them regularly. The thing is that HP and Intel's BMC GUI are a lot easier to use and more reliable.fanofanand - Wednesday, June 15, 2016 - link
I think you may have an inaccurate figure of 141 at idle (in the graph) for the Thunder. "makes us suspect that the chip is consuming between 40 and 50W at idle, as measured at the wall"JohanAnandtech - Wednesday, June 15, 2016 - link
If you look at the Column "peak vs idle", you see 82W. At peak, we assume that a 120W TDP chip will probably need about 130W. 130W - 82W (both measured at the wall) = 50W for the SoC alone at idle measured at the wall, so anywhere between 40-50W in reality. My Calculation is a "guestimate", but it is clear that the Cavium chip needs much more in idle than the Intel chips.(10-15W) .djayjp - Wednesday, June 15, 2016 - link
Many spelling/grammar issues here. It impacts readability. Please read before posting.djayjp - Wednesday, June 15, 2016 - link
That is to say in the article.mariush - Wednesday, June 15, 2016 - link
These guys are already working on ThunderX2 (54 cores, 3 Ghz , 14nm , ARMv8) and they already have functional chips : https://www.youtube.com/watch?v=ei9uVskwPNEMeteor2 - Thursday, June 16, 2016 - link
It's always jam tomorrow, isn't it? Intel is working on new chips too, you know.beginner99 - Wednesday, June 15, 2016 - link
It loses very clearly in performance/watt to Xeon-D. In this segment the lower price doesn't matter in that case and the fact that it has a process disadvantage doesn't matter either. What counts is the end result. And I doubt it would cost $800 if made on 14/16nm. I mean why would anyone buying this take the risk? Safer bet to go with Intel also due to more flexible use (single and multi threaded). The latency issue is mentioned but downplayed.blaktron - Wednesday, June 15, 2016 - link
So downplayed. Anandtech desperately wants ARM servers, but its a solution looking for a problem. Big web front ends running on bare metal are such a small percentage of the server market that developing for it seems stupid. Xeon-D was already in development for SANs, they just repurposed it for docker and nginx.Senti - Wednesday, June 15, 2016 - link
Very nice article. I especially liked the emphasis on relations of test numbers and real world workloads and what was problematic during the testing.It would be great to see the same style desktop CPU review (Zen?) form you instead of mix of reprinted marketing hype with silly benchmark numbers dump that plagues this site for quite some time now.
Some annoying typos here and there, like "It is clear that the ThunderX is a match for high frequency trading", but nothing really bad.