Investigating Cavium's ThunderX: The First ARM Server SoC With Ambition
by Johan De Gelas on June 15, 2016 8:00 AM EST- Posted in
- SoCs
- IT Computing
- Enterprise
- Enterprise CPUs
- Microserver
- Cavium
Selecting the Competition
In setting up our benchmarks, we chose four different Intel SKUs to compete with the Cavium ThunderX. Our choices are not ideal (as we only have a limited number of SKUs available) but there is still some logic behind the SKU choice.
The Xeon E5-2640 v4 (10 cores @2.4 GHz, $939) has Intel's latest server core (Broadwell EP) and features a price tag in the ballpark of the ThunderX ($800) along with a low 90W TDP.
The Xeon E5-2690 v3 (12 cores @2.6 GHz, $2090) is a less optimal choice, but we wanted an SKU with a higher TDP, in case that the actual power consumption of the Thunder-X is higher than what can be expected from the official 120W TDP. To be frank, it was the only SKU that was faster than the E5-2640 v4 that we had. The Xeon E5-2699v4 ($4115, 145W TDP) did not make much sense to us in this comparison... so we settled for the Xeon E5-2690v3.
And then we added all the Xeon Ds we had available. At first sight it's not fair to compare a 45W TDP SoC to our 120W ThunderX. But the Xeon D-1557 is in the same price range as the Cavium ThunderX, and is targeted more or less at the same market. And although they offer fewer network and SATA interfaces, Cavium has to beat these kind of Xeon Ds performance wise, otherwise Intel's performance per watt advantage will steal Cavium's thunder.
The Xeon D-1581 is the most expensive Xeon D, but it is Intel's current server SoC flagship. But if the ARM Server SoCs start beating competitively priced Xeon Ds, Intel can always throw this one in the fray with a lower price. It is the SoC the ARM server vendors have to watch.
Configuration
Most of our testing was conducted on Ubuntu Server 14.04 LTS. We did upgrade this distribution to the latest release (14.04.4), which gives us more extensive hardware support. However, to ensure support for the ThunderX, the gcc compiler was upgraded to 5.2. In case of the ThunderX, the kernel was also 4.2.0, while the Intel systems still used kernel 3.19.
The reason why we did not upgrade the kernel is simply that we know from experience that this can generate all kinds of problems. In the case of the ThunderX using a newer kernel was necessary, while for the Intel CPUs we simply checked that there were no big differences with the new Ubuntu 16.04. The only difference that we could see there is that some of our software now does not compile on 16.04 (Sysbench, Perlbench). As we already waste a lot of time with debugging all kinds of dependency trouble, we kept it simple.
Gigabyte R120-T30 (1U)
The full specs of the server can be found here.
| CPU | One ThunderX CN8890 |
| RAM | 128GB (4x32GB) DDR4-2133 |
| Internal Disks | 2x SanDisk CloudSpeed Ultra 800GB |
| Motherboard | Gigabyte MT30-GS0 |
| BIOS version | 1/28/2016 |
| PSU | Delta Electronics 400w 80 Plus Gold |
Supermicro X10SDV-7TP8F and X10SDV-12C-TLN4F (2U case)
| CPU | Xeon D-1557 (1.5 GHz, 12 cores, 45 W TDP) Xeon D-1581 (1.8 GHz, 16 cores, 65 W TDP) |
| RAM | 64 GB (4x16 GB) DDR4-2133 |
| Internal Disks | 2x Intel SSD3500 400GB |
| Motherboard | Supermicro X10SDV-7TP8F Supermicro X10SDV-12C-TLN4F |
| BIOS version | 5/5/2016 |
| PSU | Delta Electronics 400w 80 Plus Gold |
Hyperthreading, Turbo Boost, C1 and C6 were enabled in the BIOS.
Intel's Xeon E5 Server – S2600WT (2U Chassis)
This is the same server that we used in our latest Xeon v4 review.
| CPU | Xeon E5-2640 v4 (2.4 GHz, 10 cores, 90 W TDP) Xeon E5-2690 v3 (2.6 GHz, 12 cores, 135 W TDP) |
| RAM | 128GB (8x16GB) Kingston DDR-2400 |
| Internal Disks | 2x Intel SSD3500 400GB |
| Motherboard | Intel Server Board Wildcat Pass |
| BIOS version | 1/28/2016 |
| PSU | Delta Electronics 750W DPS-750XB A (80+ Platinum) |
Hyperthreading, Turboost, C1 and C6 were enabled in the BIOS.
Other Notes
All servers are fed by a standard European 230V (16 Amps max.) power line. The room temperature is monitored and kept at 23°C by our Airwell CRACs in our Sizing Servers Lab.
Facebook
Twitter
RSS
82 Comments
View All Comments
silverblue - Thursday, June 16, 2016 - link
I'm not sure how this is relevant. Johan doesn't review graphics cards, other people at Anandtech do. I bet Guru3D has a much bigger team for that, and I imagine that they have a much narrower scope (i.e. no server stuff).I don't think I've looked at a review recently that hasn't had the comments section polluted with "where is the review for x".
UrQuan3 - Wednesday, June 15, 2016 - link
Intel allows their Xeons to sometimes pull double their TDP? No wonder our new machines trip breakers long before I thought they would. I need to test instead of assuming accurate documentation.I can see why you chose C-Ray, I'm just sorry a more general ray tracer was not chosen. Still, not it's intended market, though I am suddenly very interested. Ray-tracing and video encoding are my top two tasks.
Meteor2 - Thursday, June 16, 2016 - link
The 'T' in 'TDP' is for thermal. It's a measure of the maximum waste heat which needs to be removed over a certain period of time.UrQuan3 - Wednesday, June 22, 2016 - link
Yes, it stands for thermal, but power doesn't consumed doesn't just disappear. Convert it to light, convert it to motion, convert it to heat, etc. In this case there is a small amount of motion (electrons) and the rest has to be heat. I expect much higher instantaneous pulls, but this was sustained power. Anyway, I will track down the AVX documentation mentioned below.I saw the h264ref. I'll be curious about x264 (handbrake) as the authors seem interested in ARM in the last few years. Unsurprisingly, it is far less optimized than x64. I benchmarked handbrake on the Pi2, Pandaboard, and CI-20 last year, just to see what it would do.
JohanAnandtech - Thursday, June 16, 2016 - link
C-Ray was just a place holder to measure FPU energy consumption. I look into bringing a more potent raytracer into our benchmark suite (povray)Video encoding was in the review though, somewhat (h264ref).
patrickjp93 - Friday, June 17, 2016 - link
ARM chips with vector extensions allow it as well. Intel provides separate documentation for AVX-workload TDPs.Antony Newman - Wednesday, June 15, 2016 - link
Fascinating article.Why would Cavium not try and use 54 x A73s in their next chip?
If ARM are not in the business of making Silicon, and ARM think the '1.2W Ares' will help them break into the Server market ... Then Why do we think ARM isn't working with the likes of Cavium to get a Server SoC that rocks the Intel boat?
Typos From memory : send -> sent. Through-> thought. There were a few others.
AJ
name99 - Thursday, June 16, 2016 - link
How do you know ARM aren't working with such a vendor?ARM has always said that they expect ARM server CPUs to only be marginally competitive (for very limited situations) in 2017, and to only be really competitive in 2020.
That suggests, among other things, that if they are working with partners, they have a target launch between those two dates, and they regard all launches before 2017 as essentially nice for PR and fr building up the ecosystem, but essentially irrelevant for commercial purposes.
rahvin - Thursday, June 16, 2016 - link
The problem as pointed out early in this article is that ARM keeps targeting Intel's current products, not the ones that will be out when they get their products out. We've had almost a dozen vendors get to the point of releasing the chip and drop it because it is simply not competitive with Intel. Most of these arm products were under taken when Intel was targeting performance without regard to performance/watt. Now that intel targets the later metric arm server chips haven't been competitive with them.Fact is Intel could decimate and totally take over all the markets arm chips occupy, but to do it they'd have to cannibalize their existing high profit sales. This is why they keep canceling Atom chips, the chips turned out so good they were worried they'd cannibalize much more expensive products. This is the reason Avoton is highly restricted in what products and price segments it's allowed into. If Intel opened the flood gates on Avoton they would risk cannibalizing their own server profits.
junky77 - Wednesday, June 15, 2016 - link
So, they did what AMD couldn't for years? I'm trying to figure it out.. their offering seems to be a lot more interesting than AMD's stuff currently