Investigating Cavium's ThunderX: The First ARM Server SoC With Ambition
by Johan De Gelas on June 15, 2016 8:00 AM EST- Posted in
- SoCs
- IT Computing
- Enterprise
- Enterprise CPUs
- Microserver
- Cavium
Single-Threaded Integer Performance: SPEC CPU2006
Even though SPEC CPU2006 is more HPC and workstation oriented, it contains a good variety of integer workloads. Running SPEC CPU2006 is a good way to evaluate single threaded (or core) performance. The main problem is that the results submitted are "overengineered" and it is very hard to make any fair comparisons.
So we wanted to keep the settings as "real world" as possible. We welcome constructive criticism to reach that goal. So we used:
- 64 bit gcc: most used compiler on Linux, good all round compiler that does not try to "break" benchmarks (libquantum...)
- -Ofast: compiler optimization that many developers may use
- -fno-strict-aliasing: necessary to compile some of the subtests
- base run: every subtest is compiled in the same way.
The ultimate objective is to measure performance in applications where for some reason – as is frequently the case – a "multi thread unfriendly" task keeps us waiting.
Nobody expect the ThunderX to be a single threaded performance wonder. Cavium clearly stated that they deliberately went for a high core count with pretty simple cores. As a result, single threaded performance was not a priority.
However, Facebook and other hyperscalers have indicated that they definitely prefer to get the single threaded performance of a Xeon D. So any competitor challenging Intel should try to keep up with the Xeon D in single threaded performance and offer a throughput-per-dollar/watt bonus. So it is very interesting to measure what single threaded performance the current ThunderX can offer.
| Subtest SPEC CPU2006 Integer |
Application Type | Cavium ThunderX 2 GHz |
Xeon D-1557 1.5-2.1 |
Xeon D-1587 1.8-2.4 |
Xeon E5-2640 v4 2.4-2.6 |
Xeon E5-2690 v3 2.6-3.5 |
Xeon E5-2699 v4 2.2-3.6 |
Xeon E5-2699 v4 2.2-3.6 (+HT) |
| 400.perlbench | Spam filter | 8.3 | 24.7 | 29 | 33.4 | 39 | 32.2 | 36.6 |
| 401.bzip2 | Compression | 6.5 | 15.1 | 17.2 | 19.8 | 24.2 | 19.2 | 25.3 |
| 403.gcc | Compiling | 10.8 | 23.1 | 27.2 | 30 | 37.2 | 28.9 | 33.3 |
| 429.mcf | Vehicle scheduling | 10.2 | 32.6 | 38.4 | 40.4 | 44.8 | 39 | 43.9 |
| 445.gobmk | Game AI | 9.2 | 17.4 | 20.2 | 22.7 | 28.1 | 22.4 | 27.7 |
| 456.hmmer | Protein seq. analyses | 4.8 | 19 | 21.7 | 25.1 | 28 | 24.2 | 28.4 |
| 458.sjeng | Chess | 8.8 | 19.8 | 22.8 | 25.6 | 31.5 | 24.8 | 28.3 |
| 462.libquantum | Quantum sim | 5.8 | 47.9 | 58.2 | 60.3 | 78 | 59.2 | 67.3 |
| 464.h264ref | Video encoding | 11.9 | 32 | 36.6 | 41.9 | 56 | 40.7 | 40.7 |
| 471.omnetpp | Network sim | 7 | 17.3 | 23 | 23.6 | 30.9 | 23.5 | 29.9 |
| 473.astar | Pathfinding | 7.9 | 14.7 | 17.2 | 19.8 | 24.4 | 18.9 | 23.6 |
| 483.xalancbmk | XML processing | 8.4 | 27.8 | 33.3 | 36.2 | 45.1 | 35.4 | 41.8 |
Although some of you have a mathematical mind and are able to easily decipher these kinds of tables, let the rest of us be lazy and translate this into percentages. We make the Xeon D-1581 the baseline. The Xeon D-1557's performance is more or less the single threaded performance some of the important customers such as Facebook like to have.
| Subtest SPEC CPU2006 Integer |
Application Type | Cavium ThunderX 2 GHz |
Xeon D-1557 1.5-2.1 |
Xeon D-1581 1.5-2.1 |
Xeon E5-2640 2.4-2.6 |
| 400.perlbench | Spam filter | 29% | 85% | 100% | 115% |
| 401.bzip2 | Compression | 38% | 88% | 100% | 115% |
| 403.gcc | Compiling | 40% | 85% | 100% | 110% |
| 429.mcf | Vehicle scheduling | 27% | 85% | 100% | 105% |
| 445.gobmk | Game AI | 46% | 86% | 100% | 112% |
| 456.hmmer | Protein seq. analyses | 22% | 88% | 100% | 116% |
| 458.sjeng | Chess | 39% | 87% | 100% | 112% |
| 462.libquantum | Quantum sim | 10% | 82% | 100% | 104% |
| 464.h264ref | Video encoding | 33% | 87% | 100% | 114% |
| 471.omnetpp | Network sim | 30% | 75% | 100% | 103% |
| 473.astar | Pathfinding | 46% | 85% | 100% | 115% |
| 483.xalancbmk | XML processing | 25% | 83% | 100% | 109% |
First of all, single threaded is somewhat better than we expected when we received the first architectural details (a very simple dual issue core with high latency shared L2). However, this is still a fraction of the Xeon D's single threaded performance, which means that ThunderX doesn't look very impressive to companies which feel that single threaded performance should not be lower than a low end Xeon D. The latter is 2 to 4 times faster. On average, the Xeon D-1581 delivers 3 times faster single threaded performance than the ThunderX, but not 5!
SPEC CPU2006 allows us to characterize the ThunderX core a bit better. We ignore libquantum because it has a very special profile: you can triple the score with specific compiler settings, but those settings reduce performance by 2-30%(!) in some other subtests. Those compiler settings optimize cache utilization by splitting records of an array in separate arrays. Combine this with software loop prefetching and libquantum numbers can indeed double or triple. Since libquantum is hardly relevant for the server world and is known for being a target for all kind of benchmark trickery, we ignore it in our comparison.
Mcf exhibits a large amount of data cache misses and memory controller usage. Mcf is also "horribly low IPC" software, so beefy execution backends do not help. Despite those facts, the ThunderX does not do well in mcf. Mcf does a lot of pointer chasing, so the high latency L2-cache and the high latency DRAM access are slowing things down. That is probably also true for XML processing and the network simulator: those subtests have the highest data cache misses.
The shallow pipeline and relatively powerful gshare branch predictor make the ThunderX a better than expected performer in the chess (sjeng), pathfinding (astar), compiling (gcc) and AI (gobmk). Although the gobmk has a relatively high branch misprediction rate on a gshare branch predictor (the highest of all subtests), the ThunderX core can recover very quickly thanks to its 9 stage pipeline. Notice also that gobmk and gcc have relatively large instruction footprints, which gives the ThunderX and its 78 KB I-cache an advantage.
That is also true for the perl, but that benchmark has a relatively high IPC and needs a beefier execution backend. Indeed, the more compute intensive (and thus high IPC sub tests) perlbench and hmmer perform badly relative to the Intel core. In these benchmarks, the wide architecture of the Intel cores pays off.
Facebook
Twitter
RSS
82 Comments
View All Comments
silverblue - Thursday, June 16, 2016 - link
I'm not sure how this is relevant. Johan doesn't review graphics cards, other people at Anandtech do. I bet Guru3D has a much bigger team for that, and I imagine that they have a much narrower scope (i.e. no server stuff).I don't think I've looked at a review recently that hasn't had the comments section polluted with "where is the review for x".
UrQuan3 - Wednesday, June 15, 2016 - link
Intel allows their Xeons to sometimes pull double their TDP? No wonder our new machines trip breakers long before I thought they would. I need to test instead of assuming accurate documentation.I can see why you chose C-Ray, I'm just sorry a more general ray tracer was not chosen. Still, not it's intended market, though I am suddenly very interested. Ray-tracing and video encoding are my top two tasks.
Meteor2 - Thursday, June 16, 2016 - link
The 'T' in 'TDP' is for thermal. It's a measure of the maximum waste heat which needs to be removed over a certain period of time.UrQuan3 - Wednesday, June 22, 2016 - link
Yes, it stands for thermal, but power doesn't consumed doesn't just disappear. Convert it to light, convert it to motion, convert it to heat, etc. In this case there is a small amount of motion (electrons) and the rest has to be heat. I expect much higher instantaneous pulls, but this was sustained power. Anyway, I will track down the AVX documentation mentioned below.I saw the h264ref. I'll be curious about x264 (handbrake) as the authors seem interested in ARM in the last few years. Unsurprisingly, it is far less optimized than x64. I benchmarked handbrake on the Pi2, Pandaboard, and CI-20 last year, just to see what it would do.
JohanAnandtech - Thursday, June 16, 2016 - link
C-Ray was just a place holder to measure FPU energy consumption. I look into bringing a more potent raytracer into our benchmark suite (povray)Video encoding was in the review though, somewhat (h264ref).
patrickjp93 - Friday, June 17, 2016 - link
ARM chips with vector extensions allow it as well. Intel provides separate documentation for AVX-workload TDPs.Antony Newman - Wednesday, June 15, 2016 - link
Fascinating article.Why would Cavium not try and use 54 x A73s in their next chip?
If ARM are not in the business of making Silicon, and ARM think the '1.2W Ares' will help them break into the Server market ... Then Why do we think ARM isn't working with the likes of Cavium to get a Server SoC that rocks the Intel boat?
Typos From memory : send -> sent. Through-> thought. There were a few others.
AJ
name99 - Thursday, June 16, 2016 - link
How do you know ARM aren't working with such a vendor?ARM has always said that they expect ARM server CPUs to only be marginally competitive (for very limited situations) in 2017, and to only be really competitive in 2020.
That suggests, among other things, that if they are working with partners, they have a target launch between those two dates, and they regard all launches before 2017 as essentially nice for PR and fr building up the ecosystem, but essentially irrelevant for commercial purposes.
rahvin - Thursday, June 16, 2016 - link
The problem as pointed out early in this article is that ARM keeps targeting Intel's current products, not the ones that will be out when they get their products out. We've had almost a dozen vendors get to the point of releasing the chip and drop it because it is simply not competitive with Intel. Most of these arm products were under taken when Intel was targeting performance without regard to performance/watt. Now that intel targets the later metric arm server chips haven't been competitive with them.Fact is Intel could decimate and totally take over all the markets arm chips occupy, but to do it they'd have to cannibalize their existing high profit sales. This is why they keep canceling Atom chips, the chips turned out so good they were worried they'd cannibalize much more expensive products. This is the reason Avoton is highly restricted in what products and price segments it's allowed into. If Intel opened the flood gates on Avoton they would risk cannibalizing their own server profits.
junky77 - Wednesday, June 15, 2016 - link
So, they did what AMD couldn't for years? I'm trying to figure it out.. their offering seems to be a lot more interesting than AMD's stuff currently