Introduction to ARM Servers

"Intel does not have any competition whatsoever in the midrange and high-end (x86) server market". We came to that rather boring conclusion in our review of the Xeon E5-2600 v2. That date was September 2013.

At the same time, the number of announcements and press releases about ARM server SoCs based on the new ARMv8 ISA were almost uncountable. AppliedMicro was announcing their 64-bit ARMv8 X-Gene back in late 2011. Calxeda sent us a real ARM-based server at the end of 2012. Texas Instruments, Cavium, AMD, Broadcom, and Qualcomm announced that they would be challenging Intel in the server market with ARM SoCs. Today, the first retail products have finally appeared in the HP Moonshot server.

There has been no lack of muscular statements about the ARM Server SoCs. For example, Andrew Feldman, the founder of micro server pioneer Seamicro and the former head of the server department at AMD stated: "In the history of computers, smaller, lower-cost, and higher-volume CPUs have always won. ARM cores, with their low-power heritage in devices, should enable power-efficient server chips." One of the most infamous silicon valley insiders even went so far as to say, "ARM servers are currently steamroller-ing Intel in key high margin areas but for some reason the company is pretending they don’t exist."

Rest assured, we will not stop at opinions and press releases. As people started talking specifications, we really got interested. Let's see how the Cavium Thunder-X, AppliedMicro X-Gene, Broadcom Vulcan, and AMD Opteron A1100 compare to the current and future Intel Server chips. We are working hard to get all these contenders in our lab, and we are having some success, but it is too soon for a full blown shoot out.

Micro Servers and Scale-out Servers

Micro servers were the first the target of the ARM licensees. Typically, a discussion about Micro servers quickly turns into a wimpy versus brawny core debate. One of the reasons for that is that Seamicro, the inventor of the micro server, first entered the market with Atom CPUs. The second reason is that Calxeda, the pioneer of ARM based servers, had to work with the fact that the Cortex-A9 core was a wimpy core that could not deal with most server workloads. Wikipedia also associates micro servers with very low power SoCs: “Very low power and small size server based on System-on-Chip, typically centered around ARM processor”.

Micro servers are typically associated with low end servers that serve static HTML, cache web objects, and/or function as slow storage servers. It's true that you will not find a 150W high-end Xeon inside a micro server, but that does not mean that micro servers are defined by low power SoCs. In fact, the most successful micro servers are based on 15-45W Xeon E3s. Seamicro, the pioneer of micro servers, clearly indicated that there was little interest in the low power Atom based systems, but that sales spiked once they integrated Xeon E3s.

Currently micro servers are still a niche market. But micro servers are definitely not hype; they are here to stay, although we don't think they will be as dominant as rack servers or even blade servers in the near future. To understand why we would make such a bold statement, it is important to understand the real reason why micro servers exist.

Let us go back to the past decade (2005-2010). Virtualization was (and is) embraced as the best way to make enterprises with many heterogeneous applications running on underutilized servers more efficient. RAM capacity and core counts shot up. Networking and storage lagged but caught up – more or less – as flash storage, 10 Gbit Ethernet, and SRIOV became available. But the trend to notice was that virtualization made servers more I/O feature rich: the number and speed of network NICs and PCI-e expansion slots for storage increased quickly. Servers based on the Xeon E5 and Opterons have become "software defined datacenters in a box" with virtual switching and storage. The main driver for buying complex servers with high processor counts and more I/O devices is simple: professionals want the benefits that highly integrated virtualization software brings. Faster provisioning, high availability (HA), live migration (vMotion), disaster recovery (DR), keeping old services alive (running on Windows 2000 for example): virtualization made everything so much easier.

But what if you did not need those features because your application is spread among many servers and can take a few hardware outages? What if you do not need the complex hardware sharing features such as SRIOV and VT-d? The prime example is an application like Facebook, but quite a few smaller web farms are in a similar situation. If you do not need the features that come with enterprise virtualization software, you are just adding complexity and (consultancy/training) costs to your infrastructure.

Unfortunately, as always, the industry analysts came with unrealistic high predictions for the new micro server market: in 2016, they would be 10% of the market, no less than "a 50 fold jump"! The simple truth is that there is a lot of demand for "non-virtualized" servers, but they do not all have to be as dense and low power as the micro servers inside the Boston Viridis. The "very low power", extremely dense micro servers with their very low power SoCs are not a good match for most workloads out there, with the exception of some storage and memcached machines. But there is a much larger market for servers denser than the current rack servers, but less complex and cheaper than the current blade servers, and there's a demand for systems with a relatively strong SoC, currently the SoCs with a TDP in the 20W-80W range.

Not convinced? ARM and the ARM licensees are. The first thing that Lakshmi Mandyam, the director of ARM servers systems at ARM, emphasized when we talked to her is that ARM servers will be targeting scale-out servers, not just micro servers. The big difference is that micro servers are using (very) low power CPUs, while scale-out servers are just servers that can run lots and lots of threads in parallel.

The Current Intel Offerings
POST A COMMENT

78 Comments

View All Comments

  • esterhasz - Thursday, December 18, 2014 - link

    But this is exactly why a wider array of machines based on their chips would make sense: the R&D cost is already spent anyways, since iPhone and iPad need chips, selling more units thus reduces R&D cost per unit. Economies of scale.

    I don't believe a MBA variant with ARM is down the road either, but the rumored iPad Pro could develop into something similar rather quickly.
    Reply
  • OreoCookie - Tuesday, December 16, 2014 - link

    If you want to talk about ARM on the desktop, that's a whole other discussion, but one that most certainly needs to include price: if the price difference between a Broadwell-based Core M and a fictitious Apple A9X is $200~$230, then this changes the discussion completely. Two other factors are graphics performance (the Core M has »only« 1.3 billion transistors, the A8X ~2 billion, indicating that the mythical A9X may have faster graphics) and the fact that Apple controls the release schedule and can spec the SoC to meet its projected needs. To view this topic solely through the lens of CPU performance is myopic. Reply
  • darkich - Friday, December 19, 2014 - link

    Your comparisons missed the picture spectacularly.
    A8X is a 20nm 2-4W TDP chip with a price that is probably around 70$.
    Top of the line Core M5Y70 is a 14nm 4.5 W TDP chip with a price of 270$.
    And it has a weaker GPU, btw. (raw performance). And it throttles massively, effectively giving only 50% of the benchmark performance.

    If you're going to compare that to an Apple chip, compare it to a 14nm A9X with custom derived PowerVR series 7 GPU,(scales up to 1,4 TFLOPS) vastly expanded memory controllers connected to a much faster RAM (compared to one in the iPad) upclocked to 2GHz, that are available at any time.
    Reply
  • darkich - Friday, December 19, 2014 - link

    .. *with cores upclocked to about 2GHz Reply
  • Flunk - Tuesday, December 16, 2014 - link

    Nintendo already sells ARM systems, the 3DS and the DS before it are both ARM-based. The PSVita is ARM too. I don't see an ARM Macbook Air anytime soon, they need a bigger and higher-clocking chip for that and it doesn't look like that's going to happen anytime soon. Reply
  • Nintendo Maniac 64 - Tuesday, December 16, 2014 - link

    Even the Game Boy Advance used an ARM7 for its main CPU. Reply
  • jjj - Tuesday, December 16, 2014 - link

    Obviously there are handhelds using ARM but the point was about bigger cores and clearly not handhelds. Reply
  • DLoweinc - Tuesday, December 16, 2014 - link

    Don't quote Wikipedia, not suitable for this level of writing. Reply
  • garbagedisposal - Tuesday, December 16, 2014 - link

    Says DLoweinc, master of knowledge and scholarly writing.
    In contrast to your childish and outdated opinion, Wikipedia is a perfectly valid source of information, go read about it and quit crying.
    Reply
  • Daniel Egger - Tuesday, December 16, 2014 - link

    The problem really is the custom solutions can simply not compete with Intel on any level for general purpose computing (which the majority of applications are), not on performace/price, performance/power and not even on features/price.

    For instance I can see a huge market for sub-Xeon (or Atom C) performance at a corresponding price -> not going to happen because everyone is targeting > Xeon performance at ridiculous prices because they're expecting the margin to be there however there're simply to many compromises to be made by the buyers so that has to fail.

    Also I can see a huge demand for Atom C - Xeon performance at lower power consumption however no one seems to be really targetting this, all we get are Raspberry Pi's and a bit beefier but close from even Atom C. The new virtualisation techniques (Docker et al) opened a whole new can of possibilities for non-x86(_64) devices because virtualisation is suddenly possible and much more lightweight than ever before but no one seems to want to jump this opportunity.

    I'd really like to buy some affordable general purpose (BYOM/BYOS) hardware which has a little bit of oomph and takes little power which should be the powerful sides of any of the contenders but somehow all fail to deliver and I don't even see an attempt to change that.

    If I want mind-boggling performance at decent performance/price ratio with real virtualisation and 100% standard software compatibility there's no way around the high end Xeons (and maybe AMD iff they manage to get their asses back up) and none of the contenders is ever going to challenge that so they might as well stop trying.
    Reply

Log in

Don't have an account? Sign up now