Facebook's "Open Compute" Server tested
by Johan De Gelas on November 3, 2011 12:00 AM ESTThe Facebook Server
In the basement of the Palo Alto, California headquarters, three Facebook engineers built Facebook's custom-designed servers, power supplies, server racks, and battery backup systems. The Facebook server had to be much cheaper than the average server, as well as more power efficient.
The first change they made was the chassis height, going for a 1.5U high design as a compromise between density and making the server easier to cool. 1.5U allows them to use taller heatsinks, larger (60mm) lower-RPM fans than the screaming 40mm energy hoggers used in a 1U chassis. The result is that the fans consume only 2% to 4% of the total power, which is pretty amazing as we have seen 1U fans that can consume up to one third of the total system power. It seems that air-cooling in the Open Compute 1.5U server is as efficient as the best 3U servers.
At the same time, Facebook Engineering kept the chassis very simple, without any plastic. It makes the airflow through the server smoother and reduces weight. The bottom plate of one server serves as the top plate for the server beneath it.
Facebook has designed an AMD and an Intel motherboard, both manufactured by Quanta. Much attention was paid to the efficiency of the voltage regulators (94% efficiency). The other trick was again to remove anything that was not absolutely necessary. These motherboards have no BMC, very few USB (2) and NIC ports (2), one expansion slot, and are headless (no videochip).
The only thing that an administrator can do remotely is "reboot over LAN". The idea is that if that does not help, the problem is in 99% of cases severe enough that you have to send an administrator to the server anyway.
The AMD servers are mostly used as Memcached servers, as the four channels of AMD Magny-cours Opterons 6100 are capable of using 12 DIMMs per CPU, or 24 DIMMs in total. That works out to 384GB of caching memory.
In contrast the Facebook Open Compute Xeon servers only have six DIMM slots as they are used for processing intensive tasks such as the PHP "assembling" data servers.
Facebook
Twitter
RSS
67 Comments
View All Comments
jhh - Saturday, November 5, 2011 - link
I'm not sure how much of the benchmarks depend on network bandwidth, but Facebook certainly does a lot of it. Using SRIOV based NICs and supporting drivers allows the VM to access virtual NIC hardware directly, without having to go through the hypervisor. But, all NICs aren't built equal, many of them do not support SRIOV, and those that do, may not have drivers which support it in older kernels such as Centos 5.6. Unfortunately, since most Gigabit NICs were designed before SRIOV, most gigabit NICs don't support it. We have great difficulty getting hardware vendors to describe whether the provide SRIOV capable hardware or Linux drivers. The newer 10G NICs tend to support SRIOV, but whether the server needs more than 1G is unclear, and the 10G NICs are more expensive and use more power.CPU-Hog - Sunday, November 6, 2011 - link
Good comparison of the servers however I couldn't help but think how much better it would be if we ran actual workloads that facebook etc plan to run in the datacenter vs. these enterprise workloads. How about running MemcacheD / Hadoop / HipHop etc. which are the key workloads the OpenCompute servers are designed to run well.Many of these workloads need large IO and memory vs. high compute. It will also be interesting to then use the same benchmarks to compare future servers based on technology from newbies like Calxeda, SeaMicro and AppliedMicro.
Xeon and Opterons based servers vs. ARM and Atom based servers. Now that battle of the old guard vs, the upstarts will be worth seeing.
trochevs - Wednesday, November 9, 2011 - link
Johan,Thank you for excellent article. I love to read about cutting edge technology. Keep with the good work. But, I notice something that nobody in the comments has mention yet. In the last paragraph:
"... being inspired by open source software (think ..., ..., iOS, ...)."
iOS is a Open Source Software?! When this happen?
mrgadgetgeek - Thursday, November 10, 2011 - link
Since these systems are custom designed by Facebook engineers, I'm guessing you can't purchase anything like it, correct? Will that change with that foundation that Open Compute announced recently?artemisgoldfish - Thursday, November 10, 2011 - link
Getting Power One to design a supply just right requires a LOT of testing. It's also strange to me that the supply only takes 200-277VAC. The Power One AC supplies I'm familiar with do 90VAC to 264VAC and pass 80PLUS Gold, maybe the tighter input range helps them tune it for more efficiency.ahmetmy330 - Friday, December 30, 2016 - link
you are free of charge to sort any sum of money in you intend to put in your Jurassic World job application. When using the formulated 100 % free Income and Nourishment it will be possible to fundamentally discover and acquiring everything around Jurassic World free of chargehttp://www.h2kjamaica.com/jurassicworldhack/
Chapon788 - Saturday, January 28, 2017 - link
Great Information sharing .. I am very happy to read this article .. thanks for giving us go through info.http://ultimatemobilehits.com/king-of-avalon-hack/