The Next Generation Open Compute Hardware: Tried and Tested

The Next Generation: Winterfell

With the PSU and hard disks removed from the server, the only items left in a Freedom chassis were the motherboard, fans and one boot drive. When Facebook's engineers put these things together in a smaller form factor, they created Winterfell, a compute node that's similar to a Supermicro twin node, except in ORv1 three nodes can be placed on a shelf. One Winterfell node is 2 OU high, and consists of a modified Windmill motherboard, a bus bar connector, and a midplane connecting the motherboard to the power cabling and fans. The motherboard is equipped with a slot for a PCIe riser card – on which a full size x16 PCIe card and a half size x8 card can be placed – and a x8 PCIe mezzanine connector for network interfaces. Further connectivity options include both a regular SATA and mSATA connector for a boot drive.

Winterfell nodes, fitted here with optional GPU

But Facebook's ever advancing quest for more efficiency found another target. After an ORv1 deployment in Altoona, it became apparent that having three power zones with three bus bars each was capable of delivering far more power than needed, so they took that information and went on to design the successor, the aptly named OpenRack v2. OpenRack v2 only uses two power zones instead of three, and FB's implementation has only one bus bar segment per power zone, bringing further cost reductions (though the PDU built into the rack is still able to power three). The placement of the power shelves relative to the bus bars was given another thought, this time they were put in the middle of a power zone because of the voltage drop in the bus bar when conducting power from the bottom to the top server. ORv2 also allows for some more ToR machinery by increasing the top bay height to 3 OU.

Open Rack v2, with powerzones on the left. ORv2 filled with Cubby chassis on the right. The chassis marked by the orange rectangle fit 12 nodes. (Image Courtesy Facebook)

The change in power distribution resulted in incompatibility with Winterfell, as the bus bars for the edge nodes are now missing, and so Project Cubby saw the light of day. Cubby is very similar to a Supermicro TwinServer chassis, but instead of having PSUs built in, it plugs into the bus bar and wires three internal power receptacles to power each node. The Winterfell design also needed to be shortened depth-wise, so the midplane was removed.

Another cut in upfront costs was realized by using three (2+1 redundancy) 3.3 kW power supplies instead of six per power zone. With the amount of power zones decreased, each of the two power zones can deliver up to 6.3 kW in ORv2. This freed up some space on the power shelf, so the engineers decided to end the separate battery cabinet that was placed next to the racks. The bottom half of the power shelf now contains three Battery Backup Units (BBU). Each BBU comes with a matrix of Li-ion 18650 cells, equal to those found in a Tesla's battery pack, capable of providing 3.6 kW to bridge the power gap until the generators kick in. Each BBU is paired to a PSU, when the PSU drops out the BBU is active until power is restored.

BBU Schematic (Image Courtesy Facebook)

To summarize, by broadening focus to your bog-standard EIA 19" rack and developing a better integration with the servers and battery cabinets, Facebook was able to reduce costs and added capacity for an additional rack per row.