The Next Generation Open Compute Hardware: Tried and Tested
by Johan De Gelas & Wannes De Smet on April 28, 2015 12:00 PM ESTThe Latest and Greatest: Leopard
Leopard, the latest update to the Windmill motherboard, is equipped with the Intel C226 chipset to support up to two E5-2600v3 Haswell Xeons.
One processor mode is fully supported, in which the CPU can access all RAM onboard. Increased thermal margins (mainly because of upping the chassis height to 2 OU in Winterfell), bigger CPU heatsinks, and better airflow guidance allow the system to receive CPUs with a maximum TDP of 145 Watt, which means you can insert every Xeon except for E5-2687W v3 (160W TDP). Only eight DIMM channels are connected per CPU, but DDR4 allows for a maximum capacity of 128GB per DIMM resulting in a theoretical maximum of 2TB RAM, which Facebook reckons is plenty for years to come. New in this generation is that you can now plug NVDIMM modules (persistent flash storage on a DIMM form factor), which Facebook is testing to see if it can replace PCIe-based add-in cards.
Besides the generational CPU update, other major changes include the removal of the onboard external PCIe connector, support for a mezzanine card with dual QSFP receptacles, a TPM header, the addiction of an mSATA/M.2 slot for SATA/NVMe based storage, and 8 more PCIe lanes routed to the riser card slot for a total of 24. The SAS connector has been removed, as Leopard will not be used as a head node for Knox.
Leopards, with the optional debug board (power/reset buttons and serial-to-USB) plugged in
A big addition to the board is a baseboard management controller (BMC). A simple headless Aspeed AST1250 controller provides traditional Out Of Band IPMI access to query sensor and FRU data, control system power and provide Serial Over Lan. But Facebook taught it some new tricks: to aid bare-metal debugging, it keeps 256 post codes in buffer, offers 128kB of serial console output, and you are able to remotely dump MSR data, which is done automatically after the IERR/MCERR signal is active.
A rather unique feature of the BMC is that it allows you to update the CPLD, VR, BMC and UEFI firmware (basically all the firmware present on the motherboard) remotely, a feature also fully validated by all suppliers of the mentioned components. Another feature that's been added is average power reporting, the BMC keeps a buffer of 600 power measurements, and permits you to query the buffer for a specific interval via IPMI. To improve the accuracy of the power sensor data, factory determined (non)-linear compensations are applied to the measured power usage. Lastly, another unique feature that stems from better rack-level integration is the ability to throttle CPU power usage when power demand in the power zone exceeds capacity – for instance when a PSU dies. When the load increases to the PSU capacity, it executes a quick temporary drop to 1 Volt. This triggers an 'Under Voltage' condition in the servers which in turns activates the Fast Proc Hot signal on the CPUs, causing them to clock down for a certain amount of time and thus decreasing PSU load, allowing it to remain active instead of shutting down.
Facebook
Twitter
RSS
26 Comments
View All Comments
SuperVeloce - Wednesday, April 29, 2015 - link
From Mass storage: "Compared to hard disks optical media touts greater reliability, with Blu-ray discs having a life expectancy of 50 years and some discs could even be able to live on for a century."Yeah sure. Like my expensive gold color cd's from different vendors, baked on different high quality writers, now mostly not working anymore after some 15-20 years. Despite being held in almost perfect environment all these years
Uplink10 - Wednesday, April 29, 2015 - link
Someday they are going to figure out that:-SAS HDDs are costlier but if you are using RAID it does not matter, they should use consumer drives and not overpriced enterprise drives
-I calculated sometimes back if Bluray cold storage is cheaper than HDDs but it is not and more so you cannot change the data once you write it, it is better to go with HDDs
toyotabedzrock - Wednesday, April 29, 2015 - link
You have to wonder what these networking chip vendors are hiding in the firmware that makes them so resistant to open sourcing the code.Casper42 - Monday, May 4, 2015 - link
Johan, some of the HP info at the end was interesting, but incomplete.If you (or anyone reading this) plan to talk to HP, they will also talk about their relatively new CloudLine "CL" type machines as well.
They come in standard 1RU/2RU designs as well as OpenRack designs coming soon.
And the SL line is all being morphed over to Project Apollo which uses the XL prefix.
Apollo 2500 is now live, 4X00 will replace SL4500, 6000 has already replaced S6500, and the 8000 was a net-new add for Gen9 focused on big HPC farms.
So anything SL is, or soon will be, a dead platform. (The SLs you mention could be an exception since they are not widely commercially available)
Netpower - Tuesday, June 2, 2015 - link
One general problem with this design is how to take care of power line disturbances entering the power shelves via the 277V AC lines. The 48V DC is filtered via the 48V battery but you must add a filter/power line conditioner somewhere to make sure that transients and sags doesn't kill your power shelves. The 380V DC approach by (http://www.emergealliance.org) is much more reliable and still have all the advantages with higher efficiency, lower cable losses etc.Astana - Wednesday, September 16, 2020 - link
It is believed that the nicotine addiction of a smoker prevents you from quitting. But modern research has shown that the quitting process is complicated by psychological addiction. That's why many people can't cope with the habit of smoking. But there are results with https://heroindetoxeurope.com/ drvorobjev hospital they're all successful.The problem of smoking is not only the problem of men, but also increasingly - women. Many women are wondering: "How can you quit smoking and not get better? Psychologists have proven that when a woman quits smoking, she subconsciously tries to replace, to compensate for the process of smoking with food. This feeling is also familiar to men who quit smoking, but they do not attach much importance to it. As a result, very soon an extra 3-4 kg or more will appear.