Free Cooling: the Server Side of the Story
by Johan De Gelas on February 11, 2014 7:00 AM EST- Posted in
- Cloud Computing
- IT Computing
- Intel
- Xeon
- Ivy Bridge EP
- Supermicro
Fan Speed
How do these higher temperatures affect the fans?
It is clear that the fan speed algorithm takes more than just the CPU temperature and inlet temperature into account. There's definitely an ability to detect when a low power CPU with low tCase is used. As a result the fans are spinning faster with the 2650L than with the 2697 v2. That also means that the server has more headroom for the Xeon E5-2697 v2 than we first assumed based on the CPU temperature results. At higher inlet temperatures, the fans can still go a bit faster if necessary on the 2697, as the maximum fan RPM is 7000.
Power Measurements
The big question of course is how all this affects the power bill. It's no use saving on cooling if your server simply consumes a lot more power due to increased fan speeds (and potentially down time when replacing fans more frequently).
The difference in power consumed is not large between the three inlet temperatures. To make our measurements clear, we standarized on the measurements at 20°C as the baseline and created the following table:
| Xeon E5-2697 v2 | Xeon E5-2650L | |||||
|---|---|---|---|---|---|---|
| CPU load | Inlet 20°C | Inlet 35°C | Inlet 40°C | Inlet 20°C | Inlet 35°C | Inlet 40°C |
| 0-10 | 100% | 105% | 106% | 100% | 106% | 112% |
| 10-20 | 100% | 98% | 103% | 100% | 104% | 108% |
| 20-30 | 100% | 105% | 110% | 100% | 103% | 107% |
| 30-40 | 100% | 102% | 105% | 100% | 102% | 105% |
| 40-50 | 100% | 109% | 108% | 100% | 97% | 109% |
| 50-60 | 100% | 105% | 108% | 100% | 108% | 111% |
| 60-70 | 100% | 106% | 107% | 100% | 104% | 110% |
| 70-80 | 100% | 106% | 104% | 100% | 105% | 109% |
| 80-90 | N/A | N/A | N/A | 100% | 109% | 108% |
| Average | 105% | 107% | 103% | 109% | ||
As the fans work quite a bit harder to keep the 2650L below the low Tcase, they need a lot more power. We notice a 9% increase in power when the inlet temperature doubles. The increase is smaller with the Xeon E5, only 7%.
The most interesting conclusion is that raising the inlet temperature from 20 to 35°C results in almost no increase in power consumption (3-5%) on the server side, while the savings on cooling and ventilation can be substantial, around 40% or more.
Facebook
Twitter
RSS
48 Comments
View All Comments
ShieTar - Tuesday, February 11, 2014 - link
I think you oversimplify if you just judge the efficiency of the cooling method by the heat capacity of the medium. The medium is not a heat-battery that only absorbs the heat, it is also moved in order to transport energy. And moving air is much easier and much more efficient than moving water.So I think in the case of Finland the driving fact is that they will get Air temperatures of up to 30°C in some summers, but the water temperature at the bottom regions of the gulf of Finland stays below 4°C throughout the year. If you would consider a data center near the river Nile, which is usually just 5°C below air temperature, and frequently warmer than the air at night, then your efficiency equation would look entirely different.
Naturally, building the center in Finland instead of Egypt in the first place is a pretty good decision considering cooling efficiency.
icrf - Tuesday, February 11, 2014 - link
Isn't moving water significantly more efficient than moving air because a significant amount of energy when trying to move air goes to compressing it rather than moving it, where water is largely incompressible?ShieTar - Thursday, February 13, 2014 - link
For the initial acceleration this might be an effect, though energy used for compression isn't necessary lost, as the pressure difference will decay via motion of the air again (but maybe not in the preferred direction. But if you look into the entire equation for a cooling system, the hard part is not getting the medium accelerated, but to keep it moving against the resistance of the coolers, tubes and radiators. And water has much stronger interactions with any reasonably used material (metal, mostly) than air. And you usually run water through smaller and longer tubes than air, which can quickly be moved from the electronics case to a large air vent. Also the viscosity of water itself is significantly higher than that of air, specifically if we are talking about cool water not to far above the freezing point of water, i.e. 5°C to 10°C.easp - Saturday, February 15, 2014 - link
Below Mach 0.3, air flows can be treated as incompressible. I doubt bulk movement of air in datacenters hits 200+ Mphjuhatus - Tuesday, February 11, 2014 - link
Sir, I can assure you the Nordic Sea hits ~20°C in the summers. But still that tempereture is good enough for cooling.In Helsinki they are now collecting the excess heat from data center to warm up the houses in the city area. So that too should be considered. I think many countries could use some "free" heating.
Penti - Tuesday, February 11, 2014 - link
Surface temp does, but below the surface it's cooler. Even in small lakes and rivers, otherwise our drinking water would be unusable and 25°C out of the tap. You would get legionella and stuff then. In Sweden the water is not allowed to be or not considered to be usable over 20 degrees at the inlet or out of the tap for that matter. Lakes, rivers and oceans could keep 2-15°C at the inlet year around here in Scandinavia if the inlet is appropriately placed. Certainly good enough if you allow temps over the old 20-22°C.Guspaz - Tuesday, February 11, 2014 - link
OVH's datacentre here in Montreal cools using a centralized watercooling system and relies on convection to remove the heat from the server stacks, IIRC. They claim a PUE of 1.09iwod - Tuesday, February 11, 2014 - link
Exactly what i was about to post. Why Facebook, Microsoft and even Google didn't manage to outpace them. PUE 1.09 is still as far as i know an Industry record. Correct me if i am wrong.I wonder if they could get it down to 1.05
Flunk - Tuesday, February 11, 2014 - link
This entire idea seems so obvious it's surprising they haven't been doing this the whole time. Oh well, it's hard to beat an idea that cheap and efficient.drexnx - Tuesday, February 11, 2014 - link
there's a lot of work being done on the UPS side of the power consumption coin too - FB uses both Delta DC UPS' that power their equipment directly at DC from the batteries instead of the wasteful invert to 480vac three phase, then rectify again back at the server PSU level, and Eaton equipment with ESS that bypasses the UPS until there's an actual power loss (for about a 10% efficiency pickup when running on mains power)