Low Power Server CPUs: the energy saving choice?
by Johan De Gelas on July 15, 2010 4:54 AM EST- Posted in
- IT Computing
Translated to the datacenter
Before we make any conclusions based upon what we learned in this testing session, we should not forget the lessons learned from previous experiments. And those tests tell us that you should avoid the low end server CPUs: those are the most leaky ones, consuming more power at idle and low load.
Our conclusion on our new energy measurement methodology is:
1. Low power Xeons save power but do not save energy in a typical Hyper-v consolidation ratio. Power is “capped”, but the total energy consumed for a certain task is (more or less) the same.
2. X-series Xeons offer a much better performance per watt ratio, but at the expense of brief power peaks. They do not necessarily need more energy in the long run than the lower power versions, and offer much better response times if your application is CPU bound.
So if you have to pay the actual energy consumption, you have some amp headroom left before being penalized and performance matters to your users, the faster Xeons are the right choice. The benefit is that you can offer a lower response times to your users, even when the CPU is not running at peak load! The Xeon X5670 is more flexible.
In case you pay a fixed price for a fixed amount of amps and you get heavy fines in your mailbox if you briefly breach your amp limit, buying lower power Xeons is probably the way to go.
Then again, we are not very enthusiastic about power capping at the server level. Many companies have already embraced the idea of a Dynamic Scheduled Virtualized Cluster. VMware's vSphere product is the most mature here: it is pretty easy to build a virtual cluster with dynamic power management (DPM) and scheduling (DRS). We still have to investigate this, but if this virtual cluster works well with solutions such as HP Insight Power Manager or Intel’s own power node manager, the faster Xeon will get interesting once again. The basic idea is that you should power cap your entire cluster (or rack), not one server. You should not care that one server needs a little more power than usual, but the whole cluster should not exceed the amp limits described in your contract with the datacenter. That way you can reconcile low response times with low power bills.
A big thanks to Dries Velle for assisting us in the Sizing Servers Lab.
Facebook
Twitter
RSS
49 Comments
View All Comments
WillR - Thursday, July 15, 2010 - link
10p per kWh may be low in the UK but it's not for residential in the US. $.12/kWh is average. Just pulled out a bill from earlier this year and we paid 7.3 cents per kWh at my house. What it comes down to is the data center potentially overcharges people in the, using your numbers, 19 to 38 cents per kWh range, but rates can be higher than $.20/kWh in high density areas like NYC or SF. The extra costs should go to paying for upgrades and expansion of their infrastructure so it's not unreasonable.Worth mentioning to put in perspective is 4 250 watt servers uses 720kWh/month and the average house in the US uses 920kWh/month, so it's not really as simple a setup as one might initially think.
http://www.eia.doe.gov/cneaf/electricity/esr/table... provides a nice table of average rates and usages.
knedle - Thursday, July 15, 2010 - link
I'm not sure if you're aware, but in most countries residential has much (at least twice) lower price per 1kWh, than commercial. Also commercial pays extra for using electricity during the day, and gets electricity cheaper during night.This is why there are some factories in Europe that work only during night.
WillR - Thursday, July 15, 2010 - link
That is not the case in the US. Residential pays higher rates than either Commercial or Industrial users.http://www.eia.doe.gov/cneaf/electricity/epm/table...
Average Retail Price (Cents/kWh)
Items Mar-10 Mar-09
Residential 11.2 11.33
Commercial 10.03 10.07
Industrial 6.5 6.79
Industrial settings tend to use very large amounts of energy in a very small area or number of clients so they get cheaper bulk rates for purchasing a lot with little administrative overhead. It's also often the case they can get a high voltage line installed directly to the plant which is expensive to install but increases efficiency dramatically.
These averages may reflect heavy use of off-peak consumption, but most plants I've experienced operate 24/7. Much of it is politics and bargaining for a better rate on the contract.
DaveSylvia - Thursday, July 15, 2010 - link
Hey Johan, great article as usual! Always enjoyed and appreciated your articles including those from back in the day at Ace's Hardware!JohanAnandtech - Thursday, July 15, 2010 - link
Good memory :-). I have been part of Anand's team for 6 years now, that is the same amount of time that I spend at Ace's.DaveSylvia - Thursday, July 15, 2010 - link
Yeah! One of the first tech articles I recall reading was back in 1999. It was about how pipeline length influenced CPU clock speeds. You used Pentium II, K6, DEC Alpha's as examples :). All good stuff!MrSpadge - Thursday, July 15, 2010 - link
Isn't there a power plan which lets the CPUs turbo up (as max performance does) and also lets them clock down if not needed (as balanced does)? It seems outright stupid to take turbo away from a Nehalem-like chip.
MrS
has407 - Thursday, July 15, 2010 - link
No reason you shouldn't be able to specify a power plan that does both, but for whatever reason it isn't provided out-of-the-box.I'd guess that given the relatively small difference in idle power between "performance" and "balanced" (which seems to be more of "power capped" plan), maybe they (presumably the OEM?) decided it wasn't worth it.
There may also be stability issues with some system configurations or support concerns, as there's yet another set of variables to deal with.
has407 - Thursday, July 15, 2010 - link
Johan -- That brings up an interesting question: How much of the underlying CPU's power management are you testing vs. a particular vendor or OS configuration? I'd expect them to closely reflect each other assuming everyone has their job.As you're using Win 2008, it would be interesting to see what powercfg.exe shows for the various parameters for different modes and systems; e.g., "Busy Adjust Threshold", "Increase Policy", "Time Check", "Increase Percent", "Domain Accounting Policy" etc. Are there significant differences across systems/CPUs for the same profile?
Whizzard9992 - Thursday, July 15, 2010 - link
Heat dissipation is also a concern, no? It's expensive to cool a datacenter. Low power should bring cooling costs down.There's also a question of density. You can fit more low-power cores into 1U of space because of the heat dissipation. Multi-node blades are cheaper than 2U workhorses. Rack space is expensive for a lot of reasons. Just look at the Atom HPC servers: I bet the Atom would score pretty low in performance-per-watt versus even the LP XEON, but its sheer size and thermal envelope fit it in places the XEON can't.
Frankly, I'd be surprised if the low-power XEON saved "energy" at the same workloads versus full-power, given that both are on the same architecture. LP XEONs are really an architecture choice, and greasing the transition to many-cores via horizontal scaling. A good desktop analogy would be, "Is one super-fast core better than a slower multi-core?" Fortunately for the datacenter most servers only need one or the other.
Also, with physical nodes scaling out horizontally, entire nodes can be powered down during down times, with significant power savings. This is software-bound, and I haven't seen this in action yet, but it's a direction nonetheless.
Without getting into all of the details, I think a proper TCO analysis is in order. This article seems to really only touch on the actual power-consumption, where there are really no surprises. The full-power peaks performance a little better, and the LP stays within a tighter thermal-envelope.
The value of low-power is really low-heat in the datacenter. I'd like to see something that covers node density and cooling costs as well. A datacenter with all LP-servers is unrealistic, seeing as how some applications that scale vertically will dictate higher-performing processors. It would be nice to see what the total cost would be for, say a 2,000 node data center with 80% LP population versus 20% LP population. The TDP suggests a 1/3 drop in cooling costs and 1/3 better density.