The OpenPOWER Saga Continues: Can You Get POWER Inside 1U?
by Johan De Gelas on February 24, 2017 8:00 AM ESTEnergy Consumption
We know that the POWER8 was not designed to be a performance-per-watt champion. Throughput, single threaded performance, and RAS were the main priorities. However, Tyan does position the GT75 as a virtualization server. In that market, performance-per-watt is important.
We tested the energy consumption of our servers for a one-minute period in several situations. The first one is the point where the tested server performs best in MySQL: the highest throughput just before the response time goes up significantly. Then we look at the point where throughput is the highest (no matter what response time). This is the situation where the CPU is fully loaded.
The final column is calculated by dividing the best throughput by the power usage. We define the "best throughput" as throughput where the balance between throughput and the 95th percentile response time is the best. In other words, beyond that point, throughput increases only slightly (less than 10%), but the response time increases much faster.
| SKU | Server Height | TDP (on paper) spec |
Idle Server W |
MySQL Best Throughput at Lowest Resp. Time (*) (W) |
MySQL Max Throughput (W) |
Transaction /s (**) |
Tr/watt ( = ** / * ) |
| IBM POWER8 8c@2.3 Tyan | 1U | 170 W | 171 | 323 | 330 | 10300 | 32 |
| IBM POWER8 10c@2.9 S812LC | 2U | 190 W | 221 | 259 | 260 | 14482 | 55 |
| Xeon E5-2699 v4 | 2U | 145 W | 67 | 213 | 235 | 18997 | 89 |
| Xeon E5-2640 v4 | 2U | 90 W | 76 | 135 | 145 | 9541 | 71 |
| Xeon E5-2690 v3 | 2U | 135 W | 84 | 249 | 254 | 11741 | 47 |
At idle, both of the POWER8-based servers reduce their clockspeed to 2.06 GHz and power-gate the cores they do not need. However, the Tyan GT75 PSU is probably more efficient in this case, and the GT75 is a less complex server as well. As a result, the idle power is significantly lower than the S812LC. Still, it is nowhere near the Intel Xeons.
Once we test the server under load, the Tyan GT75 demands a lot more power than the S812LC. That might seem contradictory at first sight, as the latter is equipped with more power hungry CPU. The main culprits are the small, extremely high RPM 1U fans inside the Tyan, which have to work hard to keep a 170W CPU cool in such a cramped environment.
Notice how the IPMI software reports 8800 RPM, but in reality the fan is running at a mindboggling 15600 RPM. A total of twelve such fans results in the cooling system as a whole consuming a lot of power.
This kind of "performance first" CPU policy really needs larger fans and more room. Case in point: in a roomier 2U chassis the load power consumption of a POWER8 setup comes very close to the contemporary 22 nm Xeon E5 v3. It will be interesting to see how this works out in the 1.25U high Rackspace BarrelEye.
Intel's "Broadwell-EP" (Xeon E5 v4) wins here by an vast margin. And there is little doubt that the next generation Skylake Xeons will probably do (slightly?) better.
However, don't count IBM and OpenPOWER out yet. First of all, MySQL is better optimized for x86-64 than for POWER8. Since MySQL is the second most popular database engine (and will probably overtake Oracle soon), we feel our choice is justified. However, it is worth mentioning that PostgreSQL (number 4) and MongoDB (5) have been fully optimized for OpenPOWER and show gains of up to 30%. Lastly, IBM's POWER9 should also do quite a bit better as a result of an improved microarchitecture and being baked with a state-of-the art 14 nm SOI process. The 14 nm POWER9 versus the "tweaked 14 nm" Intel Xeon E5 version 5 should prove a very interesting comparison.
Facebook
Twitter
RSS
28 Comments
View All Comments
Zzzoom - Friday, February 24, 2017 - link
"As important as performance per watt is, several markets – HPC, Analytics, and AI chief among them – consider performance the most important metric. Wattage has to be kept under control, but that is it."What a load of garbage.
JohanAnandtech - Saturday, February 25, 2017 - link
And now maybe some arguments that substantiate your opinion?SarahKerrigan - Sunday, February 26, 2017 - link
In HPC specifically, power consumption is a major issue. This was the entire root of the success of the Blue Gene line back in the day, and why NEC is shifting its supercomputing CPUs to progressively more efficient cores instead of higher-performance cores now (SX-9: 102.4GF/core; SX-ACE: 64GF/core.) . HPC is sensitive to running cost, and power dissipation is a critical factor in that.Zzzoom - Monday, February 27, 2017 - link
Go read the 7+ years worth of materials from the EE HPC Working Group.JohanAnandtech - Wednesday, March 1, 2017 - link
In a system with 2-4 GPUs, 512 GB of RAM, the TDP of the CPU is not a dealbreaker. I can agree that some HPC markets are more sensitive to perf/watt; but I have seen a lot of examples where raw performance per dollar was just as important.Zzzoom - Wednesday, March 1, 2017 - link
POWER8 TDP is 45W-102W higher per socket than the highest spec Xeon E5. That's 90W-204W higher per node where each node consumes 1500W-2000W, or 6-10% total on a site with a multi-million dollar power bill that went to great lengths to bring down the PUE by a similar amount. So for anyone to pick POWER8 it has to do better on energy to solution through its unique features, or be considerably cheaper (ha!). POWER8's advantage is NVLink, but TSUBAME3 going with Intel+PLX switches on top of NVLink shows that it's not that big of a deal.Anyway, the efficiency requirements on the CORAL procurements are pretty strict so scale-out POWER9+Volta will have to shed a lot of weight.
Zzzoom - Wednesday, March 1, 2017 - link
I forgot about the memory buffers. It's even worse.mystic-pokemon - Sunday, March 5, 2017 - link
Guys, I know shit ton of stuff about a server Johan listed above. He has a point when he says Power consumption is only so much important.In short, when you combine all aspects to TCO model: POWER8 server delivers most optimal TCO value
We consider all the following into our TCO model
a) Cost of ownership of the server
b) Warranty (Lesser than conventional server, different model of operations)
c) What it delivers (How many independent threads (SMT8 on POWER8 remember ? 192 hardware threads), how much Memory Bandwidth (230 GBPs), how much total memory capacity in 1 server ( 1 TB with 32 GB)
d) For a public cloud use-case, how many VMs (with x HW threads and x memory cap / bw ) can you deliver on 1 POWER8 server compared to other servers in fleet today ? Based on above stats, a lot .
e) Data center floor lease cost in DC ( 24 of these servers in 1 Rack, much denser. Average the lease over age of server: 3 years ). This includes all DC services like aggers, connectivity and such.
f) Cost per KWH in the specific DC ( 1 Rack has nominal power 750W)
All this combined POWER has good TCO. Its a massively parallel server, what where major advantage comes from. Choose your workload wisely. That's why companies continue to work on it.
I am talking about all this without actually combining with CAPI over PCIe and openCAPI. Get it ? POWER is going no where.
Michael Bay - Friday, February 24, 2017 - link
I think at this point in time intel has more to fear from goddamn ARM than IBM in server space.Okay, maybe AMD as well.
JohanAnandtech - Friday, February 24, 2017 - link
Personally I think OpenPOWER is a viable competitor, but in the right niches (In memory databases, GPU accelerated + NVlink HPC). Just don't put that MHz beast in a far too small 1U cage. :-)