Energy and HPC

AVX/FP intensive applications are known to be real power hogs. How bad can it get? We used the OpenFOAM test and measured both average and maximum power (the 95th percentile). Average power tells us how much energy will be consumed for each HPC job while maximum power is important as you have to allocate enough amps to your rack to feed your HPC server/cluster.

HPC maximum performance power consumption

This confirms there is more going on than just the fact that our "Wildcat Pass" server consumes more than the Supermicro server in this test. At peak, the Xeon E5-2699 v3 consumes almost 450W (!!) more than at idle. Even if we assume that the fans take 100W, that means that 350W is going to the CPUs. That's around 175W per socket, and even though it's measured at the wall and thus includes the Voltager regulators, that's a lot of power. The Xeon E5-2699 v3 is a massive powerhouse, but it's one that needs a lot of amps to perform its job.

Interestingly, the Xeon E5-2695 v3 is also using more power than all previous Xeons. The contrast with our Drupal power measurements is very telling. In the Drupal test, the CPU was able to let many of the cores sleep a lot of the time. In OpenFOAM, all the cores are working at full bore and the superior power savings of the Haswell cores deep sleep states do not matter much. But which CPU is the winner? To make this more clear, we have to calculate the actual energy consumed (average power x time ran).

Total HPC Energy Consumption per job

When we look at how much energy is consumed to get the job done, the picture changes. The old Xeon "Sandy Bridge EP" is far behind. It is clear that Intel has improved AVX efficiency quite a bit. The low power Xeon E5-2650L v3 is a clear winner. In second place, the fastest Xeon on the planet actually saves energy compared to the older Xeons, as long as you can provide the peak amps.

HPC: OpenFoam LRDIMMs: Capacity and Real World Performance
Comments Locked

85 Comments

View All Comments

  • coburn_c - Monday, September 8, 2014 - link

    MY God - It's full of transistors!
  • Samus - Monday, September 8, 2014 - link

    I wish there were socket 1150 Xeon's in this class. If I could replace my quad core with an Octacore...
  • wireframed - Saturday, September 20, 2014 - link

    If you can afford an 8-core CPU, I'm sure you can afford a S2011 board - it's like 15% of the price of the CPU, so the cost relative to the rest of the platform is negligible. :)
    Also, s1150 is dual-channel only. With that many cores, you'll want more bandwidth.
  • peevee - Wednesday, March 25, 2015 - link

    For many, if not most workloads it will be faster to run 4 fast (4GHz) cores on 4 fast memory channels (DDR4-2400+) than 8 slow (2-3GHz) cores on 2 memory channels. Of course, if your workload consists of a lot of trigonometry (sine/cosine etc), or thread worksets completely fit into 2nd level cache (only 256k!), you may benefit from 8/2 config. But if you have one of those, I am eager to hear what it is.
  • tech6 - Monday, September 8, 2014 - link

    The 18 core SKU is great news for those trying to increase data center density. It should allow VM hosts with 512Gb+ of memory to operate efficiently even under demanding workloads. Given the new DDR4 memory bandwidth gains I wonder if the 18 core dual socket SKUs will make quad socket servers a niche product?
  • Kevin G - Monday, September 8, 2014 - link

    In fairness, quad socket was already a niche market.

    That and there will be quad socket version of these chips: E5-4600v3's.
  • wallysb01 - Monday, September 8, 2014 - link

    My lord. My thought is that this really shows that v3 isn’t the slouch many thought it would be. An added 2 cores over v2 in the same price range and turbo boosting that appears to functioning a little better, plus the clock for clock improvements and move to DDR4 make for a nice step up when all combined.

    I’m surprised Intel went with an 18 core monster, but holy S&%T, if they can squeeze it in and make it function, why not.
  • Samus - Monday, September 8, 2014 - link

    I feel for AMD, this just shows how far ahead Intel is :\
  • Thermogenic - Monday, September 8, 2014 - link

    Intel isn't just ahead - they've already won.
  • olderkid - Monday, September 8, 2014 - link

    AMD saw Intel behind them and they wondered how Intel fell so far back. But really Intel was just lapping them.

Log in

Don't have an account? Sign up now