HPC: OpenFoam

Computational Fluid Dynamics is a very important part of the HPC world. Several readers told us that we should look into OpenFoam, and my lab was able to work with the professionals of Actiflow. Actiflow specializes in combining aerodynamics and product design. Calculating aerodynamics involves the use of CFD software, and Actiflow uses OpenFoam to accomplish this. To give you an idea what these skilled engineers can do, they worked with Ferrari to improve the underbody airflow of the Ferrari 599 and increase its downforce.

We were allowed to use one of their test cases as a benchmark, but we are not allowed to discuss the specific solver. All tests were done on OpenFoam 2.2.1 and openmpi-1.6.3.

Many CFD calculations do not scale well on clusters, that is unless you use InfiniBand. InfiniBand switches are quite expensive and even then there are limits to scaling. We do not have an InfiniBand switch in the lab, unfortunately. Although it's not as low latency as InfiniBand, we do have a good 10G Ethernet infrastructure, which performs rather well. So we can compare our newest Xeon server with a basic cluster.

We also found AVX code inside OpenFoam 2.2.1, so we assume that this is one of the cases where AVX improves FP performance.

OpenFoam Benchmark

Unless you recompile and tune your code for AVX2, the new Xeon E7 v3 is hardly faster than the previous one. The reason may be that the new Xeon can sometimes go as low as 2.1 GHz running AVX code due to the immense power load AVX2 workloads can cause, while the older E7 v2 is capable of sustaining 2.8 GHz.

Linux Kernel Compile HPC Watts per Job
Comments Locked

146 Comments

View All Comments

  • DanNeely - Friday, May 8, 2015 - link

    Intel's 94% market share is still only ~184k systems. That's tiny compared to the mainstream x86 market; and doesn't give a lot of (budgetary) room to make radical changes to CPU vs just scaling shared designs to a huger layout.
  • theeldest - Friday, May 8, 2015 - link

    184k for 4S systems. The number of 2S systems *greatly* outnumbers the 184k.
  • Samus - Sunday, May 10, 2015 - link

    by 100 orders of magnitude, easily.

    2S systems are everywhere these days, I picked up a Lenovo 2S Xeon system for $600 NEW (driveless, 4GB RAM) from CDW.

    4S, on the other hand, is considerably more rare and starts at many thousands, even with 1 CPU included.
  • erple2 - Sunday, May 10, 2015 - link

    Well, maybe 2 orders of magnitude. 100 orders of magnitude would imply, based on the 184k 4S systems, more 2S systems than atoms in the universe. Ok, I made that up, I don't know how many atoms are in the universe, but 10^100 is a really big number. Well, 10^105, if we assume 184k 4S systems.

    I think you meant 2 orders of magnitude.
  • mapesdhs - Sunday, May 10, 2015 - link

    Yeah, that made me smile too, but we know what he meant. ;)
  • evolucion8 - Monday, May 11, 2015 - link

    That would be right if Intel cores are wide enough which aren't compared to IBM. For example, according to this review, enabling two way SMT boosted the performace to 45% and adding two more threads added 30% more performance. On the other hand, enabling two way SMT on the latest i7 architecture can only go up to 30% on the best case scenario.
  • chris471 - Friday, May 8, 2015 - link

    Great article, and I'm looking forward to see more Power systems.

    I would have loved to see additional benchmarks with gcc flags -march=native -Ofast. Should not change stream triad results, but I think 7zip might profit more on Power than on Xeon. Most software is not affected by the implied -ffast-math.
  • close - Friday, May 8, 2015 - link

    It reminds me of the time when Apple gave up on PowerPC in mobiles because the new G5s were absolute power guzzlers and made space heaters jealous. And then gave up completely and switched to Intel because the 2 dual core PowerPC 970MP CPUs at 2.5GHz managed to pull 250W of power and needed liquid cooling to be manageable.

    IBM is learning nothing from past mistakes. They couldn't adapt to what the market wanted and the more nimble competition was delivering 25-30 years ago when fighting Microsoft, it already lost business to Intel (which is actually only nimble by comparison), and it's still doing business and building hardware like we're back in the '70s mainframe age.
  • name99 - Friday, May 8, 2015 - link

    You are assuming that the markets IBM sells into care about the things you appear to care about (in particular CPU performance per watt). This is a VERY dubious assumption.
    The HPC users MAY care (but I'd need to see evidence of that). For the business users, the cost of the software running on these systems dwarfs the lifetime cost of their electricity.
  • SuperVeloce - Saturday, May 9, 2015 - link

    They surely care. Why wouldn't they. A whole server rack or many of them in fact do use quite a bit of power. And cooling the server room is very expensive.

Log in

Don't have an account? Sign up now