Cineca’s Tesla K20-Based “Eurora” Supercomputer Unveiled; Water Cooling Unlocks Extra Efficiency
by Ryan Smith on January 31, 2013 7:40 PM ESTWe typically don’t cover a lot of supercomputing news outside of the major Top500/Green500 announcements due to the fact that the launches of so many supercomputers are clustered around the official updates to those lists, but the following came across our desk earlier this afternoon from NVIDIA and caught our attention.
Italy’s Cineca computing center has unveiled their latest supercomputer today, a Eurotech created system built on a combination of Intel’s Xeon E5 CPUs and NVIDIA’s Tesla K20 GPUs. While the supercomputer itself is still considered a prototype and is fairly modest overall – its 110 TFLOPS would be good for roughly #250 on the most recent Top500 list – what makes this latest supercomputer particularly interesting (and just a bit snark-worthy) is not its peak performance, but rather its energy efficiency and cooling mechanism. In short, someone has finally paired a K20-based supercomputer with water cooling, with some interesting results.
Cineca's and Eurotech's Water Cooled Eurora Supercomputer
By using water cooling, Eurotech and Cineca have been able to capture the waste heat off of Eurora in a far more dense and more useful manner than air would allow, in turn allowing them to put that waste heat to productive use. By using that heat to not only heat their buildings but also to drive absorption chillers, they’ve been able to reduce the amount of heat that ultimately needs to be removed from the system, and consequently the amount of energy that is spent on cooling.
Adding an extra facet to this is the fact that due to the much greater heat capacity of water over air, using water for cooling – even warm-to-hot water as in the case of Eurora – results in lower processor temperatures than chilled air. In turn, thanks to the positive relationship in semiconductors between temperature and power consumption, these lower temperature processors consume less electricity and thereby produce less heat in the first place.
The end result is that between the use the water cooling and various other optimizations made by Eurotech, they’ve been able to produce a supercomputer significantly more power efficient than other K20 supercomputers, or any other supercomputer for that matter. At 110 TFLOPS performance for 34.7KW electricity consumption, Eurora has an energy efficiency of 3,150 MFLOPS/watt, almost 25% better than the current #1 system on the Green500 supercomputer list, a Xeon Phi system that operates at 2,500 MFLOPS/watt. This makes Eurora the most energy efficient supercomputer available by a wide margin for the moment, and while the next Green500 list won’t be published until June, if Eurora is accepted then will be a likely contender for the top spot on that list.
We won’t dwell on this too much more, but if you would like some more details there’s an interesting slide deck from Eurotech published over on NVIDIA’s website. Ultimately using waste heat for productive use is something of a one-trick pony – there aren’t too many productive ways to use that heat due to its relatively low temperature – but nevertheless it offers an interesting alternative into improving supercomputer performance within a fixed power envelope, particularly at a time where we’re beginning to see diminishing returns from smaller semiconductor manufacturing processes.
Source: NVIDIA
Facebook
Twitter
RSS
17 Comments
View All Comments
xxtypersxx - Thursday, January 31, 2013 - link
As someone who has watercooled every computer I've built for the last 5 years, its nice to see commercial applications begin to follow down that road. Those who haven't tried it would be astonished at the low voltage overclocks you get when the gpu is peaking under 50C.Plifzig - Friday, February 1, 2013 - link
When you watercool you put the GPU and CPU in the loop? I've never pulled any covers/heatspreaders/anything off of a GPU and it just sounds stressful (to my nerves). I've wondered about watercooling, but haven't taken it for more than appreciation. Maybe my next build will use it.Also, what's your ballpark estimation of watercooled CPU vs. watercooled CPU+GPU ratio? 20/80? 60/40? Just curious.
kyp275 - Friday, February 1, 2013 - link
It's actually pretty simple really, just a few screws and off the entire shroud goes. The main issues for those new to watercooling is probably getting used to be leak tests/extra maintenance.Not sure what you mean by the ratios, elaborate?
Plifzig - Friday, February 1, 2013 - link
Thanks for the info and sorry for the convoluted question.I was just wondering if people use watercooling on only the CPU and leave the GPU untouched. The ratio was the mix of CPU only cooled builds to CPU+GPU cooled builds.
A better way to ask would just be: what percentage of watercooled builds put CPU+GPU in the loop?
It was a long day so sorry for the confusion :)
Death666Angel - Friday, February 1, 2013 - link
The forum I frequent for my water cooling needs have most people there running at least CPU and GPU custom rigs (the more active people also have RAM and mainboard, the more extreme of those HDDs and some even the PSU). And if a newbie comes there and asks for advice on water cooling his CPU, most of the people there will say that he shouldn't do it CPU-only and use existing high end air coolers or CLCs, because they work just as well just for the CPU. If you want to go custom water cooling, use it for the GPU mostly (that is the 200+W part of the PC generating most of the noise) and since the CPU part is pretty cheap and you already have all the equipment apart from the CPU heatsink, add that to the mix for best operation.So I would say a maximum of 10-20% of the custom water cooling rigs are CPU only (with half of those looking to upgrade to CPU+GPU when money and time allows). 80-90% then is at least CPU+GPU.
Sabresiberian - Saturday, February 2, 2013 - link
+1 for a very informative reply!kyp275 - Friday, February 1, 2013 - link
Pretty much what Death666 said. Most people who would go through the effort of building their own watercooling setup isn't going to leave out the GPU, which tends to be the hottest/loudest part of the PC.Those that would want to cool the CPU only will likely opt for a pre-built closed-loop systems instead.
Death666Angel - Friday, February 1, 2013 - link
There is no need to remove heatspreaders when using most water cooling solutions. Just get rid of the shroud, heatsink and fan.If you water cool using costum solutions, it usually makes little sense to only use GPU or CPU water cooling. The GPU is the main culprit in generating heat and noise in a modern gaming PC (~200 to 300W standard TDP compared to 80-130W standard TDP for CPUs). So you should always use water cooling for the GPU. And since CPU water blocks are usually pretty cheap compared to full cover GPU ones (~30-50€ vs 80-30€), it makes only a small dent in the pocket to include the CPU as well, but it usually offers better temperatures, better noise, better overclocking etc.
And I also don't get the ratio thing.
xxtypersxx - Friday, February 1, 2013 - link
The most recent one I've ran in both a CPU only loop as well as with GPU's is a 2600k@4.6ghz that later got a couple of extra GTX 470's in the loop. This took load cpu temperature from ~60C to ~66C. Water loops self regulate in a way as long as your room temperature stays constant: the hotter the water gets, the higher the temperature delta over ambient becomes and the more efficient the radiator(s) become at transferring extra heat. The real winners in the loop are the 470's... Load temps went from 90C to ~45C (the large area of a GPU die makes it much easier for the waterblock to pull off heat). That system has one 360 and one 120 radiator, no heatspreaders were removed.IanCutress - Friday, February 1, 2013 - link
It would be interesting to see the efficiency or flow rates of the pumps used, or the reservoir/pump combinations along the line. That much water going around a rack or three has got to equal some serious volume which needs to be shifted.