Bulldozer for Servers: Testing AMD's "Interlagos" Opteron 6200 Series
by Johan De Gelas on November 15, 2011 5:09 PM ESTPower Management in Windows Server 2008 SP2
Enabling the C-states in ESX 5i might bring the Opteron 6276 an improved performance/watt ratio. The question is whether the low power consumption at light loads will negate the performance impact. Although power consumption is lowered by using the "C-state enable" tweak, it is not spectacular: 10% lower energy consumption in idle will probably not give the Opteron 6276 an amazing performance/watt ration in ESXi. The impact of this tweak will make a difference in our EWL testing, not in the "full speed ahead" benchmarks. Also, our vApus FOS EWL testing showed that the Xeon consumed 25% less energy, so it will remain ahead.
As the virtualization benchmarks require more time to run, we will have to delay investigating them for a later article. But what about Windows 2008 R2? The idle power of the Opteron 6276 was excellent there. So which power policy should be chosen in Windows 2008? We compared Opteron performance in "High performance" to the Opteron 6276 performance when the power management policy was set to "Balanced.
|
Opteron 6276 "High Performance" |
Opteron 6276 "High Performance" + C6 enable. |
Xeon X5670 "High performance" vs. Xeon X5670 "Balanced" |
|
| Cinebench Single-threaded | +16% | +18% | +1% |
| Cinebench Multi-threaded | +5% | +5% | +1% |
| Blender | +4% | +13% | +1% |
| Encryption/Decryption AES | +43% / +42% | +43% / +44% | +28% / +28% |
| Encryption/Decryption Twofish/Serpent | +8% / +8% | +8 / +8% | +0 / +0% |
| Compression/decompression | +9% / +4% | +9 / +4% | +0 / +2% |
If we combine the our idle power consumption measurements with these numbers, things get a lot clearer. The "balanced" power policy disables turbo. Therefore, the maximum performance boost from enabling "high performance" should be 13%. The TrueCrypt benchmarks show much larger increases (see (*)), which we honestly don't understand. The performance boost (40%) is only possible if the CPU boosts to 3.2GHz, but that is not supposed to happen. First, the TrueCrypt software is well threaded and uses all clusters (32 threads). Second, we disabled C6, so normally the CPU is not able to boost to 3.2GHz. Third, our monitoring clearly indicated a 2.6GHz clock as expected.
We also did a quick x264 4.0 benchmark (1st pass) which is lightly threaded and showed the same performance (46%!) increase by simply switching from "Balanced" to "High performance" (turbo limited to 2.6GHz, no C6). The Xeon only got a 13% increase in performance..
Closer monitoring reveals that "Balanced" frequently reduces the cores to 1.4GHz. So we have a similar situation as the one where we found power management problems on the AMD "Istanbul" Opteron when the power policy was set to "Balanced".
Basically "Balanced" brings the clock speed down to a low P-state even when a thread is demanding the maximum processing power. Or in other words, the power manager is too eager to bring the clock speed down instead of looking ahead: the polling is blind for the very near future. The result is that quite often the workload gets processed at 1.4GHz (for a short time).
In contrast, the high performance setting does not make use of frequency scaling besides Turbo. So the CPU runs at 2.3GHz at the very minimum and frequently reaches 2.6GHz. So if you buy an Opteron 6200 server, it is strongly advised to chose the "High Performance" setting. Under light load, the balanced power manager saves a few percentage of power running idle, but in our opinion, it is not worth the large performance degradation. Notice also that the Xeon hardly suffers from the same problem with the exception of the AES-NI enabled TrueCrypt bench, and even then the performance impact is significantly lower.
In a nutshell: the power policy "Balanced" strongly favors the Xeon as the performance impact is non-existent or much lower. Let us see some raw performance numbers.
Facebook
Twitter
RSS
106 Comments
View All Comments
geoxx - Friday, December 9, 2011 - link
Sorry but neotiger is totally right, choice of benchmark sucks. We are not helped *at all* by your review.What company 32-core server is being used for 3D rendering, cinebench, file compression, truecrypt encryption??
You benchmarked it like it was a CPU of the nineties for a home enthusiast.
You are probably right pointing us to http://www.anandtech.com/show/2694 but your benchmarks don't reflect that AT ALL. Where are file compression, encryption, 3D rendering and cinebench in that chart?
Even performances per watt is not very meaningful because when one purchases a 2-socket or 4-socket server, electricity cost is not an issue. Companies want to simplify deployment with such a system, they want this computer to run as fast as a cluster, in order not to be bound to cluster databases which are a PAIN. So people want to see scalability of applications to full core count on this kind of system, not so much performances per watt.
Virtualization is the ONLY senseful benchmark you included.
TPC as suggested is a totally right benchmark, that's the backend and bottleneck for most of the things you see in your charts at http://www.anandtech.com/show/2694 , and objection on storage is nonsense, just fit a database in ramdisk (don't tell me you need a database larger than 64GB for a benchmark), export as block device, then run the test. And/or use one PCI-e based SSD which you certainly have.
http://www.anandtech.com/show/2694 mentions software development: how much effort does it require to set up a linux kernel compile benchmark?
http://www.anandtech.com/show/2694 mentions HPC: can you set up a couple of bioinformatics benchmarks such as BLAST (integer computation, memory compare), GROMACS (matrix FPU computations) and Fluent? Please note that none of your tests includes memory compares and FPU which are VERY IMPORTANT in HPC. Gromacs and fluent would cover the hole. Bioinformatics is THE hpc of nowdays and there are very few websites, if any, which help with the choice of CPUs for HPC computing.
For email servers (37%!) and web servers (14%) also I am sure you can find some benchmarks.
Iketh - Tuesday, November 15, 2011 - link
I'm not sure how the discovery of cores running in their power-saving state for far too long is anything new. My 2600k refuses to ramp up clocks while previewing video in a video editor even though a core is pegged at 100%. If I intervene and force it to 3.4ghz, preview framerate jumps from 8 fps to 16fps.This has been happening for YEARS! My old quad Phenom 2.2ghz did the exact same thing!
It's extremely annoying and pisses me off I can't benefit from the power savings, let alone turbo.
MrSpadge - Tuesday, November 15, 2011 - link
Sounds like you're running linux or some other strange OS, then. Or you may need a bios update. Generally Intel has its power management quit under control. In the AMD camp physical power state switches often take longer than the impatient OS expects, and thus average frequency is hurt. This was pretty bad for Phenom 1.MrS
Iketh - Tuesday, November 15, 2011 - link
win7 home premium x64 and the phenom was with xp 32bit... i haven't found another scenario that causes this, only streaming video that's rendered on-the-flyZoomer - Wednesday, November 16, 2011 - link
You have a 2600k and aren't running it at 4+ GHz?Iketh - Wednesday, November 16, 2011 - link
4.16 @ 1.32v when encoding, 3.02 @ 1.03v for gaming/internethaplo602 - Wednesday, November 16, 2011 - link
you do know that Linux did not have any problems with Phenom I power management unlike Windows ? Same is not with BD. Linux benchmarks look quite different from Windows and the gap is not that dramatic there.BrianTho2010 - Tuesday, November 15, 2011 - link
This whole review, the only thought I have is that there are no sandy bridge chips in it. When SB based Xeon chips come out I bet that Interlagos will be completely dominated.Beenthere - Tuesday, November 15, 2011 - link
Not really. SB chips don't fit in AMD sockets. AMD's installed customer base like the significant performance increase and power savings by just plugging in a new Opteron 6200/4200.C300fans - Tuesday, November 15, 2011 - link
It will. 2x6174 (24 cores) perform quite similar to 2x6274(32 cores). WTF