Benchmark Configuration

Since AMD sent us a 1U Supermicro server, we had to resort to testing our 1U servers again. That is why we went back to the ASUS RS700 for the Xeon. It is a bit unfortunate as on average 1U servers have a relatively worse performance/watt ratio than other form factors such as 2U and blades. Of course, 1U still makes sense in low cost, high density HPC environments.

Supermicro A+ server 1022G-URG (1U Chassis)

CPU Two AMD Opteron "Bulldozer" 6276 at 2.3GHz
Two AMD Opteron "Magny-Cours" 6174 at 2.2GHz
RAM 64GB (8x8GB) DDR3-1600 Samsung M393B1K70DH0-CK0
Motherboard SuperMicro H8DGU-F
Internal Disks 2 x Intel SLC X25-E 32GB or
1 x Intel MLC SSD510 120GB
Chipset AMD Chipset SR5670 + SP5100
BIOS version v2.81 (10/28/2011)
PSU SuperMicro PWS-704P-1R 750Watt

The AMD CPUS have four memory channels per CPU. The new Interlagos Bulldozer CPU supports DDR3-1600, and thus our dual CPU configuration gets eight DIMMs for maximum bandwidth.

Asus RS700-E6/RS4 1U Server

CPU Two Intel Xeon X5670 at 2.93GHz - 6 cores
Two Intel Xeon X5650 at 2.66GHz - 6 cores
RAM 48GB (12x4GB) Kingston DDR3-1333 FB372D3D4P13C9ED1
Motherboard Asus Z8PS-D12-1U
Chipset Intel 5520
BIOS version 1102 (08/25/2011)
PSU 770W Delta Electronics DPS-770AB

To speed up testing, we tested with the Intel Xeon and AMD Opteron system in parallel. As we didn't have more than eight 8GB DIMMs, we used our 4GB DDR3-1333 DIMMs. The Xeon system only gets 48GB, but this is no disadvantage as our benchmark with the highest memory footprint (vApus FOS, 5 tiles) uses no more than 36GB of RAM.

We measured the difference between 12x4GB and 8x8GB of RAM and recalculated the power consumption for our power measurements (note that the differences were very small). There is no alternative as our Xeon has three memory channels and cannot be outfitted with the same amount of RAM as our Opteron system (four channels).

We chose the Xeons based on AMD's positioning. The Xeon X5649 is priced at the same level as the Opteron 6276 but we didn't have the X5649 in the labs. As we suggested earlier, the Opteron 6276 should reach the performance of the X5650 to be attractive, so we tested with the X5670 and X5650. We only tested with the X5670 in some of the tests because of time constraints.

Common Storage System

For the virtualization tests, each server gets an adaptec 5085 PCIe x8 (driver aacraid v1.1-5.1[2459] b 469512) connected to six Cheetah 300GB 15000 RPM SAS disks (RAID-0) inside a Promise JBOD J300s. The virtualization testing requires more storage IOPs than our standard Promise JBOD with six SAS drives can provide. To counter this, we added internal SSDs:

  • We installed the Oracle Swingbench VMs (vApus Mark II) on two internal X25-E SSDs (no RAID). The Oracle database is only 6GB large. We test with two tiles. On each SSD, each OLTP VM accesses its own database data. All other VMs (web, SQL Server OLAP) are stored on the Promise JBOD (see above).
  • With vApus FOS, Zimbra is the I/O intensive VM. We spread the Zimbra data over the two Intel X25-E SSDs (no RAID). All other VMs (web, MySQL OLAP) get their data from the Promise JBOD (see above).

We monitored disk activity and phyiscal disk adapter latency (as reported by VMware vSphere) was between 0.5 and 2.5 ms.

Software configuration

All vApus testing was done one ESXi vSphere 5--VMware ESXi 5.0.0 (b 469512 - VMkernel SMP build-348481 Jan-12-2011 x86_64) to be more specific. All vmdks use thick provisioning, independent, and persistent. The power policy is "Balanced Power" unless indicated otherwise. All other testing was done on Windows 2008 R2 SP1.

Other notes

Both servers were fed by a standard European 230V (16 Amps max.) powerline. The room temperature was monitored and kept at 23°C by our Airwell CRACs.

We used the Racktivity ES1008 Energy Switch PDU to measure power. Using a PDU for accurate power measurements might same pretty insane, but this is not your average PDU. Measurement circuits of most PDUs assume that the incoming AC is a perfect sine wave, but it never is. However, the Rackitivity PDU measures true RMS current and voltage at a very high sample rate: up to 20,000 measurements per second for the complete PDU.

Inside Our Interlagos Test System Virtualization Performance: Linux VMs on ESXi
Comments Locked


View All Comments

  • geoxx - Friday, December 9, 2011 - link

    Sorry but neotiger is totally right, choice of benchmark sucks. We are not helped *at all* by your review.
    What company 32-core server is being used for 3D rendering, cinebench, file compression, truecrypt encryption??
    You benchmarked it like it was a CPU of the nineties for a home enthusiast.

    You are probably right pointing us to but your benchmarks don't reflect that AT ALL. Where are file compression, encryption, 3D rendering and cinebench in that chart?

    Even performances per watt is not very meaningful because when one purchases a 2-socket or 4-socket server, electricity cost is not an issue. Companies want to simplify deployment with such a system, they want this computer to run as fast as a cluster, in order not to be bound to cluster databases which are a PAIN. So people want to see scalability of applications to full core count on this kind of system, not so much performances per watt.

    Virtualization is the ONLY senseful benchmark you included.

    TPC as suggested is a totally right benchmark, that's the backend and bottleneck for most of the things you see in your charts at , and objection on storage is nonsense, just fit a database in ramdisk (don't tell me you need a database larger than 64GB for a benchmark), export as block device, then run the test. And/or use one PCI-e based SSD which you certainly have. mentions software development: how much effort does it require to set up a linux kernel compile benchmark? mentions HPC: can you set up a couple of bioinformatics benchmarks such as BLAST (integer computation, memory compare), GROMACS (matrix FPU computations) and Fluent? Please note that none of your tests includes memory compares and FPU which are VERY IMPORTANT in HPC. Gromacs and fluent would cover the hole. Bioinformatics is THE hpc of nowdays and there are very few websites, if any, which help with the choice of CPUs for HPC computing.

    For email servers (37%!) and web servers (14%) also I am sure you can find some benchmarks.
  • Iketh - Tuesday, November 15, 2011 - link

    I'm not sure how the discovery of cores running in their power-saving state for far too long is anything new. My 2600k refuses to ramp up clocks while previewing video in a video editor even though a core is pegged at 100%. If I intervene and force it to 3.4ghz, preview framerate jumps from 8 fps to 16fps.

    This has been happening for YEARS! My old quad Phenom 2.2ghz did the exact same thing!

    It's extremely annoying and pisses me off I can't benefit from the power savings, let alone turbo.
  • MrSpadge - Tuesday, November 15, 2011 - link

    Sounds like you're running linux or some other strange OS, then. Or you may need a bios update. Generally Intel has its power management quit under control. In the AMD camp physical power state switches often take longer than the impatient OS expects, and thus average frequency is hurt. This was pretty bad for Phenom 1.

  • Iketh - Tuesday, November 15, 2011 - link

    win7 home premium x64 and the phenom was with xp 32bit... i haven't found another scenario that causes this, only streaming video that's rendered on-the-fly
  • Zoomer - Wednesday, November 16, 2011 - link

    You have a 2600k and aren't running it at 4+ GHz?
  • Iketh - Wednesday, November 16, 2011 - link

    4.16 @ 1.32v when encoding, 3.02 @ 1.03v for gaming/internet
  • haplo602 - Wednesday, November 16, 2011 - link

    you do know that Linux did not have any problems with Phenom I power management unlike Windows ? Same is not with BD. Linux benchmarks look quite different from Windows and the gap is not that dramatic there.
  • BrianTho2010 - Tuesday, November 15, 2011 - link

    This whole review, the only thought I have is that there are no sandy bridge chips in it. When SB based Xeon chips come out I bet that Interlagos will be completely dominated.
  • Beenthere - Tuesday, November 15, 2011 - link

    Not really. SB chips don't fit in AMD sockets. AMD's installed customer base like the significant performance increase and power savings by just plugging in a new Opteron 6200/4200.
  • C300fans - Tuesday, November 15, 2011 - link

    It will. 2x6174 (24 cores) perform quite similar to 2x6274(32 cores). WTF

Log in

Don't have an account? Sign up now