Bulldozer for Servers: Testing AMD's "Interlagos" Opteron 6200 Series
by Johan De Gelas on November 15, 2011 5:09 PM ESTBenchmark Configuration
Since AMD sent us a 1U Supermicro server, we had to resort to testing our 1U servers again. That is why we went back to the ASUS RS700 for the Xeon. It is a bit unfortunate as on average 1U servers have a relatively worse performance/watt ratio than other form factors such as 2U and blades. Of course, 1U still makes sense in low cost, high density HPC environments.
Supermicro A+ server 1022G-URG (1U Chassis)
| CPU |
Two AMD Opteron "Bulldozer" 6276 at 2.3GHz Two AMD Opteron "Magny-Cours" 6174 at 2.2GHz |
| RAM | 64GB (8x8GB) DDR3-1600 Samsung M393B1K70DH0-CK0 |
| Motherboard | SuperMicro H8DGU-F |
| Internal Disks |
2 x Intel SLC X25-E 32GB or 1 x Intel MLC SSD510 120GB |
| Chipset | AMD Chipset SR5670 + SP5100 |
| BIOS version | v2.81 (10/28/2011) |
| PSU | SuperMicro PWS-704P-1R 750Watt |
The AMD CPUS have four memory channels per CPU. The new Interlagos Bulldozer CPU supports DDR3-1600, and thus our dual CPU configuration gets eight DIMMs for maximum bandwidth.
Asus RS700-E6/RS4 1U Server
| CPU |
Two Intel Xeon X5670 at 2.93GHz - 6 cores Two Intel Xeon X5650 at 2.66GHz - 6 cores |
| RAM | 48GB (12x4GB) Kingston DDR3-1333 FB372D3D4P13C9ED1 |
| Motherboard | Asus Z8PS-D12-1U |
| Chipset | Intel 5520 |
| BIOS version | 1102 (08/25/2011) |
| PSU | 770W Delta Electronics DPS-770AB |
To speed up testing, we tested with the Intel Xeon and AMD Opteron system in parallel. As we didn't have more than eight 8GB DIMMs, we used our 4GB DDR3-1333 DIMMs. The Xeon system only gets 48GB, but this is no disadvantage as our benchmark with the highest memory footprint (vApus FOS, 5 tiles) uses no more than 36GB of RAM.
We measured the difference between 12x4GB and 8x8GB of RAM and recalculated the power consumption for our power measurements (note that the differences were very small). There is no alternative as our Xeon has three memory channels and cannot be outfitted with the same amount of RAM as our Opteron system (four channels).
We chose the Xeons based on AMD's positioning. The Xeon X5649 is priced at the same level as the Opteron 6276 but we didn't have the X5649 in the labs. As we suggested earlier, the Opteron 6276 should reach the performance of the X5650 to be attractive, so we tested with the X5670 and X5650. We only tested with the X5670 in some of the tests because of time constraints.
Common Storage System
For the virtualization tests, each server gets an adaptec 5085 PCIe x8 (driver aacraid v1.1-5.1[2459] b 469512) connected to six Cheetah 300GB 15000 RPM SAS disks (RAID-0) inside a Promise JBOD J300s. The virtualization testing requires more storage IOPs than our standard Promise JBOD with six SAS drives can provide. To counter this, we added internal SSDs:
- We installed the Oracle Swingbench VMs (vApus Mark II) on two internal X25-E SSDs (no RAID). The Oracle database is only 6GB large. We test with two tiles. On each SSD, each OLTP VM accesses its own database data. All other VMs (web, SQL Server OLAP) are stored on the Promise JBOD (see above).
- With vApus FOS, Zimbra is the I/O intensive VM. We spread the Zimbra data over the two Intel X25-E SSDs (no RAID). All other VMs (web, MySQL OLAP) get their data from the Promise JBOD (see above).
We monitored disk activity and phyiscal disk adapter latency (as reported by VMware vSphere) was between 0.5 and 2.5 ms.
Software configuration
All vApus testing was done one ESXi vSphere 5--VMware ESXi 5.0.0 (b 469512 - VMkernel SMP build-348481 Jan-12-2011 x86_64) to be more specific. All vmdks use thick provisioning, independent, and persistent. The power policy is "Balanced Power" unless indicated otherwise. All other testing was done on Windows 2008 R2 SP1.
Other notes
Both servers were fed by a standard European 230V (16 Amps max.) powerline. The room temperature was monitored and kept at 23°C by our Airwell CRACs.
We used the Racktivity ES1008 Energy Switch PDU to measure power. Using a PDU for accurate power measurements might same pretty insane, but this is not your average PDU. Measurement circuits of most PDUs assume that the incoming AC is a perfect sine wave, but it never is. However, the Rackitivity PDU measures true RMS current and voltage at a very high sample rate: up to 20,000 measurements per second for the complete PDU.
Facebook
Twitter
RSS
106 Comments
View All Comments
neotiger - Tuesday, November 15, 2011 - link
Most of the benchmarks are for rendering: Cinebench, 3DSMax, Maxwell, Blender, etc.How many enterprises actually do 3D rendering?
Far more common enterprise applications would be RDBMS, data warehouse, OLTP, JVM, app servers, etc.
You touched on some of that in just one virtualization benchmark, vApus. That doesn't make sense either - how many enterprises you know run database servers on VM?
A far more useful review would be running separate benchmarks for OLTP, OLAP, RDBMS, JVM, etc. tppc, tpce, tpch would be a good place to start
JohanAnandtech - Tuesday, November 15, 2011 - link
I definitely would like to stay close to what people actually use.In fact we did that:
http://www.anandtech.com/show/2694
But the exploding core counts made it as good as impossible.
1. For example, a website that scales to 32 cores easily: most people will be amazed how many websites have trouble scaling beyond 8 cores.
2. Getting an OLTP database to scale to 32 cores is nothing to sneeze at. If your database is small and you run most of it in memory, chances are that you'll get a lot of locks and that it won't scale anyway. If not, you'll need several parallel RAID cards which have a lot of SSDs. We might pull that one off (the SSDs), but placing several RAID cards inside a server is most of the time not possible. once you solve the storage bottleneck, other ones will show up again. Or you need an expensive SAN... which we don't have.
We had an OLAP/ OLTP and Java benchmarks. And they were excellent benchmarks, but between 8 and 16 cores, they started to show decreasing CPU utilization despite using SSDs, tweaking etc.
Now puts yourself in our place. We can either spend weeks/months getting a database/website to scale (and we are not even sure it will make a real repeatable benchmark) or we can build upon our virtualization knowledge knowing that most people can't make good use of a native 32 core database anyway (or are bottlenecked by I/O and don't care anyway), and buy their servers to virtualize.
At a certain point, we can not justify to invest loads of time in a benchmark that only interest a few people. Unless you want to pay those people :-). Noticed that some of the publications out there use geekbench (!) to evaluate a server? Noticed how many publication run virtualization benchmarks?
"That doesn't make sense either - how many enterprises you know run database servers on VM?"
Lots of people. Actually besides a few massive Oracle OLTP databases, there is no reason any more not to virtualized your databases. SQL server and MySQL are virtualized a lot. Just googling you can find plenty of reports of MySQL and SQL server on top of ESX 4. Since vSphere 4 this has been common practice.
"etc. tppc, tpce, tpch would be a good place to start "
No not really. None of the professional server buyers I know cares about TPC benches. The only people that mentione them are the marketing people and hardware enthusiast that like to discuss high-end hardware.
So you prefer software that requires 300.000$ of storage hardware over a very realistic virtualization benchmarks which are benchmarked with real logs of real people?
Your "poor benchmark choice" title is disappoing after all the time that my fine colleagues and me have spend on getting a nice website + groupware virtualization benchmark running which is stresstested by vApus which uses real logs of real people. IMHO, the latter is much more interesting than some inflated TPC benchmarks with storage hardware that only the fortune 500 can afford. Just HMO.
neotiger - Tuesday, November 15, 2011 - link
While scaling to 32 cores can be problematic for some software, it's worth keeping in mind that the vast majority of dual-socket servers don't have 32 cores.In fact, a dual-CPU Intel server only has *at most* 12 cores, that's a far cry from 32-cores. Postgresql & MySQL has no problem at all to scale to 12 cores and beyond.
Now if AMD decided to make a CPU with crappy per-core performance but has so many cores that most software can't take full advantage of, that's their own fault. It's not like they haven't been warned. Sun tried and failed with the same approach with T2. If AMD is hellbent on making the same mistake, they only have themselves to blame.
My post title is a bit harsh. But it is disappointing to see a review that devotes FOUR separate benchmarks to 3D rendering, an application that the vast majority of enterprises have no use for at all. Meanwhile, the workhorse applications for most enterprises, OLTP, OLAP, and such, received far too little attention.
tiro_uspsss - Wednesday, November 16, 2011 - link
"In fact, a dual-CPU Intel server only has *at most* 12 cores..."Incorrect. There is s1567. This allows 2-8 CPUs, with a max. of 8C/16T per CPU......... which I'm wondering why Anandtech failed to include in this review?
s1567 CPUs also have quad channel memory...
I really wish s1567 was included in this review..
Photubias - Wednesday, November 16, 2011 - link
Intel's S1567?You mean the E7-8830 CPU from the E7-8800 series which has prices *starting* at $2280?
-> http://ark.intel.com/products/series/53672
bruce24 - Wednesday, November 16, 2011 - link
"You mean the E7-8830 CPU from the E7-8800 series which has prices *starting* at $2280?"I'm not sure what he meant, but there are E7-2xxx processors for dual socket servers, which are priced much lower than the E7-8xxx processors which are for 8+ socket servers.
Photubias - Thursday, November 17, 2011 - link
You mean the E7-28xx serieshttp://ark.intel.com/products/series/53670 ?
They are priced a bit lower, is there a comparison you suggest?
Sabresiberian - Wednesday, November 16, 2011 - link
I have trouble understanding why people think a review should include research into every other similar product that might be used for the same purpose.I mean, I can understand ASKING for a review of another specific product, particularly if you've actually done some research on your own and haven't found the information you want, but to imply a review isn't complete because it didn't mention or test another piece of hardware is a bit - unrealistic.
;)
JohanAnandtech - Thursday, November 17, 2011 - link
Sabresiberian, a very sincere thank you for being reasonable. :-)Frankly I can't imagine a situation where someone would have trouble to decide between a Westmere-EX and an AMD CPU. Most people checking out the Westmere-EX go for the RAS features (dual) or RAS + ultimate high thread performance (Quad). In all other cases dual Xeon EP or Opterons make more sense power and pricewise.
JustTheFacts - Thursday, November 17, 2011 - link
Really? Is it that much trouble to understand that people want to see the latest AMD cpu's compared to the most current generation of Intel hardware? Especially when the previous Intel processor review posted on this site reported on Westmere-EX performance? I have trouble understanding why people wouldn't expect it.