Bulldozer for Servers: Testing AMD's "Interlagos" Opteron 6200 Series
by Johan De Gelas on November 15, 2011 5:09 PM ESTBenchmark Configuration
Since AMD sent us a 1U Supermicro server, we had to resort to testing our 1U servers again. That is why we went back to the ASUS RS700 for the Xeon. It is a bit unfortunate as on average 1U servers have a relatively worse performance/watt ratio than other form factors such as 2U and blades. Of course, 1U still makes sense in low cost, high density HPC environments.
Supermicro A+ server 1022G-URG (1U Chassis)
| CPU |
Two AMD Opteron "Bulldozer" 6276 at 2.3GHz Two AMD Opteron "Magny-Cours" 6174 at 2.2GHz |
| RAM | 64GB (8x8GB) DDR3-1600 Samsung M393B1K70DH0-CK0 |
| Motherboard | SuperMicro H8DGU-F |
| Internal Disks |
2 x Intel SLC X25-E 32GB or 1 x Intel MLC SSD510 120GB |
| Chipset | AMD Chipset SR5670 + SP5100 |
| BIOS version | v2.81 (10/28/2011) |
| PSU | SuperMicro PWS-704P-1R 750Watt |
The AMD CPUS have four memory channels per CPU. The new Interlagos Bulldozer CPU supports DDR3-1600, and thus our dual CPU configuration gets eight DIMMs for maximum bandwidth.
Asus RS700-E6/RS4 1U Server
| CPU |
Two Intel Xeon X5670 at 2.93GHz - 6 cores Two Intel Xeon X5650 at 2.66GHz - 6 cores |
| RAM | 48GB (12x4GB) Kingston DDR3-1333 FB372D3D4P13C9ED1 |
| Motherboard | Asus Z8PS-D12-1U |
| Chipset | Intel 5520 |
| BIOS version | 1102 (08/25/2011) |
| PSU | 770W Delta Electronics DPS-770AB |
To speed up testing, we tested with the Intel Xeon and AMD Opteron system in parallel. As we didn't have more than eight 8GB DIMMs, we used our 4GB DDR3-1333 DIMMs. The Xeon system only gets 48GB, but this is no disadvantage as our benchmark with the highest memory footprint (vApus FOS, 5 tiles) uses no more than 36GB of RAM.
We measured the difference between 12x4GB and 8x8GB of RAM and recalculated the power consumption for our power measurements (note that the differences were very small). There is no alternative as our Xeon has three memory channels and cannot be outfitted with the same amount of RAM as our Opteron system (four channels).
We chose the Xeons based on AMD's positioning. The Xeon X5649 is priced at the same level as the Opteron 6276 but we didn't have the X5649 in the labs. As we suggested earlier, the Opteron 6276 should reach the performance of the X5650 to be attractive, so we tested with the X5670 and X5650. We only tested with the X5670 in some of the tests because of time constraints.
Common Storage System
For the virtualization tests, each server gets an adaptec 5085 PCIe x8 (driver aacraid v1.1-5.1[2459] b 469512) connected to six Cheetah 300GB 15000 RPM SAS disks (RAID-0) inside a Promise JBOD J300s. The virtualization testing requires more storage IOPs than our standard Promise JBOD with six SAS drives can provide. To counter this, we added internal SSDs:
- We installed the Oracle Swingbench VMs (vApus Mark II) on two internal X25-E SSDs (no RAID). The Oracle database is only 6GB large. We test with two tiles. On each SSD, each OLTP VM accesses its own database data. All other VMs (web, SQL Server OLAP) are stored on the Promise JBOD (see above).
- With vApus FOS, Zimbra is the I/O intensive VM. We spread the Zimbra data over the two Intel X25-E SSDs (no RAID). All other VMs (web, MySQL OLAP) get their data from the Promise JBOD (see above).
We monitored disk activity and phyiscal disk adapter latency (as reported by VMware vSphere) was between 0.5 and 2.5 ms.
Software configuration
All vApus testing was done one ESXi vSphere 5--VMware ESXi 5.0.0 (b 469512 - VMkernel SMP build-348481 Jan-12-2011 x86_64) to be more specific. All vmdks use thick provisioning, independent, and persistent. The power policy is "Balanced Power" unless indicated otherwise. All other testing was done on Windows 2008 R2 SP1.
Other notes
Both servers were fed by a standard European 230V (16 Amps max.) powerline. The room temperature was monitored and kept at 23°C by our Airwell CRACs.
We used the Racktivity ES1008 Energy Switch PDU to measure power. Using a PDU for accurate power measurements might same pretty insane, but this is not your average PDU. Measurement circuits of most PDUs assume that the incoming AC is a perfect sine wave, but it never is. However, the Rackitivity PDU measures true RMS current and voltage at a very high sample rate: up to 20,000 measurements per second for the complete PDU.
Facebook
Twitter
RSS
106 Comments
View All Comments
UberApfel - Wednesday, November 16, 2011 - link
If anyone finds me a madman; let me explain this simply by example. Benchmark choices aside...If this test were to compare any of the top or middle-tier processors on the "AMD vs. Intel 2-socket SKU Comparison" chart ( http://www.anandtech.com/show/5058/amds-opteron-in... ) with their matching competition; this article would tell a different story in essence. Which does in fact, regardless of how fair the written conclusion may be, makes it biased.
Examples:
X5650 vs 6282 SE
E5649 vs 6276
E5645 vs 6272
JohanAnandtech - Thursday, November 17, 2011 - link
"Yet handpicking the higher clocked Opteron 6276 (for what good reason?) seems to be nothing but an aim to make the new 6200 series seem un-remarkable in both power consumption and performance"Do you realize you are blaming AMD? That is the CPU they sent us.
"The 6272 is cheaper, more common, and would beat the Xeon X5670 in power consumption which half this review is weighted on."
The 6272 is nothing more than a lower speedbin of the 6276. It has the same power consumption but slightly lower performance. Performance/wat is thus worse.
"PostgreSQL/SQLite? Facebook's HipHop? Node.js? Java? Something relevant to servers and not something obscure enough to sound professional? "
We use Zimbra, Phpbb, Apache, MySQL. What is your point? that we don't include every server software on the planet? If you look around how many publications are running good repeatable server benchmarks? If it would be so easy as running Cinebench or Truecrypt, I think everybody would be.
"Even the chart on Page 1 is designed to make Intel look superior all-around. For what reason would you exclude the Opteron 4274 HE (65W TDP) or the Opteron 4256 EE (35W TDP) from the 'Power Optimized' section?"
To be honest, those CPUs were not even in AMD's presentation that we got. We were only briefed about Interlagos.
UberApfel - Thursday, November 17, 2011 - link
Did they send you the Xeon X5670 also? I suppose who's ever handling media relations at AMD is either careless or disgruntled. eg. Sending a slightly overclocked processor with a 30% staple that happens to scale unusually bad in terms of power efficiency.Please just answer this honestly; if you had compared a Opteron 6272 w/ a E5645 ... would your article present a different story?
Fair as you may have tried to be; you don't have to look far to find a comment here that came to the "BD is a joke" conclusion.
---
Using a phpbb stress test is hardly useful or relevent as a server benchmark; nevermind under a VM. Unless configured extensively; it's I/O bound. "Average Response Time" is also irrelevant; how is the reader to know if your 'response time' does not favor processors better with single-threaded applications?
Additionally; VM's on a better single-threaded processor will score higher in benchmarks due to the overhead as parallelism isn't optimized. Yet these results make zero sense in real-world usage. It contradicts the value of VM's; flexible scalability for low-usage applications.
Finally; I'd estimate that less than 5% of servers are virtual (if that). VM's are most popular with web servers and even there they have a small market share as they only appeal to small clients. Large clients use clusters of dedicated; tiny clients use shared dedicated.
Did you even use gcc 4.7 or Open64? In some applications; the new versions yield up to 300% higher performance for Bulldozer.
JohanAnandtech - Thursday, November 17, 2011 - link
"if you had compared a Opteron 6272 w/ a E5645 ... would your article present a different story?"You want us to compare a $551 80W TDP Intel cpu with a $774 115 AMD CPU?
"Unless configured extensively; it's I/O bound."
We know how to monitor with ESX top. There is a reason why we have a disk system of 2 SDDs and 6 x 15k SAS disks.
"Average Response Time" is also irrelevant
Huh? That is like saying that 0-60 mph acceleration times are irrelevant to sports cars.
"Finally; I'd estimate that less than 5% of servers are virtual (if that)"
....Your estimate unfortunately was true in 2006. We are 2011 now. Your estimate is 10x off, maybe more.
UberApfel - Thursday, November 17, 2011 - link
"You want us to compare a $551 80W TDP Intel cpu with a $774 115 AMD CPU?"$539
"The 6272 is nothing more than a lower speedbin of the 6276. It has the same power consumption but slightly lower performance. Performance/wat is thus worse."
By your logic; the FX-8120 and FX-8150 have equal power consumption. They don't.
"We know how to monitor with ESX top. There is a reason why we have a disk system of 2 SDDs and 6 x 15k SAS disks."
It's still I/O bound unless configured extensively.
"Huh? That is like saying that 0-60 mph acceleration times are irrelevant to sports cars."
Yeah; it is if you're measuring the distance traveled by a number of cars. The opteron is obviously slower in handling single requests but it can handle maybe twice as many at the same time. Unless your stress test made every request @ T=0 and your server successfully qued them all, dropped none, and included the que time in the response time... it would favor the xeon immensely. Perhaps it does do all this; which is why I said "how is the reader to know" when you could have just as easily done 'Average Requests Completed Per Second'.
"....Your estimate unfortunately was true in 2006. We are 2011 now. Your estimate is 10x off, maybe more."
Very funny. Did the salesman that told you that also recommend these benchmarks? Folklore tells that Google alone has over a million servers, 20X that of Rackspace or ThePlanet, and they aren't running queries on VM's.
boomshine - Thursday, November 17, 2011 - link
I hope you included MS SQL 2008 performance just like in opteron 6174 review:http://www.anandtech.com/show/2978/amd-s-12-core-m...
JohanAnandtech - Thursday, November 17, 2011 - link
Yes, that test failed to be repeatable for some weird reason. We will publish it as soon as we get some reliable numbers out of it.JohanAnandtech - Thursday, November 17, 2011 - link
"SMT can only execute a single thread at once. "The whole point of SMT is to have one thread in one execution and another thread in the other execution slot.
In fact, the very definition of SMT is that two or more threads can execute in parallel on a superscalar execution engine.
TC2 - Thursday, November 17, 2011 - link
another joke from AMD with their BD "server-centric" architecture - bla-bla! amd 8\16 against intel 6\12 and again can't win!pcfxer - Thursday, November 17, 2011 - link
" make of lots of DLLs--or in Linux terms, they have more dependencies"Libraries is the word you're looking for.
I also see the mistake of mixing programming APIs/OS design/Hardware design...
Good software has TLBs, asynchronous locking where possible, etc, as does hardware but they are INDEPENDENT. The glue as you know, is how compiled code is treated at the uCode level. IMO, AMD hardware is fully capable of outperforming Intel hardware, but AMD uCode is incredibly good.