Bulldozer for Servers: Testing AMD's "Interlagos" Opteron 6200 Series
by Johan De Gelas on November 15, 2011 5:09 PM ESTWhat Makes Server Applications Different?
The large caches and high integer core (cluster) count in one Orochi die (four CMT module Bulldozer die) made quite a few people suspect that the Bulldozer design first and foremost was created to excel in server workloads. Reviews like our own AMD FX-8150 launch article have revealed that single-threaded performance has (slightly) regressed compared to the previous AMD CPUs (Istanbul core), while the chip performs better in heavy multi-threaded benchmarks. However, high performance in multi-threaded workstation and desktop applications does not automatically mean that the architecture is server centric.
A more in depth analysis of the Bulldozer architecture and its performance will be presented in a later article as it is out of the scope of this one. However, many of our readers are either hardcore hardware enthusiasts or IT professionals that really love to delve a bit deeper than just benchmarks showing if something is faster/slower than the competition, so it's good to start with an explanation of what makes an architecture better suited for server applications. Is the Bulldozer architecture a “server centric architecture”?
What makes a server application different anyway?
There have been extensive performance characterizations on the SPEC CPU benchmark, which contains real-world HPC (High Performance Computing), workstation, and desktop applications. The studies of commercial web and database workloads on top of real CPUs are less abundant, but we dug up quite a bit of interesting info. In summary we can say that server workloads distinguish themselves from the workstation and desktop ones in the following ways.
They spend a lot more time in the kernel. Accessing the network stack, the disk subsystem, handling the user connections, syncing high amounts of threads, demanding more memory pages for expending caches--server workloads make the OS sweat. Server applications spend about 20 to 60% of their execution time in the kernel or hypervisor, while in contrast most desktop applications rarely exceed 5% kernel time. Kernel code tends to be very low IPC (Instructions Per Clockcycle) with lots of dependencies.
That is why for example SPECjbb, which does not perform any networking and disk access, is a decent CPU benchmark but a pretty bad server benchmark. An interesting fact is that SPECJBB, thanks to the lack of I/O subsystem interaction, typically has an IPC of 0.5-0.9, which is almost twice as high as other server workloads (0.3-0.6), even if those server workloads are not bottlenecked by the storage subsystem.
Another aspect of server applications is that they are prone to more instruction cache misses. Server workloads are more complex than most processing intensive applications. Processing intensive applications like encoders are written in C++ using a few libraries. Server workloads are developed on top of frameworks like .Net and make of lots of DLLs--or in Linux terms, they have more dependencies. Not only is the "most used" instruction footprint a lot larger, dynamically compiled software (such as .Net and Java) tends to make code that is more scattered in the memory space. As a result, server apps have much more L1 instruction cache misses than desktop applications, where instruction cache misses are much lower than data cache misses.
Similar to the above, server apps also have more L2 cache misses. Modern desktop/workstation applications miss the L1 data cache frequently and need the L2 cache too, as their datasets are much larger than the L1 data cache. But once there, few applications have significant L2 cache misses. Most server applications have higher L2 cache misses as they tend to come with even larger memory footprints and huge datasets.
The larger memory footprint and shrinking and expanding caches can cause more TLB misses too. Especially virtualized workloads need large and fast TLBs as they switch between contexts much more often.
As most server applications are easier to multi-thread (for example, a thread for each connection) but are likely to work on the same data (e.g. a relational database), keeping the caches coherent tends to produce much more coherency traffic, and locks are much more frequent.
Some desktop workloads such as compiling and games have much higher branch misprediction ratios than server applications. Server applications tend to be no more branch intensive than your average integer applications.
Quick Summary
The end result is that most server applications have low IPC. Quite a few workstation applications achieve 1.0-2.0 IPC, while many server applications execute 3 to 5 times fewer instructions on average per cycle. Performance is dominated by Memory Level Parallelism (MLP), coherency traffic, and branch prediction in that order, and to a lesser degree integer processing power.
So is "Bulldozer" a server centric architecture? We'll need a more in-depth analysis to answer this question properly, but from a high level perspective, yes, it does appear that way. Getting 16 threads and 32MB of cache inside a 115W TDP power consumption envelope is no easy feat. But let the hardware and benchmarks now speak.
Facebook
Twitter
RSS
106 Comments
View All Comments
zappb - Tuesday, November 29, 2011 - link
Thanks Johan for the ungodly amount of time you and your team spent on this review, also thanks to all contributors to the comments which was very useful to get more context to someone like myself who is not very up to speed with server tech.The 45 Watt and 65 watt Opterons not mentioned on the front page of the article (but mentioned in the comments - are these based on Interlagos?)
To me it looks like a big win for AMD - and these benchmarks are are not even optimised for the architecture (Linux kernel 3 was not used - can't wait to see updated benchmarks, something like FreeBSD or when we get an updated scheduler for the windows server OS's...should make a big difference.
Really low idle power consumption is nice, and Im planning to pick one of these up (for home use) to play around with FreeBSD, vm's, etc...just for training purposes,
The other point about Intel's sandybridge Xeons, these are just going to be 8 core 3960x right? Which may not change the current server landscape very much depending on their prices.
JWesterby - Friday, February 10, 2012 - link
Respect is due, Johan! You did a very useful review under significant limitations. The very best part is to point an unbiased light at a damned interesting CPU. There is an important "next step," which I will address shortly.As always, just the mention of AMD brings out hysterical attacks. One would think we were talking about Stem Cell research!! There is no real discussion -- it's pitchforks, lit torches, and a stake ready for poor Johan and anyone else ready and willing to consider the mere possibility that AMD have produced worthy technology!!
Computer technology - doing it, anyway - has changed. It's become ALL about the bloody money, and the "culture" of the people doing technology has also changed -- it has become much more cut-throat, there is far less collegiality, and the number of people willing to take risks on projects has become really uncommon. Qualified people doing serious technology just because they can is uncommon.
There is no end to posers (including some on this board), Machiavellian Fortune 500 IT managers, and "Project Managers" who are clueless (there ARE some great IT managers and wonderful PM's but their numbers are shrinking). My hat to those in Open Source - they are the Last Bastion of decency for the sake of decency, and technology merely for the joy of doing it !!
"Back in the day" people seemed really into the technology, solving difficult problems, and making good things happen. There was truly a culture. For example not taking a moment to help someone on the team or otherwise made you a jerk. Development was a craft or an art, and we were all in it together. We are loosing that, and it's become more dog-eat-dog with a kind of mean spirit. What a shame. Many of the comments here are perfect examples -- people who would rather burn down the temple than give a new and challenging technology a good think.
Personally I can't wait to get my hands on a couple of AMD's new CPU's, build a decent server, and carefully work out the issues with patience. These new Opterons are like a whole new tech that may be the beginning of all new territory.
My passion and some professional work is coding at the back end in C/C++ and I'm just beginning to understand CUDA and using GPU's to beef up parallel code. My work is all around (big) data warehousing, cutting edge columnar databases, almost everything running virtual, all the way through to analytics on BI platforms. I do all of that both on MS Server 2008, Solaris and FreeBSD. All that is a perfect environment to test AMD's "new territory."
Probably worth a blog at some point because these processors are new territory and using them well will take some work just keeping track of all the small things that shake out. That's the "next step" that this and other reviews require to really understand AMD's Bulldozers. Doing that well, if AMD is right with these chips, means being able to build some great back-end servers at a much more approachable price; more importantly without paying an "Intel" tax, and in the end having two strong vendors and thereby more freedom to make the best choice for the requirement.
PhotoPrint - Sunday, December 25, 2011 - link
you should make fair comparison at the same price range!lts like comparing GTX580 VS AMD RADEON 6950!
g101 - Wednesday, January 11, 2012 - link
Wow, anad let the truth about bulldozer leak out.ppennisi - Wednesday, March 7, 2012 - link
To obtain maximum performance from my Dell R715 server equipped with dual Interlagos processor I had to DISABLE C1E in the BIOS.Under VMware the machines performance changed completely, almost doubled in performance.
Maybe you should try it.
anti_shill - Monday, April 2, 2012 - link
Here's a more accurate reflection of Bulldozer/ interlagos performance, untainted by intel ad bucks...http://www.phoronix.com/scan.php?page=article&...
But if u really want to see what the true story is, have a look at AMD's stock price lately, and their server wins. They absolutely smoke intel on virtualization, and anything that requires a lot of threads. It's not even close. That would be the reason this review pits Interlagos against an Intel processor that costs twice as much.