Rendering and HPC Benchmark Session Using Our Best Servers
by Johan De Gelas on September 30, 2011 12:00 AM ESTInvestigating the Opteron Performance Mystery
What really surprised us was the Opteron's abysmal performance in Stars Euler3D CFD. We did not believe the results and repeated the benchmark at least 10 times on the quad Opteron system. Let us delve a little deeper.
Notice that the Intel Xeons scale very well until the number of threads is higher than the physical core count. The performance of the 40 core E7-4870 only drops when we use 48 threads (with HT off). The Opteron however only scales reasonably well from 1 to 6 threads. Between 6 and 12 threads scaling is very mediocre, but at least performance increases. From there, the performance curve is essentially flat.
The Opteron Performance Remedy?
We contacted Charles of Caselab with our results. He gaves us a few clues:
1. The Euler3d CFD solver uses an unstructured grid (spider web appearance with fluid states stored at segment endpoints). Thus, adjacent physical locations do not (cannot!) map to adjacent memory locations.
2. The memory performance benchmark relevant to Euler3D appears to be the random memory recall rate and NOT the adjacent-memory-sweep bandwidth.
3. Typical memory tests (e.g. Stream) are sequential "block'' based. Euler3D effectively tests random access memory performance.
So sequential bandwidth is not the answer. In fact, in most "Streamish" benchmarks (including our own compiled binaries), the Quad Opteron was close to 100GB/s while the Quad Xeon E7 got only between 37 and 55GB/s. So far it seems that only the Intel compiled stream binaries are able to achieve more than 55GB/s. So we have a piece of FP intensive software that performs a lot of random memory accesses.
On the Opteron, performance starts to slow down when we use more than 12 threads. With 24 or even better 48 threads the application spawns more threads than the available cores within the local socket. This means that remote memory accesses cannot be avoided. Could it be that the performance is completely limited by the threads that have to go the furthest (2 hops)? In others words, some threads working on local memory finish much faster, but the whole test cannot complete until the slowest threads (working on remote memory) finish.
We decided to enable "Node Interleaving" in the BIOS of our Dell R815. This means that data is striped across all four memory controllers. Interleaved accesses are slower than local-only accesses because three out of four operations traverse the HT link. However, all threads should now experience a latency that is more or less the same. We prevent the the worst-case scenario where few threads are seeing 2-hop latency. Let us see if that helped.
Facebook
Twitter
RSS
52 Comments
View All Comments
Phynaz - Friday, September 30, 2011 - link
If you are overclocking in a business environment, what other moronic decisions has your company made?When does the going out of business sale start?
MrSpadge - Friday, September 30, 2011 - link
Don't condemn him blindly. By overclocking they can get substantially more performance from a similar budget. that's more efficiency - if done right.The question is "what happens in case of failure". If it's just a crashing machine, the rendering can be repeated by another one and this machine can be tuned down a bit. If it's a visual artefact during rendering, the redering can be repeated by any machine and this machine tuned down a bit. What else could go wrong in rendering? Obviously you wouldn't want to OC your web server or database..
BTW: there was an article here some time ago, showing Cyrix doing their testing on OC'ed i7s.
MrS
Kvarta - Tuesday, October 4, 2011 - link
Don't be so sure. Recently You can see standard desktop CPU beating expensive Xeons in professional applications. Example:http://www.solidworks.com/sw/support/shareyourscor...
So You don't need to buy very expensive DELL or other workstation, instead go to PC boutique near the corner :)
JohanAnandtech - Saturday, October 1, 2011 - link
This was not meant to be a professional rendering test. It was more an experiment to give the enthusiasts an idea what these servers are capable off. If you have a suggestion on which animation we should use in our benchmarking scenario's let me know. I have solid background in the "web- database - virtualization" field (I have been active in the field for more than 10 years now, teaching and consulting) , but rendering and HPC is something I only know from a benchmarking background :-).WeaselITB - Friday, September 30, 2011 - link
I'll echo the other sentiments here. If a Xeon system renders something twice as fast as the Opteron system, but takes five times the power draw to do it, it's a net-win for the Opteron system. Performance / watt would be a useful metric in these comparisons, especially as systems like these will be going in data rooms where excess wattage = excess heat = excess money.I would also be interested to know what a comparision would be between a "big iron" system like this versus a "traditional" render farm composed of some Core i7 machines.
Awesome review, though! I'm especially happy with the fact that you didn't just say "Oh, the Operteron kinda sucks in this test. Oh well." but actually took a look deeper into what's going on with the benchmark and the workload. THAT's the type of analysis that makes me keep coming back to AnandTech. :-)
Thanks,
-Weasel
JarredWalton - Friday, September 30, 2011 - link
Pretty sure perf/Watt isn't going to be in Opteron's favor, but there's a lot of stuff you need to account for. Johan did some measurements of power use on these servers previously (http://www.anandtech.com/show/4285/6), but as pointed out the Intel setup has a lot more RAS features and such that could be adding to the power use. Even so, the "load" power measured (using vAPUS, which may use less than something like 3D rendering) is around 875W (HT off) to 920W for the Intel E7-4870 server compared to around 590W for the Opteron 6174 server.In terms of perf/Watt, if those figures are relatively close for the benchmarks Johan has done here, then the best scores in Cinebench give 0.0355 CB/Watt for E7-4870 vs. 0.0425 CB/Watt for the Opteron 6174 -- and again, note that the 64-thread limit (tested with 40) means CB11.5 isn't able to make maximum use of the Intel platform. For the second test, best-case we measure 0.0194 CFD/W for Intel compared to 0.0145 CFD/W for AMD. So AMD wins in 3D rendering by 20% and Intel wins in the Euler3D CFD test by 34%, at least given the current estimates.
My gut feeling is that if all other elements and features were identical, other than the necessary chipset and CPU differences (e.g. the PSU, amount of RAM, HDDs, fans, RAS features, etc.), the difference in power draw for the two platforms should be within 100W, not the up to 340W spread measured in the earlier article. (There's also a 310W difference at idle, which gives some indication of all the other things that appear to be running on the Intel setup, as normal idle power looking at just the CPUs should be very nearly equal.) So these figures I list here are specific to the Intel Quanta QSCC-4R and Dell PowerEdge R815 and may not hold for other AMD/Intel servers. In other words, take with a grain of salt.
RandomUsername3245 - Friday, September 30, 2011 - link
The intel compiler is a very good compiler for Intel CPUs, but in the past it was well known for producing poor quality binaries for non-Intel CPUs. I would still be wary of benchmarking Intel vs. AMD when running code compiled with Intel's compiler.FWIW, I heard a while ago that Intel was "officially" going to stop artificially penalizing AMD CPUs that run Intel-compiled code.
James5mith - Friday, September 30, 2011 - link
Just a note:We are doing some in house testing for high end database testing using solid state storage connected via infiniband to multisocket servers.
An example:
Dell R910
4x 8C/16T X7560 2.26GHz Xeon CPU (32C/64T total)
512GB RAM
2x 146GB 15K SAS hdd's in RAID1 (OS)
2x Mellanox QDR Infiniband 40Gbps adapters
Hooked up to some seriously fast external flash storage, we got around 6GB/s+. This allowed us to do massively multi-threaded workloads, like building an index on a 2TB database.
During these tests, we can max out all 64 Threads and put the entire box under 100% load. It was during these tests we found out that Dell has a flawed implementation of the Intel SpeedStep technology which keeps the fans from ramping up under load.
Without the fast storage, we could never have fully stress tested the box.
mczak - Friday, September 30, 2011 - link
I think part why the opteron has bad scaling without interleaving and xeon does not is not just due to the coherence engine.Don't forget that while both have 4 sockets the Opteron is a 8 node system. The article states that there are "4 memory controllers" and "3 out of 4 operations traverse the HT link" which isn't really true as there are 8 memory controllers (and 7 out of 8 operations traverse HT, though some of them are internal HT links).
You can see that this makes a difference with the bad scaling from 6 to 12 threads (though not as bad as with even more threads...).
extide - Friday, September 30, 2011 - link
Dont forget the Xeon E7 is 4 sockets with 4 memory channels each.