LRDIMMs, RDIMMs, and Supermicro's Latest Twin
by Johan De Gelas on August 3, 2012 4:45 AM EST- Posted in
- IT Computing
- Intel
- Samsung
- Xeon
- Cloud Computing
- Supermicro
Virtualized Cluster Testing
To fully understand the impact of adding more RAM, we test with both two tiles (72GB allocated) and three tiles (108GB). At two tiles, there will be some RAM left in the 64GB setup. ESXi allocates very little memory for the file system caches ("cached" in Windows) and gives priority to the active memory pages, the pages that are actually used by the most important application inside the VM.
To keep the benchmark results easily readable, we standardized all performance to the performance of the system configured with the LRDIMMs (128GB) and two tiles. So that system always scores 100 (%). All other performance numbers are relative to that.
First we check the total throughput. This is the geometric average of the standardized (a percentage relative to our LRDIMM system) throughput of each VM.
The 15% higher bandwidth and slightly better latency of the RDIMMs at 1600MHz allows the RDIMM configured server to outperform the one with LRDIMMs by 6% when running two tiles.However, once we place four more VMs (three tiles) on top of the machine, things start to change. At that point, ESXi had to create "Balloon memory" (10GB) and swap memory (4GB) on the 64GB RDIMM machine. So it is not like we went far beyond the 64GB of active memory. ESXi still managed to keep everything running.
The 128GB LRDIMM server is about 7% faster running three tiles instead of 6% slower (two tiles). That is not spectacular by any means, but it is interesting to delve a little deeper to understand what is happening. To do so, we check out the average latency. The average latency is calculated as follows: the average of the response time of each VM is divided by the response time of the same VM on the LRDIMM system (two tiles). We then calculate the geometric mean of all percentages.
As soon as we add more VMs both systems suffer from higher response times. This shows that we made our test quite CPU intensive. The main culprit is SPECJBB, the VM that we added to save some time, as developing a virtualization test is always a dire and complex undertaking. But trying to save time unfortunately reduced the realism of our test. The problem is that SPECJBB runs the CPU at close to 100%, and as a result the CPU becomes the bottleneck at three tiles. In our final "vApus Mark Mixed" we will replace the SPECjbb test.
We decided to show you this data already until we develop a completely real world benchmark. We are currently evaluating a real world Ruby On Rails website and a Drupal based site. Please feel free to share your opinion on virtualization benchmarking if you are active in this field. Until then, can the extra RAM capacity help even if your applications are CPU intensive? 17% better response times might not be impressive, but there is more to it.
Facebook
Twitter
RSS
26 Comments
View All Comments
ddr3memory - Monday, August 6, 2012 - link
Your article is very interesting - and the first mainstream (and belated) examination of the LRDIMM (new standard - incompatible with RDIMMs) vs. HCDIMM (100% DDR3 RDIMM compatbile) choice for Romley.I have whittled down the use case for HCDIMMs/LRDIMMs and RDIMMs as follows:
The HCDIMM use case is at:
- 16GB at 3 DPC use
- 32GB (outperform both RDIMMs and LRDIMMs)
LRDIMMs are not viable at:
- 16GB (RDIMMs are better)
- 32GB (HCDIMMs are better)
RDIMMs are not viable at:
- 32GB (because they are 4-rank - trumped by LRDIMMs/HCDIMMs)
There is a reason the Netlist HCDIMMs were only released on the virtualization servers from IBM/HP - because at 16GB levels the only niche available for LRDIMM/HCDIMM vs. RDIMM is the 3 DPC space. This will expand considerably at 32GB to mainstream levels as soon as 32GB HCDIMMs are released (they are currently in qualification with IBM/HP and have not been announced yet - though maybe expected shortly).
I had created an infographic covering the memory choices - search the net for the article entitled:
Infographic - memory buying guide for Romley 2-socket servers
HCDIMMs are not available at SuperMicro (as they are for IBM/HP) - so I was surprised you even covered HCDIMMs (since the article is after all referring to the SuperMicro line of servers).
Casper42 - Friday, August 10, 2012 - link
BTW, Johan, I work for HP and asked some of the guys in ISS Technical Marketing why we don't send you our servers for eval like you get from SuperMicro and sometimes DellThey felt that you guys didn't do alot of Server Reviews, and that your readership wasn't generally the kind of folks that buy HP Servers.
So I am curious if you could spin up a poll or something in the future to prove them wrong.
If there is enough support I'm sure we can you some gear to play with.
I sometimes giggle when I see the stuff people on here get excited about in these reviews though. "Can you see the BIOS through IPMI?". Thats the kind of thing Compaq offered back with the RILOE II and have been integrated into the motherboard since iLO 1 which is like 4 or 5 years old at least.
iLO4 on the Gen8 line have taken that a step further and we now hook the Display system BEFORE POST starts so instead of an invalid memory config getting you a series of beeps, you now get a full blown screen either on local VGA or on the Remote Console that straight up tells you you have a memory mismatch and why. i have seen his demo'd with NO DIMMs even installed in the server and you still get Video and obvious status messages.
Casper42 - Friday, August 10, 2012 - link
Also you are about $2000 high on the HP SL unless I am missing something.I found these prices with QuickSpecs part numbers and Google, nothing magical inside HP.
Half of one of these:
http://www.provantage.com/hewlett-packard-hp-62923...
Includes 8 fans and 3 PS
2 of these
http://www.provantage.com/hewlett-packard-hp-65904...
2x2665 with 8GB
Comes to about $11,600
JohanAnandtech - Tuesday, August 14, 2012 - link
Hey Casper, contact me on my mail... thx!ad99 - Monday, April 1, 2013 - link
You say：a quad rank DIMM with 4Gb chips is a 32GB DIMM (4 Gbit x 8 x 4 ranks)，but I think 4 Gbit x 8 x 4 ranks make only 16GB,is that right?ad99 - Monday, April 1, 2013 - link
No,4 Gbit x 8 x 4 ranks should be 128GB，not 32GB or 16GB,is that right?