Server Buying Decisions: Memory
by Johan De Gelas on December 19, 2013 10:00 AM EST- Posted in
- Enterprise
- Memory
- IT Computing
- Cloud Computing
- server
Measuring Latency
To measure latency, we use the open source "TinyMemBench" benchmark. The source was compiled for x86 with gcc 4.8.1 and optimization was set to "-O3". We used the latency values measured with the largest buffer size: 64MB. The measurement is described well by the manual of TinyMemBench:
Average time is measured for random memory accesses in the buffers of different sizes. The larger the buffer, the more significant the relative contributions of TLB, L1/L2 cache misses and DRAM accesses become. All the numbers represent extra time, which needs to be added to L1 cache latency (4 cycles).
First we tested with 1 DPC. RDIMM-1866 means we use the Micron 1866 RDIMMs. So even though they only work at 1600 MT/s when the Xeon E5-2690 is used, we describe them as "1866". The RDIMM-1600 are Samsung DIMMs which are designed to run at 1600 MT/s.
The iMB buffer increases latency by about 5 to 10%. But LRDIMMs are hardly useful when you insert only one DIMM per channel. They are made and bought to run at 3DPC, so that's what we tested neext. Both LRDIMMs and RDIMMs have to clock back to 1333 MT/s at 3DPC.
The small latency advantage that RDIMM had is gone. In fact, LRDIMMs seem to have a very small latency advantage over the RDIMMs in this case. Again, memory performance of the Ivy Bridge Xeon is a bit better, but the small clock speed advantage (2.8 vs 2.7 GHz) is probably the simplest and best explanation.
In summary, the differences in latency and bandwidth are pretty small between similar LRDIMMs and RDIMMs. And in that case, it will be nearly impossible to measure any tangible effects on real world applications.
Facebook
Twitter
RSS
27 Comments
View All Comments
JohanAnandtech - Friday, December 20, 2013 - link
First of all, if your workload is read intensive, more RAM will almost always be much faster than any flash cache. Secondly, it greatly depends on your storage vendor whether adding more flash can be done at "dramatically lower cost". The tier-one vendors still charge an arm and a leg for flash cache, while the server vendors are working at much more competitive prices. I would say that in general it is cheaper and more efficient to optimize RAM caching versus optimizing your storage (unless your are write limited).blaktron - Friday, December 20, 2013 - link
Not only are you correct, but significantly so. Enterprise flash storage at decent densities is more costly PER GIG than DDR3. Not only that, but you need the 'cadillac' model SANs to support more than 2 SSDs. Not to mention fabric management is a lot more resource intensive and more prone to error.Right now, the best bet (like always) to get performance is to stuff your servers with memory and distribute your workload. Because its poor network architecture that creates bottlenecks in any environment where you need to stuff more than 256GB of RAM into a single box.
hoboville - Friday, December 20, 2013 - link
Another thing about HPC is that, as long as a processor has: enough RAM to do its dataset on the CPU/GPU before it needs more data, the quantity of RAM is enough. Saving on RAM can let you buy more nodes, which gives you more performance capacity.markhahn - Saturday, January 4, 2014 - link
headline should have been: if you're serving static content, your main goal is to maximize ram per node. not exactly a shocker eh? in the real world, at least the HPC corner of it, 1G/core is pretty common, and 32G/core is absurd. hence, udimms are actually a good choice sometimes.mr map - Monday, January 20, 2014 - link
Very interesting article, Johan!I would very much like to know what specific memory model (brand, model number) you are referring to regarding the 32GB LRDIMM—1866 option.
I have searched at no avail.
Johan? / Anyone?
Thank you in advance!
/ Tomas
Gasaraki88 - Thursday, January 30, 2014 - link
A great article as always.ShirleyBurnell - Tuesday, November 5, 2019 - link
I don't know why people are still going after server hardware. I mean it's the 21st century. Now everything is on cloud. Where you have the ability to scale your server anytime you want to. I mean the hosting provider companies like: AWS, DigitalOcean, Vultr hosting https://www.cloudways.com/en/vultr-hosting.php, etc. has made it very easy to rent your server.