Searching for the Memory Holy Grail - Part 2
by Wesley Fink on August 26, 2003 11:11 PM EST- Posted in
- Memory
Test Design
One of the difficulties in testing memory is that most of the memory benchmarks available are synthetic. While synthetic benchmarks can be useful in comparing performance, they can also paint a distorted picture of real-world performance. This is the reason why AnandTech has always preferred benchmarking with real applications. Benchmarks using games are dependent on many system components for their results, including the CPU speed and Video Card, which have a major impact on the final scores. While memory does impact the game benchmark score, it is only one small part of the total score. Finding a game benchmark that is sensitive to memory is not always easy. We discovered Gun Metal 2, for instance, tends to be video-card bound, making it very useful for testing video cards, but not so useful for measuring subtle differences in system performance. After looking at available game benchmarks, we found Quake3 and Unreal Tournament 2003 to be the most useful for our memory testing.
The following Benchmarks were used in our Memory Testing:
1) SiSoft Sandra Max3 UNBuffered Memory Test
Part 1 of “Searching for the Memory Holy Grail” demonstrated the usefulness of the SiSoft Sandra UNBuffered Memory Test as a sensitive benchmarking tool for memory bandwidth. The Sandra UNBuffered Memory Test turns off Memory Buffering schemes in an attempt to improve the measure of raw memory bandwidth. As a result, it also correlates well with bandwidths reported with Memtest86, an industry-standard memory testing tool.The idea of the UNBuffered Memory Benchmark is very simple — you merely turn-off all memory buffering techniques. Sandra makes this very easy to do. Select “Memory Benchmark”, right-click “Module Options”, and uncheck the nine boxes that are related to buffering.
2) SiSoft Sandra Max3 Standard Memory Test
The UNBuffered Memory Benchmarks are quite different from what you may be accustomed to seeing in memory testing with SiSoft Sandra. For reference, we are again including the Sandra Max3 standard Memory Test, sometimes called the Buffered Memory Test.
3) Super PI
Pure number-crunching benchmarks are very useful for measuring system bandwidth. Some of the more popular number-crunchers are the MPEG/DIVX encoding tests, such as the ones that we used in our standard motherboard testing, and Super PI. MPEG/DIVX tests are valuable for a single motherboard benchmark and in cross-platform testing — Athlon vs. Pentium4, for example. However, they are often very sensitive to the test environment or system configuration, and can be difficult to use reliably in an environment that tests a large number of conditions with the same test, such as we will be doing here in our memory testing. Super PI, on the other hand, is very simple to use and has been shown to be less sensitive to the operating system environment. In other words, we don’t have to reinstall the operating system on a clean hard drive each time we run a benchmark just to get reliable numbers.Super PI for Windows 1.1 is a freeware program developed by the Super Computer Consortium at the University of Tokyo. The concept of Super PI is very simple — it calculates the value of pi to “x” number of places, and reports the time this calculation requires. We chose to use 2 million places in our tests. Super PI measures total system bandwidth, and memory is only part of that bandwidth, since the CPU has a significant impact on results. We therefore would expect to see smaller changes in Super PI relative to larger changes in memory-only benchmark tests like Sandra.
4) Quake3 Demo FOUR.dm_66
Quake 3 Demo FOUR is one of our standard game benchmarks. As Evan Lieb showed in his PC3200 memory tests, Quake3 can also reveal variations in memory performance. You will likely be surprised how sensitive Quake3 can actually be in testing wide variations in Memory Speed. We run the benchmark three times, check for score consistency, repeat if we see any wide variation in individual scores, and then average the three scores for the reported Frames per Second (FPS) value.
5) Unreal Tournament 2003 Demo
The Benchmark program built into the UT2003 demo is a contemporary game test that does respond to variations in memory bandwidth. We used it mainly to show the impact of memory speed on UT2003 scores, and to confirm the validity of Quake3 as a real-world test of memory performance. With our new standard ATI 9800 PRO video card, UT2003 shows variation in both Flyby and Botmatch in tests with memory of different speeds. All benchmarks are run at our standard 1024x768 resolution.
Motherboard, CPU, and Peripherals
In Part 1 of “Searching for the Memory Holy Grail”, we used the Abit IC7 for our 875 tests and the Asus P4P800 Deluxe for 865 tests. Because of the impact of PAT on/off at different FSB, we decided to use only the Intel 875 for testing in Part 2. This presented our first problem, since the Abit IC7 has a strange quirk in the way it handles 1:1 memory. No matter what we did, we could not operate the Abit IC7 at greater than a 255 setting (1020FSB). We talked with Abit about this issue, and they are hard at work on a BIOS revision to correct this problem. To be fair, most users will not need to run memory at faster than a setting of 255, and 5:4 and 3:2 memory ratios do not have this issue. However, for our tests here, we expected that we might reach a speed of 275 (1100FSB), and the 255 limitation was not acceptable.Next, we considered the DFI 875PRO LanParty as the motherboard for our testbed. The DFI had no problem handling settings above 255, which corrected that problem. However, as Evan Lieb pointed out in his review of the 875PRO, the vDIMM range to only 2.7V was too limiting for our high-speed memory tests. DFI has told us that they are releasing an updated version of the 875PRO in the near future with expanded vDIMM options. If that were available today, the DFI would have worked well for our testbed.
The latest revision of the ASUS flagship 875 motherboard is the ASUS P4C800-E. This board adds Intel GigaLAN (using the dedicated Intel CSA bus), and incorporates the ICH5R Southbridge with Intel SATA RAID. We will be doing a review update on this new revision of the ASUS flagship Canterwood shortly. The P4C800-E met our requirements of high speed 1:1 operation and a vDIMM adjustment range that was useful. vDIMM is available to 2.85V on the P4C800-E. As a bonus, we were able to use Intel SATA RAID with SATA drives for all testing.
Our 3.0C Pentium 4 800FSB chip was not very useful for testing DDR500 memory. With a maximum overclock of around 245 (980FSB), we could not even reach the rated speed of the memory. We settled on a 2.4C 800FSB Pentium 4 that has been proven to perform very well at high speeds. On the ASUS P4C800-E, this 2.4C was able to reach a stable 288 setting (1152FSB) at default 1.525V, and a setting of 298 (1192FSB) with a modest vCore setting of 1.6V. We were confident that this test setup would allow us to reach the maximum speeds possible with memory rated at a high as DDR500, since we did not anticipate that synchronous operation would exceed DDR596 in our testing.
Since all testing would be done on a single testbed configuration and only compared to test results on that testbed, we chose the best-performing components that we had available. For CPU cooling, we used the Thermalright SLK-900U heatsink with a 120mm adjustable-speed Vantec Tornado cooling fan. The idea here was to remove any concerns about CPU cooling or overclocking ability from the memory test as much as possible. For the video card, we used our new standard ATI Radeon 9800 PRO with 128 MB memory. Hard drives were a pair of Western Digital Raptor 10,000RPM Serial ATA drives running in a SATA RAID 0 (Striping) configuration on the stock Intel ICH5R.
Facebook
Twitter
RSS
77 Comments
View All Comments
Anonymous User - Wednesday, August 27, 2003 - link
Interesting review....I've been waiting for this one. As for the criticism, I would say don't just look at this one review. There seem to be plenty of reviews from other sites listed in the memory news section here. Why not check out what other sites have to say about the memory listed in this review...unless you think they are all shills for a particular product. IMO I think Wesley and AT are on the level...but if you have doubts there are other reviews out there. I'm still dissapointed tho...It seems that most of the 500mhz modules are just overclocked 400mhz modules with poor timings...I wonder if there are going to be true 500mhz modules with better timings in the works or is this the last stop before we see DDR2.artifex - Wednesday, August 27, 2003 - link
Ok, I stopped being lazy. It's me!Anonymous User - Wednesday, August 27, 2003 - link
hey 43/44, you must be a stringer for Mushkin, because you're cheering at the end (Mush! Mush!)Just kidding. I usually post as "artifex" but am too lazy to find my password right now. You'll see I've mentioned them before. Also comments about Anandtech being the site (along with SharkyExtreme) that I rely on heavily...
I bought my first ever Asus mobo (a7n8x deluxe) a couple months ago because of the reviews here (my first and last Soyo had just died from some bad capacitors, I won't trust them again) so what I want now is to see a shootout of the best memory for dual-channel nforce2 boards (especially mine :) )
Speaking of, I'm running one of the new weird 333FSB 2600+ XPs that's still a Tbred B, and I'm wondering, would I really benefit from memory that's pc3200 (probably not) or even pc2700, or are my current 2 512mb sticks of 2-3-3 pc2100 memory fine?
p.s. ironically, I also bought the Soyo based on reviews, but reviews don't make up for long-term experience with hardware (when cheap components fail). It'd be cool to see a follow up article or two, even just anecdotal, from the reviewers about how their own personal systems are working a year or so after they assemble them... and see if that changes their views about the vendors.
retrospooty - Wednesday, August 27, 2003 - link
Cool, Muskin rocks...I would REALLY like to see some Mushkin PC3500 at 5:4 2-2-2 against all those other PC4000 at 1:1 3-4-4-8... I know the PC3500 would win, I would just like to see it in print. !
Anonymous User - Wednesday, August 27, 2003 - link
Good points #42. I should have included similar in my last comment as I feel exactly the same. A quick explanation (by name) of which major manufacturers did not make the deadline or will be reviewed later would satisfy my curiousity on why large manufacturers weren't present. I cannot wait to see what the Mushkin stuff rates at and THEN I will choose between them, OCZ, Geil, and Corsair. Just in case I did not make it clear before, I do feel that when AnandTech does put out new articles they are often very good. The basis of the general readers scrutiny comes from how things look overall with the manufacturers listed on the same page as their reviews. Since the use of META data has become so prevailent in page content, it is easy to believe that all advertisements for memory would show up on a memory review. One would think that was a good idea, right?! As to the integrity of AnandTech, I still think you guys are unbiased and professional. Just remember to foster that perception in bold stroke so the goobs who cannot fathom the idea that any one company can be in it's prime and actually BE producing the best product on several reviews. That said, I still want you guys to make more content and faster. Come on! Get to work! Make it 60... No. 80 hours a week each!! Mush! Mush!Cheers,
Wiley
Anonymous User - Wednesday, August 27, 2003 - link
cool. thanks for saying that about Mushkin.I hadn't seen much of them in Anandtech reviews or advertising on the site lately, so I was wondering.
I used to be a big Crucial.com fan until they started slipping (recommending wrong types and amounts of memory, customer service, etc), and then went with Mushkin when the chance presented itself, based entirely on seeing their name associated with good things here in the past. My first set of sticks from them actually gave errors about a year into use, and I had bought them at Fry's (stupid, I know), but Mushkin overnighted me replacements anyway, so I'm definitely loyal to them now (the cost of overnight probably ate the marginal extra I paid over Corsair, which I also considered).
I know this is long winded, and probably sounds like astroturfing, but I want the editors to remember that we really do buy stuff based on what you say, and when it looks like "favorite" brands are disappearing without any comment, it makes us wonder. Hey, even if you guys get into a fight with a vendor and they stop lending you review stuff, that's a legitimate reason... just tell us.
KristopherKubicki - Wednesday, August 27, 2003 - link
Tom Duong from Mushkin had contacted AnandTech several weeks before this review to let us know that their memory sample would not make to us in time for the review. We do have some other articles with Mushkin coming up.Cheers,
Kristopher
Anonymous User - Wednesday, August 27, 2003 - link
Personally, I would just like some parity for these reviews. Every time I read these I wodner why Mushkin is never listed. I realize that teh manufacturer needs to provide product for testing and by a deadline. However, it just seems silly that a company the rates its DDR400 memory at Cas2-2-2 (Black Level II) has not been listed here. Can't AnandTech sport for the 400 bucks to GET memory when a reviewer cannot send it or does not do so in a timely manner. I always had the impression that sites like this were produced with the best interest of the end reader in mind. Anytime a single manufacturer is lauded repeatedly for their product and that product happens to be promoted profusely in advertising on the reviewers site, there will be questions regarding favoritism. I'm not saying you fudge your numbers to keep people at OCZ happy, there is too much data to support your conclusions regarding the quality of OCZ memory. But it does make you wonder. The lask of Mushkin in the original DDR400 porion of your latest memory benchmarks on this site being a perfect example. Who is to say that the Mushkin stuff was not left out just to leave a better window of opportunity for somone like OCZ to succeed i your tests? I hope that is not the case but again, I have to wonder why I never see them here. Get on them and get some of their product to test too! I have to agree with many here that teh content at AnandTech is slipping. The updates are way to far apart, there are few articles when an update DOES occur, and the content is subject to scrutiny for bias. In all fairness, any of the ideas I have posited could be true or false. However, with this sites reputation being the MAIN provider of it's marketability, AD sales potential, and reader support, don't you think that making sure there is NEVER a question of integrity would be a good business practice? And update the site more often! I used to read this site everyday. Now I just come here once a week tops.Cheers,
Wiley
Anonymous User - Wednesday, August 27, 2003 - link
Mushkin did not have PC4000 ready in time for the review.Im sure Wes will review it as soon as he recieves samples.Wesley has got to be the most unbiased reviewer i know.He upset a few at abxzone and im sure he will do the same here.
Just because OCZ has been doing well is NOT because they paid for a good review....its because the modules are the best at the moment..nothing more.
OCZ have worked hard these past 12 months or more,they are gaining market share and more sites are starting to use their ram for board reviews etc..the only people this will upset is Corsair who have had a strong hold on review sites for a long while.
So all you "employee's" coming here to bash OCZ--- please give it a rest.If you have an issue with OCZ product contact me. oczguy2@ocztechnology.com
Please remember this thread is about Wes's review after all, not bashing OCZ so please lets all get back on topic.
Thanks.
bigtoe
Anonymous User - Wednesday, August 27, 2003 - link
The differece is small on synthetic tests like Sandra mem tests, but large on real stuff like games,and 3dmark real apps.the same thory hold true at any speed, try it at 250, 230 or even 200
5:4 at 2-2-2 is faster than 1:1 at 3-4-4 period
Try it.