Western Digital Raptor Preview: 10,000RPM & Serial ATA

Let's Talk about Hard Drive Benchmarks

If you ask how fast a 3.06GHz Pentium 4 will make your PC, all it takes are a few benchmarks to give you an indication of the performance boost; ask the same question about upgrading to the IBM Deskstar 180GXP and the question becomes much more difficult to answer.

While synthetic benchmarks almost never carry any weight when evaluating CPU or GPU performance, for whatever reason they have become entirely too accepted when looking at disk performance. Figures like transfer rates and access times are all users look at when evaluating a disk's performance, mostly because the only performance tools available for disk testing only report those two very easy to obtain numbers.

What we need are application-level benchmarks or real-world tests but for hard drives. Synthetic numbers are generally useless as they don't represent access patterns you encounter on even a semi-regular basis. What's necessary is something like a Winstone or SYSMark for disk drives, something we haven't had for years.

Ziff Davis was on the right track a few years ago when they had disk tests in their Winbench suite. The idea was simple; record all of the disk accesses that occurred during the Business and High End Winstone tests and play them back as quickly as possible. Recording all of the disk accesses made this a great test of drive performance and since the access patterns were derived from real-world applications and usage patterns. Unfortunately, there hasn't been a release of the benchmark since Winbench 99, with no immediate plans to release one based on the current-generation Winstone tests.

The problem with just sticking to Winbench 99 is that the applications and usage models represented are so old that their usefulness as a real-world performance measurement is beginning to fade (akin to using Quake III Arena as a benchmark for today's high performing GPUs).

Luckily, there are utilities out there that can help us find this holy grail of hard drive benchmarks. Intel developed a tool a while back that can be used to do exactly what Winbench did, record and playback disk accesses that occur during application runs. Although Intel's utility is no longer supported or made available, its usefulness hasn't even come close to expiring. So we have a tool that lets us record only the disk accesses that occur while using your computer, but what do we record?

We toyed with the idea of putting together our own usage patterns, but re-inventing the wheel made very little sense considering the multitude of real-world usage models that are already available - primarily the Winstone and SYSMark tests. What we ended up doing was in fact creating a Disk Winbench 2003 for Winstone and SYSMark, by recording all of the disk accesses that occur during Business Winstone 2002, Multimedia Content Creation Winstone 2003, Internet Content Creation SYSMark 2002 and Office Productivity SYSMark 2002.

With those four benchmarks we have general usage as well as high-end content creation usage patterns covered, which are the vast majority of what matters to desktop users. We contemplated recording the disk accesses that occur while running games or 3D rendering applications, but in the end we concluded that disk performance in both of those cases isn't nearly as important as in general usage (e.g. MS Word, Outlook, zip compression, web surfing, etc…) or content creation applications (e.g. Photoshop, Sound Forge, Premier, etc…).

If a hard drive happens to be great at loading levels in games but is mediocre at general usage performance, does it make sense to purchase it just because you're a gamer? The vast majority of your time spent waiting on your disk doesn't occur in games, but rather waiting for applications to start or complete an I/O intensive task; in gaming environments you're more performance limited by your CPU and GPU rather than your I/O subsystem, which is the exact opposite of most productivity applications.

For server tests we didn't have any industry standard benchmarks to use, but we did have our SQL Server tests that we use in our Enterprise CPU reviews. For this review we used our most I/O intensive SQL server trace - a 30 minute recording of activity on the AnandTech Forums DB Server. In the future we hope to expand our server I/O tests beyond this one SQL server trace, but for our first attempt we decided to keep things as simple as possible.

Our focus for our benchmark suite was to isolate the hard drive and test it using real-world access patterns, but this approach fails to offer an indication of what sort of real-world performance improvement you can expect from upgrading to a different hard drive. The benefit of using industry standard benchmarks as the access patterns we recorded for our disk tests is that in order to gauge the overall increase in system performance all we have to do is simply run the benchmark itself. Thus for all of the Winstone scores you will find two comparison charts; one illustrating only the performance of the hard disks in running the trace of disk accesses, and one showing off the overall real-world performance improvement; this way you'll be able to see not only how much faster one hard drive is when compared to another one, but you'll be able to determine how much of an improvement your overall system will receive because of the hard drive upgrade.