Western Digital’s Advanced Format: The 4K Sector Transition Beginsby Ryan Smith on December 18, 2009 12:00 AM EST
- Posted in
We had a chance to talk to Western Digital last week about a new hard drive technology they have coming down the pipe called Advanced Format. Ahead of the impending launch of the first Advanced Format drives, Western Digital is engaging in an education campaign about the technology, as it represents the first step in a general transition for hard drives, and is a transition that can’t be done completely behind the scenes. Specifically, Advanced Format is going to bring about some issues where older operating systems (and we’re looking almost exclusively at Windows XP here) will inadvertently harm drive performance by making certain assumptions about the drives that will no longer be true.
We’ll get in to the nitty-gritty about the impact of this in a bit, but first let’s start with Advanced Format. What is it? In a nutshell, it’s replacing the traditional 512 byte hard drive sector with a 4 kilobyte (4,096B) hard drive sector for a number of practical and technological reasons. Advanced Format is the name Western Digital is giving to drives implementing 4KB sectors.
The move to 4K-sectors has been some 10 years in the making, as research in the late 90’s identified that 512B sectors made less and less sense as drives continued to get larger. At a time when hard drives were measured in megabytes, 512B sectors were a good balance between technical factors and a desire to keep wasted space at a minimum (you can’t normally put 2 files in 1 sector), but times have changed and hard drive capacities are several orders of magnitude larger.
The crux of the problem is that there are 3 factors that are in constant need of balancing when it comes to hard drive design: areal density, the signal-to-noise ratio (SNR) in reading from drive platters, and the use of Error Correcting Code (ECC) to find and correct any errors that occur. As areal density is increases, sectors become smaller and their SNR decreases. To compensate for that, improvements are made to ECC (usually through the use of more bits) in order to maintain reliability. So for a drive maker to add more space, they ultimately need to improve their error-correction capabilities, which means the necessary ECC data requires more space. Rinse, wash, repeat.
The SNR for various areal densities
At some point during this process drive manufacturers stop gaining any usable space - that is, they have to add as much ECC data as they get out of the increase areal density in the first place - which limits their ability to develop larger drives. Drive manufacturers dislike this both because it hinders their ability to develop new drives, and because it means their overall format efficiency (the amount of space on a platter actually used to store user data) drops. Drive manufacturers want to build bigger drives, and they want to spend as little space on overhead as possible.
But all is not lost. The principle problem here is that ECC correction takes place in 512B chunks, while ECC can be more efficient when used over larger chunks of data. If ECC data is calculated against a larger sector, even though more ECC data is necessary than for a single 512B sector, less ECC data than the sum of multiple sectors is needed to maintain the same level of operational reliability. One estimate for 4K sector technology puts this at 100 bytes of ECC data needed for a 4K sector, versus 320 (40x8) for 8 512B sectors. Furthermore the larger sectors means that larger erroneous chunks of data can be corrected (burst error correction), something that was becoming harder as greater areal densities made it easier to wipe out larger parts of a 512B sector. As a result, the need for the larger sector is born.
As for why Western Digital and other drive makers are transitioning specifically to a 4K sector, the rationale is based on the earlier problems. 4K sectors are much larger than 512B sectors, which means they benefit more from our earlier ECC optimizations, which in turn brings a greater increase in format efficiency than using smaller sectors (i.e. 1K).
4K also happens to be a magical number elsewhere when it comes to computers – this is primarily rooted in the fact that a normal page of memory on an x86 processor continues to be 4KB (ed: 4MB pages also exist). The x86 page size in turn has also lead to file system clusters (the smallest unit of storage in a file system) becoming 4KB as 4KB clusters neatly fit in to a page of memory, while the need for smaller clusters has subsided amidst a general increase in file size (i.e. fewer files are smaller than 4KB and waste space). NTFS, EXT3, and HFS+ all default to 4KB clusters on modern hard drives. So 4KB physical sectors map perfectly with 4KB file system clusters, which in turn map perfectly with 4KB memory pages. And hence 4KB is the largest practical size for a hard drive sector at this time.
|Old Sector||New Sector||x86 Mem Page||NTFS Cluster||HFS+ Cluster|
So what’s the net benefit from all of this for consumers? At the moment, not a lot, which is why this is a low-key launch for Western Digital, and the focus of this is an education effort on what the use of 4K sectors means for older operating systems. The biggest benefit is going to be that this will enable Western Digital to more easily design drives over 2TB in size.
From a numbers perspective, Western Digital estimates that the use of 4K sectors will give them an immediate 7%-11% increase in format efficiency. ECC burst error correction stands to improve by 50%, and the overall error rate capability improves by 2 orders of magnitude. In theory these reliability benefits should immediately apply to all 4K sector drives (making the Advanced Format drives more reliable than regular drives), but Western Digital is not pushing that idea at this time.
IDEMA projected format efficiency and ECC burst sizes
With all of that said, to make this kind of advancement practical, a transition period is necessary. That transition period will be done through the use of 512B emulation technology, which will expose Advanced Format drives to the drive controller and operating system as having 512B sectors, when in reality they will have 4K sectors. Unfortunately, this method has a downside, which is the ultimate reason for Western Digital’s education campaign.
512 byte sector emulation: how the drive is laid out (physical) and what the OS sees (logical)
With the emulation of 512B sectors, there’s the risk that a partition could be misaligned compared to the 4K physical sectors - where it would be unwittingly started in the middle of such a sector. As a result, the clusters of a file system on that partition would end up straddling 4K sectors, which would cause performance problems. Specifically, in testing IDEMA (the drive manufacturers’ trade group) found that random writes in would be particularly impacted, as a Read-Modify-Write(RMW) would need to take place to update each sector, rather than a straight write(ed: This is very similar to writing to a dirty block on a SSD). Although this isn’t mechanically/electronically harmful in any way, the performance hit compared to a straight write makes it undesirable.
A single unaligned 4K cluster straddling 2 physical sectors
A misaligned partition under WinXP. Only LBAs divisible by 8 are aligned
Post Your CommentPlease log in or sign up to comment.
View All Comments
Donuts123 - Sunday, December 20, 2009 - linkThe "problem" with Windows XP and earlier (and many other OSes) is that they create partitions which are not aligned on a 4KB boundary.
If you're installing XP on a new drive, the workaround is to partition the drive correctly before installing XP, and have XP install into the existing partition. You can use a Linux boot CD, but you might need to use command-line tools to ensure the partition is aligned, so that's not for novices.
So, what's the effect on performance when using a drive with 4KB physical sectors? If your partition is correctly aligned, there should be no performance decrease. The OS filesystem cluster size defaults to 4KB, so all reads and writes will normally be done in multiples of 4KB anyway. Since there is an effective areal density increase due to the larger physical sectors, you can expect 4KB-sector drives to have a higher data transfer rate than an equivalent 512-byte-sector drive.
So "it's all good", providing your partitions are aligned on a 4KB boundary.
If the partition is not correctly aligned, each filesystem read of n 4KB clusters requires the drive to read n+1 4KB physical sectors. The performance impact of that will be pretty small I think. Most filesystems have a read-ahead feature that reads in larger chunks anyway. The worst-case would be a 4KB random-read benchmark, and even there, no extra disk revolutions are needed.
Writes are the big problem. The impact will be decreased to some extent due to the drive's cache and the computer typically writing data in larger chunks. But read-modify write requires more revolutions of the disk. Each revolution takes 8.3ms for a 7200rpm drive. The worst-case scenario will be much slower than a 512-byte-sector drive.
Consider a 4KB random-write benchmark, i.e. one that writes 4KB of data to random places on the disk. (If your partition is not aligned, then none of the 4KB clusters are aligned.) For each 4KB write, the drive has to do this:
- Read two 4KB sectors.
- Wait while the disk makes one revolution (1/120th sec for a 7200rpm drive)
- Write back the two modified sectors
For every write in that case, an extra revolution of the disk is necessary. The actual worst-case scenario is even worse, but unlikely to be a problem in practice. Consider the case where the two sectors that the drive needs to read are on adjacent physical tracks. (The computer typically won't know where physical track boundaries are.) Then to write a 4KB sector, the drive needs to do this:
- Read the first sector (last sector of a track)
- Wait for a disk revolution
- Write the modified first sector
- Seek to the next track
- Read the second sector (first sector of the track)
- Wait for a disk revolution
- Write the modified second sector
So that requires two extra disk revolutions and a seek.
Solid-state drives have many of the same issues, along with a limited number of flash write cycles. Typically, the SSD flash page size is 4KB. If you use an SSD, you can improve drive performance and lifetime by ensuring your partitions are 4KB-aligned.
For what it's worth, drives with larger-than-512-byte sector sizes have been around for years. Magneto-optical (MO) drives first appeared in the late 1980s, and could use disks with 1024-byte sectors. (In that case though, the drive doesn't emulate a smaller sector size, so there is no problem.)
9.1GB MO drives, which first came out in late 2000, can use disks which have 4KB physical sectors, but the drive emulates 512- or 1024-byte sectors. Just like these new 4KB-sector hard drives.
DVD-RAM discs have 32KB physical sectors, but emulate 2048-byte logical sectors.
nowayout99 - Saturday, December 19, 2009 - linkI can't imagine wanting one of these drives until after the transition. Who would want the performance hit?
leexgx - Saturday, December 19, 2009 - linkwindows 7 own disk backup i am quite sure is alignment aware as well
johnsonx - Saturday, December 19, 2009 - linkI think Steve Gibson better get off his arse one of these days and update Spinrite. I suppose it will probably still work on 4KB drives, but I suspect it will be VERY non-optimal. It needs better support for modern SATA interfaces anyway, so it's time.
TechDicky - Friday, December 18, 2009 - link"Along with the need for 4K-aware operating systems, drive imaging software needs to be 4K-aware. Otherwise such imaging software may inadvertently create misaligned partitions too. As such, Win 6.x is also affected by alignment issues when imaging software is used, as some (and perhaps all?) imaging products currently available will write misaligned partitions and/or clusters."
That is not true of file based imaging such as WIM.
ppelleg - Sunday, February 21, 2010 - linkI think it's correst, instead!!!
When You clone to an empty drive, the disk cloning/imaging software must create the destination partition that MUST be 4K boundaries aligned so they MUST be 4K aware.
I can state it for sure, as I run into the problem!!! After cloning my 500GB to a 1TB WD drive I got a very slow system...
Problem solved using WD utility
puffpio - Friday, December 18, 2009 - linkthey talk about using jumpers on the drive to provide various workarounds to the 512b address mapping.
what about a jumper to say "address w/ 512b" which is enabled by default and if you pull the jumper, the hdd will bypass that translation layer and allow direct 4kb addressing?
yuhong - Friday, December 18, 2009 - linkDon't forget the 2TB MBR barrier too, which will require either an increase in the sector size that is presented to software, or an move to GPT.
hechacker1 - Friday, December 18, 2009 - linkYeah, when I initialized my 3x7200.11 raid5 array I had to use GPT.
Does anybody happen to know if Windows 7 (intel matrix raid) automatically align's the disk correctly? I have tried searching google but Windows 7 and Intel Matrix has me confused.
I am not sure if Windows 7 knows about the underlying disk structure of the matrix raid?
Thanks to anybody who knows.
leexgx - Saturday, December 19, 2009 - linkwindows vista/2008 or higher are aware of alignment and norm correctly set the partitions correctly
if you run diskpart there is an option you can pick to show if it has been incorrectly aligned or not (not sure where thought or there is an tool that lists it)