We had a chance to talk to Western Digital last week about a new hard drive technology they have coming down the pipe called Advanced Format. Ahead of the impending launch of the first Advanced Format drives, Western Digital is engaging in an education campaign about the technology, as it represents the first step in a general transition for hard drives, and is a transition that can’t be done completely behind the scenes. Specifically, Advanced Format is going to bring about some issues where older operating systems (and we’re looking almost exclusively at Windows XP here) will inadvertently harm drive performance by making certain assumptions about the drives that will no longer be true.

We’ll get in to the nitty-gritty about the impact of this in a bit, but first let’s start with Advanced Format. What is it? In a nutshell, it’s replacing the traditional 512 byte hard drive sector with a 4 kilobyte (4,096B) hard drive sector for a number of practical and technological reasons. Advanced Format is the name Western Digital is giving to drives implementing 4KB sectors.

The move to 4K-sectors has been some 10 years in the making, as research in the late 90’s identified that 512B sectors made less and less sense as drives continued to get larger. At a time when hard drives were measured in megabytes, 512B sectors were a good balance between technical factors and a desire to keep wasted space at a minimum (you can’t normally put 2 files in 1 sector), but times have changed and hard drive capacities are several orders of magnitude larger.

The crux of the problem is that there are 3 factors that are in constant need of balancing when it comes to hard drive design: areal density, the signal-to-noise ratio (SNR) in reading from drive platters, and the use of Error Correcting Code (ECC) to find and correct any errors that occur. As areal density is increases, sectors become smaller and their SNR decreases. To compensate for that, improvements are made to ECC (usually through the use of more bits) in order to maintain reliability. So for a drive maker to add more space, they ultimately need to improve their error-correction capabilities, which means the necessary ECC data requires more space. Rinse, wash, repeat.


The SNR for various areal densities

At some point during this process drive manufacturers stop gaining any usable space - that is, they have to add as much ECC data as they get out of the increase areal density in the first place - which limits their ability to develop larger drives. Drive manufacturers dislike this both because it hinders their ability to develop new drives, and because it means their overall format efficiency (the amount of space on a platter actually used to store user data) drops. Drive manufacturers want to build bigger drives, and they want to spend as little space on overhead as possible.

But all is not lost. The principle problem here is that ECC correction takes place in 512B chunks, while ECC can be more efficient when used over larger chunks of data. If ECC data is calculated against a larger sector, even though more ECC data is necessary than for a single 512B sector, less ECC data than the sum of multiple sectors is needed to maintain the same level of operational reliability. One estimate for 4K sector technology puts this at 100 bytes of ECC data needed for a 4K sector, versus 320 (40x8) for 8 512B sectors.  Furthermore the larger sectors means that larger erroneous chunks of data can be corrected (burst error correction), something that was becoming harder as greater areal densities made it easier to wipe out larger parts of a 512B sector. As a result, the need for the larger sector is born.

As for why Western Digital and other drive makers are transitioning specifically to a 4K sector, the rationale is based on the earlier problems. 4K sectors are much larger than 512B sectors, which means they benefit more from our earlier ECC optimizations, which in turn brings a greater increase in format efficiency than using smaller sectors (i.e. 1K).

4K also happens to be a magical number elsewhere when it comes to computers – this is primarily rooted in the fact that a normal page of memory on an x86 processor continues to be 4KB (ed: 4MB pages also exist). The x86 page size in turn has also lead to file system clusters (the smallest unit of storage in a file system) becoming 4KB as 4KB clusters neatly fit in to a page of memory, while the need for smaller clusters has subsided amidst a general increase in file size (i.e. fewer files are smaller than 4KB and waste space). NTFS, EXT3, and HFS+ all default to 4KB clusters on modern hard drives. So 4KB physical sectors map perfectly with 4KB file system clusters, which in turn map perfectly with 4KB memory pages. And hence 4KB is the largest practical size for a hard drive sector at this time.

  Old Sector New Sector x86 Mem Page NTFS Cluster HFS+ Cluster
Size 512B 4K 4K 4K 4K

So what’s the net benefit from all of this for consumers? At the moment, not a lot, which is why this is a low-key launch for Western Digital, and the focus of this is an education effort on what the use of 4K sectors means for older operating systems. The biggest benefit is going to be that this will enable Western Digital to more easily design drives over 2TB in size.

From a numbers perspective, Western Digital estimates that the use of 4K sectors will give them an immediate 7%-11% increase in format efficiency. ECC burst error correction stands to improve by 50%, and the overall error rate capability improves by 2 orders of magnitude. In theory these reliability benefits should immediately apply to all 4K sector drives (making the Advanced Format drives more reliable than regular drives), but Western Digital is not pushing that idea at this time.


IDEMA projected format efficiency and ECC burst sizes

With all of that said, to make this kind of advancement practical, a transition period is necessary. That transition period will be done through the use of 512B emulation technology, which will expose Advanced Format drives to the drive controller and operating system as having 512B sectors, when in reality they will have 4K sectors. Unfortunately, this method has a downside, which is the ultimate reason for Western Digital’s education campaign.


512 byte sector emulation: how the drive is laid out (physical) and what the OS sees (logical)

With the emulation of 512B sectors, there’s the risk that a partition could be misaligned compared to the 4K physical sectors - where it would be unwittingly started in the middle of such a sector. As a result, the clusters of a file system on that partition would end up straddling 4K sectors, which would cause performance problems. Specifically, in testing IDEMA (the drive manufacturers’ trade group) found that random writes in would be particularly impacted, as a Read-Modify-Write(RMW) would need to take place to update each sector, rather than a straight write(ed: This is very similar to writing to a dirty block on a SSD). Although this isn’t mechanically/electronically  harmful in any way, the performance hit compared to a straight write makes it undesirable.


A single unaligned 4K cluster straddling 2 physical sectors


A misaligned partition under WinXP. Only LBAs divisible by 8 are aligned

What Operating Systems Are Affected
POST A COMMENT

87 Comments

View All Comments

  • tophe - Friday, January 15, 2010 - link

    I've bought this drive without knowledge of 4k technology and thus I've big problem because I'm using XP and W7 in my computer. I'd like to use this drive as single partition data drive, but will my data accesible from both OS? And then is it better to format it with W7 ( readable with xp w/out align software? ) or to format it with XP with the align software ( no trouble with W7 after? )? Or maybe I should change it for a normal 512bytes drive... If anyone can help I'd be very happy :) Reply
  • foxyshadis - Saturday, January 23, 2010 - link

    You'll be fine, but you MUST partition the drive from windows 7 or with another 4k-aware tool. Do NOT partition it with XP. Both of them will work absolutely fine after that, there's no other tweaks you have to make to get XP working, since it has 512b emulation. Reply
  • rree - Wednesday, January 06, 2010 - link


    http://ecartshopping.biz">http://ecartshopping.biz

    Air jordan(1-24)shoes $33

    Nike shox(R4,NZ,OZ,TL1,TL2,TL3) $35
    Handbags(Coach lv fendi d&g) $35
    Tshirts (Polo ,ed hardy,lacoste) $16

    Jean(True Religion,ed hardy,coogi) $30
    Sunglasses(Oakey,coach,gucci,Armaini) $16
    New era cap $15

    Bikini (Ed hardy,polo) $25

    FREE sHIPPING
    http://ecartshopping.biz">http://ecartshopping.biz
    Reply
  • carl0ski - Monday, December 28, 2009 - link

    Since the workarounds basically require the partition to shift across 512 bytes,
    From the notes it looks like Windows 5.x will be able to read the drives without issue.

    However the partition manager in the Windows installer on the original CD is broken and will not work. along with the partition manager in Computer Manager.

    The windows installer didnt change much if at all from the blue screen wizard from NT 4 to Windows 2003 R2 (4.0 - 5.2)

    Vista and 2003 use a different installer.

    It is impossible or very unlikely that MS will issue new updated Windows Installation Disks that is the main problem/symptom.


    If you create a Partition table in Vista / Windows 7 + you may be able to install from the 5.x CDs without trouble since the partitions already exist.
    Reply
  • Bill Todd - Thursday, January 07, 2010 - link

    For the next few years the new 4 KB sector drives will emulate older drives and thus be able to be used on any system that could use 512 B sector drives, right back to DOS x.x. They just won't *perform* as well for unaligned writes as the old drives did save in cases where the partition alignment has been fiddled to make things line up more efficiently.

    Note, by the way, that this partition alignment can be accomplished manually using older software: just adjust such that each primary partition starts on a physical 4 KB boundary (e.g., for the first partition on a disk start it on the 9th logical 'cylinder' of 63*255 512-byte sectors, since 8*63*255*512 is divisible by 4096). Logical partitions in the 'extended partition' need slightly different alignment to account for the fact that they don't include the first logical 'track' of 63 512-byte sectors, used by the 'extended partition boot record' in the same manner that the first logical 'track' on the disk is occupied by the 'master boot record'. Of course, mere partition alignment won't solve the performance problem if the file system residing in the partition doesn't observe neat 4 KB alignment internally.

    (It also may be possible to change the size of the 'track' occupied by the MBR or EPBR to 64 512-byte sectors, which would be a cleaner solution but might confuse some software.)

    Over the longer term, it's worth noting that Windows NT and its descendants have always been able to process data drives with sector sizes other than 512 bytes - they just can't boot from them. So save up a few boot drives for the time when no more 512-byte sector drives are available and keep using these systems as long as you care to.
    Reply
  • taltamir - Friday, December 18, 2009 - link

    it is fixed, in non ancient operating systems. Get windows 7...
    oh, you mean you want microsoft to upgrade their old OS for free to make it compatible with new technology?
    Reply
  • Taft12 - Monday, December 21, 2009 - link

    Yes, that is what I would like too. Service packs and patches have always been free. Reply
  • Calin - Sunday, December 20, 2009 - link

    Yes, that would be great.
    Also, USB for Windows NT 4.0 would have been great at that time.
    Reply
  • Ryan Smith - Friday, December 18, 2009 - link

    XP has reached its Extended Support phase. Microsoft will not be releasing anything besides security updates.

    There may be other issues too in trying to make Win 5.x 4K-aware, but the fact of the matter is that they're not going to lift a finger while its in Extended Support anyhow.
    Reply
  • mcnabney - Friday, December 18, 2009 - link

    It appears the WHS is also impacted by this issue and is not in Extended Support. Should I be expecting a fix for my Home Server? Reply

Log in

Don't have an account? Sign up now