Bad Sector Blues

[glc_engineer]

This document is arranged as follows:
1.0 Introduction
2.0 Hard Disk Basics
2.1 The physical structure
2.2 The logical structure
2.3 Computer interface
3.0 Fixing bad sectors
3.1 How bad sectors form
3.2 Low-level formatting
3.3 High-level formatting
3.4 Disk diagnostic utilities
4.0 Conclusion

[glc_engineer]

1.0 Introduction

This document seeks to answer common questions about one of the most troubling hardware failures of disk devices; the sector read error (AKA bad sector). The intended audience of this document are those "power" users who know that their hard disk has developed a bad sector, and want to remedy the problem themself. A basic knowledge of how hard disks operate and are accessed is beneficial to the reader, but this document does briefly explain these things. For a much more in-depth review of hard disk technologies and problems you should see;

برای مشاهده محتوا ، لطفا وارد شوید یا ثبت نام کنید

A knowledge of basic Linux command line programs will be assumed. The main focus will be on handling hard disk problems under the Linux operating system. Other *NIX systems may be similar. Windows users might be able to extrapolate some things, but a vendor program is frequently necessary.

Bad sectors can be particularly annoying on Linux (*NIX) systems, not because Linux fails to handle the problem, but because the kernel actually tells the operator that the problem has occurred. In fact, no OS actually gets rid of bad sectors, they just handle them differently. For example, Windows, depending on the version, will either read over the bad sector and give you a corrupt file (leaving you to realize it's corrupt), or complain about some type of read error. Linux stops reading the file at the bad sector and, after timing out several times, returns an I/O error. Of course, both of these are known as finding your bad sector "the hard way".

If you have properly configured your machine, then the SMART feature of your hard disk was already enabled, and the hard disk has been continuously monitoring itself and attempting self-repair. Of course, you also perform regular file system scans to block out bad sectors at the OS level. When you do loose data due to a bad sector you have a backup of your important files at hand. If data security is very important to you, then you were using a RAID 1 mirrored array to store your files. But what are these bad blocks in the first place, and how do they appear?

[glc_engineer]

2.0 Hard Disk Basics

2.1 The physical structure

A hard disk is composed of three basic components:
Platters on which to store the data
Read/Write heads to read and write data
A controller board

The platter is generally composed of three layers; a supportive substrate usually made of aluminum or glass, a thin film of magnetic material, and a thin protective layer. These platters are created by sputtering the magnetic material onto the substrate to a thickness usually around one micron. If you were to view the magnetic domains on the surface of the platter you would see concentric rings called tracks or cylinders, which would be broken radially into sections called sectors. In very old hard (and floppy) disks, the data was stored with a single magnetic domain representing a single bit; however, if the magnetic information in a modern hard disk were to be viewed directly it would appear to be a modulated RF signal, and frequently encrypted with some type of error checking and correcting algorithm.

To record this signal onto the platter and subsequently read it, magnetic heads are suspended a few microns above the surface of the platter on an armature. The heads are positioned over the correct track by an actuator servo which moves the armature. Increased air pressure is created between the platter and head when air moving with the platter impacts the bottom of the head, preventing the head from running into the surface. The actual manner in which different models of hard disk record their data varies from model to model, so a controller board, which "speaks" a common language called ATA, receives read and write requests from the computer and handles the actual process of reading or writing to the platter. The technology of embedding the controlling logic of the hard disk on the drive is called "Integrated Drive Electronics" or IDE, and is not exclusive to drives commonly called IDE hard disks (i.e. SCSI). What is commonly referred to as IDE is really ATA. As an aside, the old MFM and RLL hard disks only contained stepper motors and position sensors on the drive. All of the circuitry to control the reading and writing was contained on a controller board in an ISA slot.

[glc_engineer]

2.2 The logical structure

The physical structure of the hard disk can be quite imposing, but with IDE the modern computer does not have to worry about this. Instead, the modern hard disk can be thought of as a completely separate computer whose sole purpose is to answer read and write requests sent by the disk interface on the mother board.

In the past, the physical and logical structure of the disk were one-in-the-same. A disk was physically divided into cylinders and sectors, with a head for each surface containing data. This led to the logical C/H/S description of the disk to the disk controller, where C is the number of cylinders (AKA tracks), H is the number of heads, and S is the number of sectors per cylinder. A sector always held 512 bytes of data. This was also the original way for the BIOS to "number" the sectors, which resulted in the two subsequent size barriers of 528MB (maximum size of 1024/16/63), and 8.4GB (maximum size of 1024/256/63). The correlation of physical cylinders, sectors, and heads to logical C/H/S no longer exists in modern hard disks, and is a holdover from ancient times. Modern hard disks refer to the sectors (which still contain 512 bytes) by number, a method frequently called LBA. Software can still request a sector using the old C/H/S description, but this is converted into the LBA sector number by the BIOS or OS before the request is sent to the drive. There are two size limitations for LBA, 137GB (from 28-bit address), and 144GGB (that's giga-giga bytes, from 48-bit address). There was also a proposed 64-bit address, but who needs 18GGGB of storage.

How long will the 48-bit address last? The 20-bit address was exceeded in 1995, and the 28-bit address was recently exceeded in 2003, which indicates a bit (or a doubling) every year, so we might expect the 48-bit address to last another 20 years. However, there are two limiting factors to disk size, consumer demand and technological capability. Program space once accounted for 50% of the hard disk usage, but now its only accounts for about 5%. Almost all of the 8-bit increase has been due to the amount of data people want to store.

On the consumer side, most of this size increase has been due to digital movies, music, and pictures, but these are reaching a plateu. In the area of digital music, compression techniques are already optimal, sample rates are dictated by anatomy, and song lengths have been consistent for hundreds of years. Movies might increase their size 4 times in order to accomodate the new High Definition standard, which isn't likely to be improved upon for many decades (or ever) due to human limitations. Pictures are going to max-out at about 10MB per image (compressed) due to optimal compression techniques, limitations of the human eye, and technological hardships in making bigger ones. If we figure that movies have driven disk size, then we would only expect consumer needs to increase by a factor of 4 (or 2 bits). Of course, pak-rat-itis might add another 2 or 3 bits bringing the consumer demand to a maximum size of 4.4TB (33-bit address).

On the research side, disk requirements increase as computing power enables larger and more accurate models. Currently, research based demands are roughly equal to consumer demands at the 137GB per disk level. The relationship between storage and processing power depends on the type of research, but for the most processor and storage intensive research, storage space is proportional to the cube root of processing power. If processing power continues to double every 1.5 years, then storage requirements will double every 4.5 years. This means research needs will add a bit every 4.5 years. In roughly 23 years research and consumer needs will be equal again at 4.4TB per disk.

From a technological standpoint, my collegues have just sought funding to do research which might bring magnetic storage to its highest possible data densities. Assuming we continue to use 3in platters in our hard drives, this maximum will deliver a disk which will offer 5TB per platter per side. Generally, 3 platters tend to max-out a disk drive, so 30TB disks (requiring a 34-bit address) might be possible in the next couple of decades. Of course, it might be infeasable to use such large platters with such high data densities, but that's a problem for the future. Regardless, it appears unlikely that the 48-bit barrier will pose any problem.

[glc_engineer]

2.3 Computer interface

To request data the computer simply passes a read command to the hard disk which contains the sector number to read from. The controller board on the hard disk then decodes the sector number, moves the correct head over the correct area of the correct platter, reads the magnetic data, translates that data back into bits, and sends the sector back to the computer. The process for writing follows very similarly to that of reading, except that the computer sends a write command and the 512 bytes to write, and the hard disk returns its success in writing. The interface is extremely simple in this manner. The computer doesn't need to know anything about the hard disk, except for the total number of sectors.

[glc_engineer]

3.0 Fixing Bad Sectors

3.1 How bad sectors form

There are six ways in which a hard disk can form bad sectors; three of them all relate to defects in manufacturing the platters, the first four are non-repairable, physical damage, and the last two are repairable, soft damage. These mechanisms are:

A hole in the magnetic thin film on the platter when it was coated.
A weak interface between the thin film and the substrate which eventually separates, leaving a hole.
A defect in the substrate which worsens with time.
A mechanical failure or physical abuse which causes a head to contact the platter.
Thermal fluctuations which corrupt magnetic data
Powerloss during a write operation

Recalling from section 2.1, the hard disk platter is a piece of substrate coated with a film of magnetic material. One of the important aspects of thin film epitaxy is that defects in the surface of the substrate show through to the thin film applied on top. For this reason, it is important that the substrate be as defect free and clean as possible. For example, if someone were to touch the surface of the substrate with their bare finger before applying the magnetic film, then you would clearly see that finger print after applying the film. As another example, if there was a small pit in the substrate, then there would be a small pit (or void) in the magnetic film. Both of these are examples of defects that would be seen and repaired at the time of manufacture (during the low level format).

Let's say that the magnetic film managed to bridge the pit in the previous example. In this case the interface between the film and substrate at that point is very weak, and the film will probably break away from that spot as the hard disk ages. This is something that would not be caught during manufacturing, and would probably result in data loss. Another defect that might not show up at the factory is a microscopic fracture in the substrate. As the disk ages these fractures can (and usually do) become larger. The end result is that the fracture shows itself on the surface and a whole mess of bad blocks show up. This is normally the end result because the drive becomes quite unusable at that point, but if you kept it spinning for another few years the platter would probably just break.

The last defect related source of bad sectors is the mechanical abrasion of the magnetic film. This usually happens when a head smacks into the surface of the disk (usually from a jar while operating). It can also happen when a piece of the magnetic film breaks off and skitters across the disk (on its way to the edge) or momentarily jams under the head.

There are also two non-defect related mechanism which can cause Unrecoverable Read Errors, and these are thermodynamics and poorly timed power loss. There is a probability that a magnetic domain on the disk platter can spontaneously change its orientation. The result of this is one or more bad bits that show up when the data is read. This is what ECC is meant to fix, but sometimes the thermal corruption has occurred in just the right way to cause ECC to fail. This failure of ECC can mean that the data is corrupted but the error has not been detected, or it can mean that the error was detected but not fixed. In the first case, the data is returned to your computer and everyone thinks that it's perfectly fine, until you notice that your text file says "ay the power" instead of "By the power". It is very rare that ECC fails in a non-detectable manner. In the second case, the hard disk returns a 0x40 error (read error) instead of the 512 bytes of data.

When hard disks write data, they write a whole sector at a time, even if only 1 byte in the sector has changed. Sometimes, for error checking and performance reasons, they may write many sectors at a time. If the power fails while the drive is writing a sector or group of sectors, then when one of those sectors is subsequently read, the error check will fail and you'll have a small cluster of bad sectors. Some high-end drives are designed so that even if the power fails, a write-in-progress will complete. This is the primary reason for the "It is now safe to turn off your computer" screen in Windows. Who wants to corrupt the important OS related files Windows is updating before it shuts down?

Hard disks only attempt to fix read errors by using ECC. If the error was corrected the disk just returns the data. Some of the better disks might even try rewriting the sector to stabilize the data. If the error was not corrected the disk returns the 0x40 error and the sector it occurred at. The hard disk makes no further attempt at recovering data or fixing the error (because it doesn't know what the data should be). Whether the sector has a media defect or just a thermal corruption is unknown until the computer asks the disk to write to the sector. When the hard disk writes it verifies that it wrote. If the bad sector occurred because of thermal corruption, then the bad data is simply overwritten. If the media has become damaged the hard disk will invisibly remap the bad sector onto a reserve section of the platter, if any are available. If the remap fails the write will also fail, and the hard disk will return a write error.

[glc_engineer]

3.2 Low level formatting
The low level format serves two purposes; 1) It creates the tracks and sectors where data will be stored, and 2) It remaps bad sectors to good ones. In the past, the low level format used to be done by the hard disk controller attached to the computer, with the use of a special program run by the end user. Since the introduction of IDE the low level format has been done at the factory. This is very important to note: true low level formatting cannot be done by the end user. At one time, it was sometimes necessary to reformat a hard disk at the low level once every few years because thermal drift would eventually cause a loss of sync between the heads and the tracks. This is no longer necessary with modern hard disks (let's say anything over 120MB, or made since 1992). Some BIOSes had low level format utilities built into them, but most modern hard disks will ignore any commands sent to reformat them.
No user or computer level actions interact with the low level format of the hard disk. The low level format simply exists to tell the on-drive controller where sectors are located on the physical media. The only thing the computer "sees" is an attached storage device which has an array of sectors with data, and answers read and write requests.

[glc_engineer]

3.3 High-level formatting
The high level format is file system dependent and mainly serves to identify which sectors a file occupies. Most files systems don't even work with sectors, but work with groups of sectors called clusters or blocks. Even disk partitioning could be considered a high level format because the partitioning only tells the BIOS where logical disks exist on the single hard disk. In fact, you don't even have to partition a disk to create a file system on it (of course, only partitioned disks can be boot devices on x86 machines). Most file systems were made with the expectation that unusable sectors would appear on storage devices. They therefore have means to mark bad sectors (or clusters or blocks containing bad sectors) so that they aren't used to store files. This way the user can continue to have data on a device which is no longer defect free, in much the same way that the badmem module allows the use of memory that has developed bad bits. Of course, you wouldn't want to use memory that is developing bad bits, and you shouldn't want to use hard disks which are developing bad sectors.

[glc_engineer]

3.4 Disk diagnostic utilities
So what are these low level format utilities available from the manufacturer? They are better described as diagnostic and zero-fill utilities, and they perform functions to test and "restore" the hard disk. In particular, these utilities identify whether the drive can be seen by the computer, what the SMART feature says about the condition of the drive, and say they do low level formatting. None of them do true low level formatting. What they actually do is write 0's to the start and end of the drive, and sometimes to everything in between. This usually causes the drive to refresh (or remap) all of the bad sectors, and it will sometimes get rid of those pesky bad sectors (along with destroying all of your data). Basically, the drive tries to write 0's everywhere, and the sectors it can't write to get remapped. But most drives already remap bad sectors when they are written to, so this "low level format" rarely fixes bad sectors (unless the drive doesn't remap on write). However, it certainly does clean the drive of the data it contains. Are the handfull of blocks which might be recovered (as usable) worth all of the data on the drive? Usually not, so a file system remap of the bad blocks is generally the better option. In Linux, this is the fsck utility with the badblocks option. In Windows, it's scandisk with a full surface scan.

Most of the functions of these zero-fill / diagnostic utilities can already be done using the computer BIOS, the kernel, smartctl, hdparm, and dd. If the computer BIOS can't see the hard disk at all (and you've triple checked the connectors) then the control board (on the drive) is probably fried, especially if the machine can see another disk and you've moved the questionable disk to the known-good hard disk controller. The drive model information can be found in /proc/hd?/model, or it can be determined using smartctl or hdparm. The current SMART status of the drive can be viewed with smartctl. If, for some reason, you want to fill your drive with 0's, this can be done with /dev/zero and dd. Hard disk errors are reported by the kernel and show up on the console, dmesg, or /var/log/messages. If you want to test a sector reported by the kernel you can use dd to read that sector by skipping to it, and you can use dd to try and write to it (hopefully relocating it), although you shouldn't try this without backing up your data first.

[glc_engineer]

4.0 Conclusion

Many people ask, "how can I low level format my hard disk?" Of course what they are really asking is, "how can I get rid of those bad sectors?" The brief answer is, "you can't." The hard disk should have been remapping bad sectors as they popped up, invisibly moving them to good areas of the disk. By the time you start seeing them it means that the reserve area has been filled up, and the only way to get rid of them is to sweep them under the rug by marking them as "bad" in your file system. Over time, some manufacturing defects do show up, and it is possible for one or two too many to appear. However, no hard disk should have new bad sectors showing up regularly, or at all. This is a sign that something has gone wrong.

Here's a good example of what happens to drives when bad sectors appear. I had a machine with an IDE RAID controller using two 60GB drives in a RAID 1 array. This type of array mirrors one drive to the other. One day, the computer froze and I restarted it, only to get a big warning from the RAID controller that the array was critical. I removed the defective drive and attached it to the regular IDE controller. When I booted the machine and looked in the logs I saw several kernel warnings from the RAID driver. Three days before the machine froze, the controller detected a failure (bad sector) on the failed disk. At this time it completely rewrote the bad disk using the good disk as the image (AKA rebuilding the array). Remember, writing to bad sectors causes their remapping, so the first rebuild completed 100% and the bad sectors had disappeared. About three days later the array noticed another bad sector on the failed disk and attempted to rebuild it, but only got 13% of the way. Why? Because the drive ran out of space to reallocate bad sectors. Within an hour the controller tried rebuilding the array again, but only got 6% complete. I ran badblocks on the failed disk after I pulled it from the array, and then re-ran badblocks. Before my very eyes new bad sectors were appearing. Even if I could somehow reformat this drive at the low level, do you think I'd want to?

In conclusion, it is impossible to low level format a hard disk outside of the factory. If you see bad sectors appearing on the device (outside of ones created by a power failure) then it is time to replace it, because others will soon join it, even if you could do a low level format. Most hard disk warranties are long enough to shake down the