Spare track percentage on a hard disk?

[Philip]

A group of us are refurbishing donated disused lapops for disadvantaged school kids. In order to reassure donors, I need to quantify the risk of personal data surviving DBAN in spared tracks. I'm pretty sure it's going to be very small but I could do with some figures to justify that. Can anyone tell me the proportion of a typical hard drive's capacity that is hidden and reserved as spare tracks? I thought MHDD might tell me if I rummaged through its more obscur options but apparently not. I'm sure Steve would know if he stumbles across this.

 

[DiskTuna]

You can see If and how many sectors were added to G-List, so that were reallocated, just look up RAW values in SMART for reallocated sectors and to be on safe side include pending sectors as well. How many the harddrive sets aside for the purpose is irrelevant. If you guarantee donors you zero fill a drive and do not donate drives with reallocated sector value > 0 (to be checked after zero fill, after zero fill you should not have pending sectors anyway) then they pretty much have a guarantee no personal data remains on drives that are donated to the kids. So then it becomes a trust issue between you and donators.

 

[Philip]

Thank you DiskTuna - that may well tell me the number of tracks that have been spared on a particular disk (though I understand the interpretation of SMART data is manufacturer-specific and not published, hence not necessarily reliable). But that's not actually what I asked.

I might tell a donor of a device that the risk is "extremely low" (or whatever) - trust me. That's almost certainly true but without figures I can't justify it. What is the worst case, where nearly all the available spare tracks are used? Just what percentage of the disk capacity is that, representing bad tracks which may contain recoverable data not wiped by DBAN?

 

[DiskTuna]

I don't understand your problem then I suppose or you do not understand my answer. I know it's not what you asked but I tried to explain why knowing amount of spare sectors is useless knowledge in relation to the problem at hand. Only useful info is whether sectors were actually reallocated or not.

Spare sectors that are not used pose no threat, only those that were reallocated to do. I don't see how knowing exact number of spare sectors tells us anything about risk to client data as they are empty until used. A spare sector contains nothing until it is 'swapped'. If a sector is reallocated this is recorded in raw SMART value. No reallocations means no spare sectors were used, regardless the amount of sectors set aside for the purpose of reallocation.

With regards to SMART standards: This is exactly why I suggested to not donate any drives to kids IF any of following SMART attribute raw values > 0: reallocated sector count, pending sector count and reallocation events. If these are all zero, no client data was ever written to spare sectors so we do not need to worry about them being erased or not.

where nearly all the available spare tracks are used? Just what percentage of the disk capacity is that, representing bad tracks which may contain recoverable data not wiped by DBAN?

In that case you'd see large values for reallocated sector count, reallocation events and this would indicate it's not safe to donate the drive as these can not be wiped using DBAN.

Again, if reallocated sector count, reallocation events raw values are zero then you can guarantee all user data was wiped using DBAN. Number of available spares means nothing with regards to potentially recoverable user data, it's the ones that are used that count and these are recorded by SMART.

So:

IF reallocated sector count = 0 AND reallocation event count = 0 AND DBAN Erased THEN
safe to donate to kinds = TRUE
ELSE
safe to donate to kids = FALSE
END IF

Edit: Maybe I should explain sector reallocation.

SPARE SECTOR POOL: excess sectors reserved to replace bad sectors, these are OUTSIDE user addressable LBA and can therefor not be accessed by DBAN. This is NOT a problem as these sectors are EMPTY.

BAD Sector triggers reallocation upon write: BAD SECTOR is taken out of user addressable LBA, a spare sector is ADDED to user addressable LBA.

So what we see is spare sectors are NOT the problem. The problem is the reallocated BAD sector that is now outside of addressable LBA and therefor can NOT be wiped by DBAN.

 

[Philip]

Thank you again, but forgive me - you still haven't understood what I'm getting at. I'm not looking at a specific hard disk but the general case in order to enable me to write a reasoned policy and procedures document. Note that DBAN is not acceptable in UK Government use bcause it only cleans LBA-addressable sectors. You have to use Blancco or some other accredited utility, which also wipes spare sectors and ex-LBA sectors that have gone bad and been spared out (grown defects). It's also why Steve has Beyond Recall as a slated future project - his knowledge of the ATA command set would allow him to do a much more thorough job than DBAN.

Donor: I've got this laptop I'd give you but I'm worried about my personal data.

Me: Don't worry, we'll wipe it with DBAN. But I have to tell you there is a very small risk of residual data remaining.

Donor: <Sharp intake of breath> Hmmm... Can you quantify it? I'm a bit paranoid and need to know just what the risk is before I agree to let you touch it.

Me: Well, if a sector mis-reads, even if it's only a transitory problem due to static, the disk may mark the sector (containing your data) as bad and reallocate it. Forensic programs might still be able to read it.

Donor: Err, well, we get quite a lot of static. My disk is fairly full but I reckon just a proportion n of my data might be sensitive. So if the spare sectors are a proportion m of the advertised size of the disk, in the worst case where it's used nearly all its spare sectors, the chance is no greater than n * m that some of my sensitive data might survive. I reckon a one in a million chance might be within my risk appetite. So what exactly is the the value of "m"?

Me: Not actually quite sure about that. I might have to ask @Steve.

I don't understand your problem then I suppose or you do not understand my answer. I know it's not what you asked but I tried to explain why knowing amount of spare sectors is useless knowledge in relation to the problem at hand. Only useful info is whether sectors were actually reallocated or not.

...snip...

SPARE SECTOR POOL: excess sectors set reserved to replace bad sectors, these are OUTSIDE user addressable LBA and can therefor not be accessed by DBAN. This is NOT a problem as these sectors are EMPTY.

- unless of course Spinrite has determined that the spared sector is ok after all and un-spared it!

 

[DiskTuna]

Thank you again, but forgive me - you still haven't understood what I'm getting at. I'm not looking at a specific hard disk but the general case in order to enable me to write a reasoned policy and procedures document. Note that DBAN is not acceptable in UK Government use bcause it only cleans LBA-addressable sectors.

I understand what you're getting at. And I gave you a policy. I understand it only cleans LBA addressable sectors and those are all you need to worry about if no sectors were reallocated. It's you who doesn't understand the concept of sector reallocation.

If sectors were reallocated, which you can check using raw SMART data for attributes I gave, whether it was just ONE sector or 1000 sectors, then you can not guarantee all data was wiped so you need to destroy the disk.

You're stuck in some kind of scenario where you think it is useful to know the capacity of the spare pool. It isn't, not for this anyway. What IS relevant is amount of reallocated sectors as you can not wipe those. Spares that were never put to use do not contain user data by definition so you don't need to worry about wiping them.

I am not talking about one specific hard drive. I am giving you a strategy / policy you can apply to ANY hard drive.

TLDR. If donor is paranoid about this data he should instead destroy the drive or hang on to it. You can guarantee you'll erase all user data as long as drive has no reallocated sectors. If you can not wipe all sectors you can guarantee to destroy the drive.
Even in event there are some remapped sectors you'll be needing specialized equipment to attempt to read those sectors.

You have to use Blancco or some other accredited utility, which also wipes spare sectors and ex-LBA sectors that have gone bad and been spared out (grown defects).

You can not wipe reallocated sectors using DBAN. So therefor my recommendation to NOT pass on drives with ANY reallocated sectors. Drives with NO reallocated sectors are safe to donate.

One thing I'd like to add, check if DBAN detects and can disable HPA and DCO if you need to wipe those too. In general these will not contain user data though. These are also outside of LBA space but can be modified using ATA commands.

Blancco as per their documentation can only erase remapped sectors if the drive supports it (enhanced secure erase). Actual wiping is not done by the software but by the drive firmware itself. So IF remapped sectors can be erased is not some advanced feature of the software but of the drive itself.

It's also why Steve has Beyond Recall as a slated future project - his knowledge of the ATA command set would allow him to do a much more thorough job than DBAN.

Doesn't matter, ATA commands are publicly available. If a drive has no reallocated sectors, you do not need to worry about pool of spares containing user data. ATA command set does not offer anything to access reallocated sectors though some drives reset G-List using vendor specific format unit command. As a consequence these sectors will then end up in P-List. And although G-List is now cleared you simply moved the problem. In addition a drive may support enhanced secure erase ATA command which will wipe remapped sectors. This command affects entire drive, you can not secure erase only specific sectors.

Donor: I've got this laptop I'd give you but I'm worried about my personal data.

Me: Don't worry, we'll wipe it with DBAN. But I have to tell you there is a very small risk of residual data remaining.

No. Again IF you discard, dispose of drives, destroy them, with reallocated sectors there's no risk. Only drives WITH sectors reallocated you can not guarantee all data is wiped as those reallocated sectors are out of LBA space.

Donor: <Sharp intake of breath> Hmmm... Can you quantify it? I'm a bit paranoid and need to know just what the risk is before I agree to let you touch it.

Me: Well, if a sector mis-reads, even if it's only a transitory problem due to static, the disk may mark the sector (containing your data) as bad and reallocate it. Forensic programs might still be able to read it.

Yes, and AGAIN, this is why you guarantee your donors you will destroy drives with reallocated sectors. That's your policy. Drives without sector reallocations you can guarantee you will wipe every sector that has ever been used.

Donor: Err, well, we get quite a lot of static. My disk is fairly full but I reckon just a proportion n of my data might be sensitive. So if the spare sectors are a proportion m of the advertised size of the disk, in the worst case where it's used nearly all its spare sectors, the chance is no greater than n * m that some of my sensitive data might survive. I reckon a one in a million chance might be within my risk appetite. So what exactly is the the value of "m"?

Just dispose of ANY drives with ANY reallocated sectors, reallocation events and sectors pending reallocation. If raw values for those attributes are zero after DBAN wipe, you can guarantee all data was wiped.

Me: Not actually quite sure about that. I might have to ask @Steve.

...snip...

- unless of course Spinrite has determined that the spared sector is ok after all and un-spared it!

No, Spinrite can not 'unspare' a reallocated sector, that's just a ridiculous statement and illustrating total misunderstanding of sector reallocation and what Spinrite can and can't do. Spared sector isn't even a thing. It can not even reach it as it is outside LBA addressable space. Spinrite does not reallocate sectors, nor can it undo this. It can also not address the spare area.

It's the drive firmware that does the reallocations, not Spinrite. It can trigger reallocation as most drives reallocate pending sector upon write. But DBAN wipe will have same effect on the pending sectors as wiping implies writing to the sector.

A spare sector isn't anything special, It's an ordinary sector that is kept in reserve, outside of LBA space. Once a sector is reallocated the only thing that happens is that it's address is added to a table in the SA, and in addition a pointer to the spare is added to this table. The bad sector goes nowhere and nor does the spare. It is simply a table in the SA being updated mapping the previously spare that was outside LBA space to LBA space while at the same time removing the 'bad' sector.

To edit this table or to read content from reallocated sectors you need sophisticated equipment like PC3000 complex or DFL. This equipment only works with supported drive models and families as each of these have the firmware to be reverse engineered. So it's impossible to write some generic software only tool that will be able to do this.

 

[SeanBZA]

Question is, a reallocated sector will be what, 4k, 8k, 16k of data. Just what part of a sensitive file from the most common document types ( zip, JPG, any office document) will you be able to actually use as just a 4/8/16k section, out of perhaps the smallest document. Smallest Openoffice document I have is 20k, and it is basically a blank table with a touch of text, and as most document types these days are stored as some form of compressed archive, you will not get much out of them, as only the header and the first part will be surviving in the recovered sector, and it would have to be the one with the header to be able to use it in any sort of form.

You are not going to recover much from the tiny snippet, especially as you will have no context as to file type, and where it was in the original file. Might be enough to prove that a certain file was there, but the overall chance of it being useful is vanishingly small. If you are worried about this, it is trivial to use full disk encryption, or if you are on Linux use /home encryption, which only encrypts the data you store, not the whole OS, which has a good speed advantage, even on modern hardware.

Yes good to erase all data, but in most cases the scraps are almost unusable, I would worry more about scraps of data left in slack space, as most OS versions use a buffer to store data before it is written to dick, and the buffer is often not erased fully after completing a write, so every partial cluster write does write out, completely by default, the full uncleared data that was there before, and this is faithfully copied to disk as well. You final bits of log files and such, that windows and all OS are so determined to keep, always contain a chunk of data that was currently written before, and these scraps are often going to be around for a while in the tail of the log files.

 

[DiskTuna]

Yes good to erase all data, but in most cases the scraps are almost unusable, I would worry more about scraps of data left in slack space, as most OS versions use a buffer to store data before it is written to dick, and the buffer is often not erased fully after completing a write, so every partial cluster write does write out, completely by default, the full uncleared data that was there before, and this is faithfully copied to disk as well. You final bits of log files and such, that windows and all OS are so determined to keep, always contain a chunk of data that was currently written before, and these scraps are often going to be around for a while in the tail of the log files.

This is of no concern if you wipe disk start to end. And this "and the buffer is often not erased fully after completing a write" is likely to be untrue. Because if my line of work I frequently snoop around in raw disk data using a hex editor and if I for example look at a JPEG as it is out on disk I never observe such stale data after the end of file even if last cluster of file is only partially used.

You are not going to recover much from the tiny snippet, especially as you will have no context as to file type, and where it was in the original file. Might be enough to prove that a certain file was there, but the overall chance of it being useful is vanishingly small.

And there's the astronomically small chance it's a text file with the password to his bitcoin wallet, or credit card number. But you are correct, if donor is so worried he should either not donate the drive or should have encrypted it etc.. But yes, chances are tiny those remapped sectors contain anything useful, and you need quite sophisticated stuff to even be able to read them. Not the type of equipment you find in classrooms.

From from the donation project point of view you want to be able to present a solid protocol. So then I repeat, remapped sectors are only a problem if they're there and if they are not DBAN will all user data. Problem is OP does not understand sector remapping and worries about never used spare sector pool.

 

[Philip]

Having heard every SN episode since #1, I'm sure Steve has said that if a track is automatically spared out by a drive as a result of a transitory error, Spinrite level 4 can test the orginal track, and if it looks good after all, can (or can tell the drive) to "unspare" the track. In that case you would have a non-LBA sector potentially containg user data.

Level 4 saves the content of a track before hammering it. Since every LBA sector potentially contains user data it can only save it to a non-LBA track - it has nowhere else. If you pull the plug while it's doing so, your data won't be lost because the drive has remapped that sector to the spare sector where Spirite (or the drive at Spinrite's prompting) put it. Blancco and other government accredited tools can similarly access and wipe non-LBA sectors, presumably in the same way as Spinrite accesses them (and BeyondRecall will), whereas DBAN runs under a Linux kernel and accesses a disk through the Linux drivers, which only give you access to LBA tracks.

As a retired government-accredited security consultant it was my job to be paranoid so I could tell my clients when they didn't need to be. Or more often when they needed to be a good deal more paranoid than they usually were! In those circles, we were looking for proveably secure solutions. You had to assume that any bit of magnetic coating that could possibly store data potentially would. That's why Blancco is accredited by DBAN isn't.

Obviously (and you don't need to tell me), that's a million miles from the situation with our charity, but habits die hard. For my own satisfaction and integrity I still want to know how many spare tracks there typically are which might, however remotely, contain user data. (And I still want to know whether there's intelligent life amongst the stars. What practical use would the answer be to me? Absolutely zero.) You'd never have thought Heartbleed or Spectre/Meltdown could possibly leak private keys, but they can. Everything that can possibly happen, will, given long enough.

 

[DiskTuna]

Having heard every SN episode since #1, I'm sure Steve has said that if a track is automatically spared out by a drive as a result of a transitory error,

What does this even mean? 'A track is spared out' sound like mumbo jumbo, try explain in your own words and you'll see what I mean.

Spinrite level 4 can test the orginal track, and if it looks good after all, can (or can tell the drive) to "unspare" the track. In that case you would have a non-LBA sector potentially containg user data.

There seem to be some misunderstandings then. Like 'spare sectors' and 'reallocated sectors' or 'remapped sectors'. And further you appears to be mixing those up with 'bad clusters' at the file system level, which indeed in case of FAT Spinrite can return to service.

On a modern drive Spinrite can not touch spare sectors, nor can it change the status of already reallocated sectors.

Level 4 saves the content of a track before hammering it. Since every LBA sector potentially contains user data it can only save it to a non-LBA track - it has nowhere else. If you pull the plug while it's doing so, your data won't be lost because the drive has remapped that sector to the spare sector where Spirite (or the drive at Spinrite's prompting) put it.

No. Drives themselves leave data in 'bad' sectors, thus allowing us for a chance to try recover it. The drive will only replace if we write to it. Writing is like signaling the drive, we give up on this data. Drive will now reallocate the sector. Indeed this means the old data is now in a sector outside of LBA space, but it is still in this same problematic sector, containing same unreadable data. So, EVEN if we could reach it in an attempt to wipe it, we still couldn't if there is a fundamental issue with the sector. This at the same time is an issue for the ATA enhanced secure erase. There's no guarantee it can actually wipe the reallocated sector. Reallocation or remapping took place due to the pending sector was unable to being written to (if we can actually write data to the pending sector it will not be remapped). To wipe a sector we need to write to it, enhanced secure erase will have the same problem to write to remapped sector.

And the spare sector is where the NEW data ends being written. So the formerly spare sector is now remapped to a LBA address.

Blancco and other government accredited tools can similarly access and wipe non-LBA sectors, presumably in the same way as Spinrite accesses them (and BeyondRecall will),

No, all they can do is issue an ATA enhanced secure erase command. No this is not the same way Spinrite accesses them, Spinrite can not access reallocated sectors.

whereas DBAN runs under a Linux kernel and accesses a disk through the Linux drivers, which only give you access to LBA tracks.

Yes. Same goes for Spinrite using extended int13h or ATA commands.

As a retired government-accredited security consultant it was my job to be paranoid so I could tell my clients when they didn't need to be. Or more often when they needed to be a good deal more paranoid than they usually were! In those circles, we were looking for proveably secure solutions.

I am not easily impressed by job titles.

Okay. If paranoid, then destroy drive. But this isn't what you were asking. With regards to the project:

1 Forget about spare sectors. Spare sectors do NOT contain user data.
2 Check if sectors were reallocated (SMART, raw values)
3 If there are reallocated sectors do NOT donate drive
4. If there are NO reallocated sectors nuke drive with DBAN
5 Now check again for reallocated sectors, if any do NOT donate drive
6 If all SMART attributes related to sector reallocation are zero (raw value), drive is safe to donate.

You had to assume that any bit of magnetic coating that could possibly store data potentially would. That's why Blancco is accredited by DBAN isn't.

That's false security, because in event of reallocated sectors even enhanced secure erase can not guarantee these are wiped.

Obviously (and you don't need to tell me), that's a million miles from the situation with our charity, but habits die hard. For my own satisfaction and integrity I still want to know how many spare tracks there typically are which might, however remotely, contain user data.

Again, spare sectors do not contain user data so how many of them are is irrelevant.

(And I still want to know whether there's intelligent life amongst the stars.

Just life for starters would be cool. I do not doubt it.

What practical use would the answer be to me? Absolutely zero.)

Yeah, but understanding the concept of spare sectors and sector remapping would actually help in this context.

You'd never have thought Heartbleed or Spectre/Meltdown could possibly leak private keys, but they can. Everything that can possibly happen, will, given long enough.

Yeah, but at this level of paranoia it is NEVER safe to donate a drive.

Edits: Sorry keep spotting typos that may cause confusion.

 

[Philip]

This thread has becom nugatory. You seem to have no interest in understanding my point of view but only in berating my intelligence.

One tiny point before I un-watch it: the same national intelligence agency that says it trusts Blancco but not DBAN also says that ATA Secure Erase is not reliably implemented on all drives and hence no better (possibly worse?) than DBAN.

And I've never mentioned bad clusters and I'm perfectly well that they are at a totally different level of data abstration.

 

[DiskTuna]

Let's try this:

Think of a hard drive as huge pool of sectors. These all have an address, the physical address or PBA.

LBA is a pool of sectors for which a table exists that map LBA addresses to PBA addresses. We call this the translator.

Any PBA addresses that aren't mapped in LBA can not be accessed. The spare sectors you keep referring to are NOT mapped. Hence NO user data can be written to them. So from a standpoint of security these are of no interest.

Only if a sector is moved out of LBA due it being bad, this sector is a potential security breach hence my recommendation not to donate the drive in this case as you can not guarantee it will ever be wiped. In this event also a spare will be mapped to LBA, and it being in LBA space means it's no longer a security risk.

 

[DiskTuna]

This thread has becom nugatory. You seem to have no interest in understanding my point of view

I say it is you not having interest in what I am trying to explain, a workable solution for your specific situation, because you have to abandon this idea of spare sectors somehow being relevant.

but only in berating my intelligence.

I have been trying over and over to get a point across. It is you that seems to be doubting the validity of my points and have been incorrectly paraphrasing them.

One tiny point before I un-watch it: the same national intelligence agency that says it trusts Blancco but not DBAN also says that ATA Secure Erase is not reliably implemented on all drives and hence no better (possibly worse?) than DBAN.

With regards to accredited or not solutions: I have read those in the past (NIST), little interest in them now, but their 'recommendations' were always in relation to a class of data secrecy and highest risk levels never were covered by any software only solutions. So in itself I can understand Blanco offering added value compared to DBAN for example with it's ability to issue an ATA enhanced secure erase command. And again, even enhanced secure erase will be unable to wipe a sufficiently worn out sector. In itself the phrase 'accredited' has little meaning without context.

And I've never mentioned bad clusters and I'm perfectly well that they are at a totally different level of data abstration.

My bad then, sorry. But bad clusters in a FAT file system is the only thing Spinrite can put to use again.

EDIT: Someone appears to have restricted my options to post an additional reply, reason: unfriendly interaction. I have edited that part now.

Point is, I am told I am misunderstanding or have no interest in understanding OP while I suspect OP is misunderstanding my answers and the relevance to the issue/problem he raised. Maybe I was a tad harsh but reporting my post because of that is rather childish, specially if I explain where the misunderstandings are.

 

[Philip]

Someone on another forum provided a meaningful answer: on a 8TB Seagate drive, a proprietary utility reported spare sectors numbering roughly 0.045% of the LBA size. Whilst a sample size of 1 has its limitations that's a good starting point.

But people have kept telling me there's no way of accessing non-LBA sectors. That's at variance with 2 facts that I've picked up from Steve talking in-depth about Spinrite some years ago:

1. Spinrite is coded ultra-defensively, so that even if you pull the plug on Level 4 at any time, Steve guarantees you'll not loose any data. So where does he save a sector he's working on if not in a non-LBA sector? (I'm not talking about pulling the plug on the drive - only on Spinrite.)
2. Steve has long had an ambition to implement BeyondRecall because he claims he can sanitise areas that tools like DBAN can't. DBAN sanitises all LBS sectors, so what else?

So what deep magic does he use, or are all those nay-sayers wrong? Looking at ATA/ATAPI-7 V1 I don't see it.

 

[PHolder]

so that even if you pull the plug on Level 4 at any time, Steve guarantees you'll not loose any data.

This is not possible to guarantee. Modern drives have caches, and they won't listen to Steve or anyone else telling them how to disposition their data. If you allowed the caches to be fully disabled, then the drive would be so slow you'd never want to use SpinRite on it.

 

[SeanBZA]

Simple way to implement the storage is to use unallocated clusters, either those spares used to align a partition table, or just file system spare space. Those are available and unused, so will be a convenient place to assign to store scratch data, as you only write a block there, then do the suite of tests, and then write the data back from a memory store afterwards, then write the next block to the on disk cache. That way you have a block on disk where you have the current block being worked on, the data there and likely a magic signature used to identify it to Spinrite. Only time this breaks down is on a drive with an unknown file system, so there the default likely is not to write any data as cache, and simply hope there is no power loss during operation, though of course Steve likely also assigned a NMI for power fail detection, to do a last ditch write of the buffer to the disk and a flush cache command on detecting power loss, so there is at least 50ms of power available for the drive to complete the writes, and then enough time for the drive to return a completed status before it is commanded to shut down, instead of the normal OS just abandoning the drive to do an uncommanded power down.