SSD Health: Bit Rot in read-only areas is Quite Real! A SpinRite opportunity?

[Papa Pete]

(I'm an OLD spinrite hand. Been away too long. My apologies! I sent this to support at grc way back in 2018... probably lost in the pile )

SSD Health

I recently discovered that one of Steve's often stated assertions about how drives detect and correct errors is **incorrect.**

What he has often said:
- A level one scan to read an entire drive...
- Will help the drive discover any weak areas
- And when necessary, it will map out a bad spot and move the data to a good area

Recently experienced symptoms:
- Entirely good SSD
- Suddenly block zero went bad and things got worse from there
- All diagnostics said the drive is dying
- Unfortunately, it was mSATA in an MS Surface Pro
- After making a mirror image of the entire drive (ddrescue)...
- I attempted to rebuild the GPT and partition map, since that was the only real problem
- And I got a HUGE surprise: suddenly the drive had no bad spots and was perfect according to all diagnostics!

My research showed:
- SSD's can degrade in areas where they are only read and not written (in fact fascinating IBM research shows *nearby* areas can go bad! See below.)
- Current firmware does NOT necessarily detect nor fix-on-read
- HOWEVER, it immediately detects and fixes-on-write!
- SO, by attempting to overwrite block 0, the drive literally healed itself!

I am thinking that a cheap/free little utility could be written, perhaps part of SpinRite?
- Focusing on the obvious static parts of a drive (eg boot sector, GPT table, etc)
- Rewrites data every N months
- NOTE: The closest thing I've seen out there is a little free Windows utility: "Disk Fresh" nicely rewrites entire SSD's, and schedules itself every N months.

IMPORTANT evidence:
- A quite interesting (reasonably technical) research paper (note graphs pp 23-25) -- you can find this at
https://web.archive.org/web/20190622101205/http://smorgastor.drhetzler.com/library/
- Original link: http://smorgastor.drhetzler.com/wp-content/uploads/2014/08/SSD-Reliability.pdf
- (NOTE: Dr Steven Hetzler is no longer an IBM Fellow. He's now a senior guy over at Meta...)


Among other insights:
- Consumer SSD's are likely to see read bit-rot within a year
- Enterprise SSD's are worse: a few months (they are designed for many overwrites...)
- Thus, do NOT count on SSD's as long term storage!
- And so far, I've not found ANY manufacturers willing to discuss this. Nobody talks about mitigating this... even though it is seemingly a simple issue that COULD be addressed.

 

[peterblaise]

Papa Pete 2025-02-02 ...​

- Consumer SSD's are likely to see read bit-rot within a year​

- Enterprise SSD's are worse: a few months (they are designed for many overwrites...)​

- Thus, do NOT count on SSD's as long term storage!​

- And so far, I've not found ANY manufacturers willing to discuss this. Nobody talks about mitigating this... even though it is seemingly a simple issue that COULD be addressed.​

- - - - -

Thanks for your insight.

I reviewed your experience, and I interpret my experience of similar SSD behavior this way:

- not 'bit rot' as much as sector access delays - the data is there, just not always easily or quickly readable,​

- 'consumer' versus 'enterprise' is too imprecise a demarcation,​

- SSDs do hold data forever, they just read it back slowly when they're 'tired',​

- one SSD manufacturer offers what appears to be a SpinRite-Level-5-like self-maintenance program, a clue that at least one manufacturer 'sees' a problem, and is trying to figure out a way to deal with it.​

- - - - -

I have found that each SSD is it's own game.

All are different, and most are reliable data storage-wise but unreliable performance-wise.

That is, I have some SSDs that reliably read data, but eventually as slowly as 5 megabits per second, not the OOB Out of Box 500 MB/s promise that I saw on day one.

ALL SSDs I have seen appear to trudge on under SpinRite 6.1 LEVEL 5 without damage to the drive, performance, or compromising data - they do not die, even though they feel like they are going to, I see them going slowly or even stopping for any length of time then resuming.

I have found that each SSD is a special case, so I have to heuristically tweak SpinRite 6.1 settings, looking for the settings that seem to best 'fit' the drive.

Some SSDs take SpinRite 6.1 LEVEL 5 with no performance compromise end-to-end for the entire SpinRite LEVEL 5 run.

Some SSDs feel sloggy under a direct SpinRtite 6.1 LEVEL 5, slowing down on various sectors or sector ranges - whatever the drive is apparently internally mapping at the moment.

Some behave better with SpinRite 6.1 LEVEL 4 XFER 1024 or XFER 2048.

Perhaps I am hunting to find the best data transfer size to fit inside an SSD's cache management, or the next largest data transfer size to 'break' a drive's cache and require the cache to be continuously interrupted and re-filled, or something else?

Folks with chip-savvy might understand better the possible churn going on inside any number of SSD controllers and cache and whatever management chips, plus the storage chips and supporting amplifiers, the dynamics of interface, BIOS, and drivers.

Maybe it's a timing thing, and we need to go slower during a full drive rewrite.

But, as I have explored, each SSD seems unique.

Maybe I'll try the SpinRite HASH command line option next time to see if it slows SpinRite down enough to allow the drive to do whatever it does internally between reads and writes.

- - - - -

I have found that SSDs don't stay 'fixed', performance-wise.

Some SSDS feel like they are 'fixed' by re-writing, and some of them then cascade back to minimum prior poor performance rather quickly.

Here's a KinDian S280 240GB SATA3 SSD before and after four differently-set Spinrite re-write passes, then decaying after only 4 read-only passes:

- - - - -

And the SpinRite log doesn't tell the whole story of the experience of watching the progress speed up, slow down, stop, struggle, speed up, slow, down, stop, struggle.

I found a better correlation of my moment-to-moment 'feel' of drive responsiveness by looking at the data transfer and sector access response curve graphs produced by such programs as free HDD Scan, HD Tune, and others.

Here's an HDD Scan of a WD Green 2TB NVMe, I note the contrast between the responsiveness of the data transfer versus the responsiveness of the sector access - as if it's quick to get to each sector, but then slow to read that sector's data:

I note that the 'average' performance on that well-used NVMe SSD is about half of it's peak performance, where peak tested performance probably appears in the graph in as-yet unused areas.

Yet even the slowest performance of that NVMe SSD is faster than a SATA3 HDD.

Hence our happiness with NVME SSDs.

An SSD at it's worst may still be faster than an HDD at it's best.

I compare this to HDDs that are slower than SSDs in both sector access and data transfer, but HDDs appear to never be as variable, sector-to-sector, but HDDs usually are just half as responsive at the end compared to peak performance at the front, a smooth curve slowing down:

- - - - -

So, on the topic, "... SSD Health: Bit Rot in read-only areas is Quite Real! A SpinRite opportunity? ...", I'd suggest defining our terms.

Let me Google 'bit rot' for me:

- "Bit rot" refers to the gradual corruption of digital data over time, where individual bits that make up a file change from their intended state (0 or 1), leading to data becoming partially or completely inaccessible or unusable; essentially, it's the slow deterioration of data stored on physical storage media like hard drives, often caused by aging components and environmental factors, also known as "data decay" or "data degradation.". Key points about bit rot:​

• How it happens: Individual bits within a digital file can flip from 0 to 1 or vice versa due to physical degradation of the storage media over time.
  
  
• Impact: Corrupted data can become unreadable or lead to errors when accessed.
  
  
• • Factors contributing to bit rot:
    • • Aging storage devices
      • Extreme temperatures
      • Environmental factors like humidity
      • Frequent read/write operations on a storage device

How to prevent bit rot:​

• • Regular backups:
    Regularly back up important data to different storage mediums to mitigate data loss if one storage device fails.
  • File integrity checks:
    Use checksum tools like MD5 or SHA-256 to verify if a file has been corrupted
  • Proper storage conditions:
    Store data in a cool, dry environment
  • Consider archival storage solutions:
    For long-term data preservation, use specialized storage media designed for longevity

- - - - -

OK, given that we each got all our data back, perhaps 'bit rot' is not an appropriate term.

So:

What is going on when an SSD becomes poorly responsive, but 100% recoverable nevertheless?​

​

What 'tool' or process, if any, confirms and or repairs whatever is going on with slow SSDs?​

- - - - -

Regarding drive 'refresh', I suggest:
- cloning to another drive, as slowly as it takes for compromised drives,
- then using the new cloned target,
- and then preparing the original source drive for reuse by SE secure erase.

For permanently surface-mounted drive, we need an alternative, maybe:
- cloning out,
- secure erasing,
- then cloning back?

- - - - -

Does that all make sense?

Is there more?

- - - - -

Thanks for the opportunity to explore this and share.

.

 

[peterblaise]

Also, there are extensive discussions of SSDs and SpinRite:

- on GRC.com/Groups:
https://www.google.com/search?q=ssd+site:grc.com/groups/

- on Forums.GRC.com:
https://www.google.com/search?q=ssd+site%3Aforums.grc.com

.

 

[DanR]

I recently discovered that one of Steve's often stated assertions about how drives detect and correct errors is **incorrect.**

What he has often said:
- A level one scan to read an entire drive...
- Will help the drive discover any weak areas
- And when necessary, it will map out a bad spot and move the data to a good area

Could you provide a reference to this???

This is contrary to my understanding of Level 1.

My understanding is that Level 1 is read only. L1 will read all areas of the drive, mark any unreadable areas with a U on the GSD display, but will take no further action. That is, Level 1 does NO writing!

Level 2 (recovery ) will do it's best to read all areas of the drive. Hard to read areas can be re-written (refreshed) by Level 2.

Level 3 (maintenance) will read and re-write (refresh) ALL areas of the drive thus restoring the drive to optimum performance.

- I attempted to rebuild the GPT and partition map, since that was the only real problem
- And I got a HUGE surprise: suddenly the drive had no bad spots and was perfect according to all diagnostics!

SpinRite Level 3 would have done precisely that.

Unfortunately SpinRite 6.1 will not work on UEFI boot only Surface devices.

- SSD's can degrade in areas where they are only read and not written

Right. This is well known. It is called Read Fatigue. This is why periodically re-writing read only areas of an SSD is considered good normal procedure (SR Level 3).

- Current firmware does NOT necessarily detect nor fix-on-read
- HOWEVER, it immediately detects and fixes-on-write!

Agreed!

I am thinking that a cheap/free little utility could be written, perhaps part of SpinRite?

SpinRite does this. It is Level 3. However, it does require some knowledgeable manual effort by the user. When combined with a benchmark (e.g. GRC's ReadSpeed), SpinRite Level 3 can be confined to just those areas of the drive that need it while not subjecting other areas of the drive to unnecessary write activity.

do NOT count on SSD's as long term storage!

With proper care and maintenance SSD's will be fine for long term storage.

SSD's are subject to degradation from frequent reading (Read Fatigue) and as well as long term storage (Bit Rot). Judicious periodic use of Level 3 maintenance will effectively address both of these concerns.

Final Observation: My comments above apply to SSD's in general. More specifically however, they cannot be used with SpinRite 6.1 on a UEFI boot only Surface device as SR 6.1 is NOT compatible with UEFI booting. That would require SpinRite 7 Pro - perhaps 2-3 years away?

Thus, as the OP noted, other means would currently be needed for a Surface device.

A simple SpinRite utility, as the OP suggested, is not possible at this point as SpinRite does not yet have UEFI boot capability.

 

[Papa Pete]

Sorry, I couldn't post at all if I included the links the first time. Let me try again...

LINKS: Here's an archived link to an informative presentation. Page 24 is quite interesting. WARNING: this is more than a little technical. Sadly, it is too big to attach.

Wayback Machine

There is a LOT of other info at the same archived site. Sadly no long live-online. I've got most if not all of it downloaded in case archive.org goes away.

• @peterblaise it's NOT about "slow reads" -- his testing goes to an NRRE - Non Recoverable Read Error, using modified firmware to do direct Host Managed Interface (HMI) so he's in full control of the process and can see the errors developing.
• In my case, yes I could recover because it was "only" the partition record that had completely failed. By rebuilding it, all was well.

OLD? ALREADY DISCUSSED? Perhaps... yet in searching, including the links you kindly provided Peter, I've found nothing approaching what I'm talking about.

This isn't just degraded read speed. It's actual bit destruction, ie real Bit Rot (yes, nice definition there ).

SR LEVEL 3: Yep, sounds like an appropriate capability. My thought: while I used to do that (offline SR) on spinners whenever needed... I rarely if ever wanted to do "maintenance" rewrites. But now with SSD's, I absolutely 100% DO want that. Thus, this is a case where having a utility that works in the background on a live system would be Rather Nice. I have a number of drives in production that simply cannot afford to be offline regularly.

 

[peterblaise]

DanR "... contrary to my understanding of Level 1. My understanding is that Level 1 is read only. L1 will read all areas of the drive, mark any unreadable areas with a U on the GSD display, but will take no further action. That is, Level 1 does NO writing! ..."

I'm guessing that when Papa Pete wrote "... A level one scan to read an entire drive ... Will help the drive discover any weak areas, And when necessary, it will map out a bad spot and move the data to a good area ...", 'it" means the drive itself will reallocate. Is that your understanding, @Papa Pete? We know this to be false. We know an SSD drive will ONLY reallocate on a WRITE request, not on a READ request.

- - - - -

Regarding 'bit rot' on an SSD, I've have seen one and only on sector that could not be read no matter what I did, requiring a SPINRITE LEVEL 3 or 4 or 5 DYNASTAT 0 ( no need to wait for DYNASTAT 5 minute default ) - and that's after testing dozens of SSDs, new and old - so maybe I have seen one sector of SSD 'bit rot' after all, who knows? How would I know what happened?

Otherwise, all other SSDs I have tested return ALL data, even if at only 5 MB/s, and work and survive under any LEVEL of SPINRITE rewrite..

.

 

[peterblaise]

​

@Alice "... The great Peter Blaise ..."​

Thank you, blush.

- - - - -

Please do share your own experience testing SSDs with SpinRite.

Everyone, pitch in, and share experiences and links on the topic.

I'm sharing my experience, and I treasure the personal experiences, testimonies, and supporting evidence shared by others.

And, as Steve Gibson asked, duplicate and document.

@ColbyBouma generously has a repository of any SPINRITE LOG records we've shared.

Are there any LOG records that show SSD single sector full or partial recovery?

https://gitlab.com/GRC-Community/SpinRite-Logs​

In my own testing, I've seen one unreadable SSD sector out of billions of sectors under SpinRite read/write testing.

I still have no recorded documentation of what went wrong, and what actually happened when it got fixed.

Yeah, it got fixed, and it has stayed fixed.

Does anyone have any clues, any tools for 'knowing' what happens when an SSD can't read a single sector at all, or can't read the entire contents of one sector, getting some of the sector's data bits, but not all data bits?

Sample SPINRITE LEVEL 5 NOREWRITE LOG:

  |--------------------------------------------------------------------------|
  | Sector 3,031,506,936 (77.5911%) Trouble was encountered writing          |
  | inverted data block.                                                     |
  |--------------------------------------------------------------------------|
  | Sector 3,031,506,936 (77.5911%) Retrying write to restore original       |
  | data.                                                                    |
  |--------------------------------------------------------------------------|
  | Sector 3,031,506,936 (77.5911%) Rewriting original sector                |
  | sector-by-sector.                                                        |
  |--------------------------------------------------------------------------|
  | Sector 3,031,506,936 (77.5911%) A media defect was located within this   |
  | sector.                                                                  |
  |--------------------------------------------------------------------------|

Resolved by not using NOREWRITE.

- - - - -

Regarding 'bit rot', nobody owns the term, and for me, it implies that data storage bits, as read back, do not match the data storage bits that were originally recorded.

https://www.datacore.com/glossary/bit-rot/​

https://www.techtarget.com/searchstorage/definition/bit-rot​

https://en.wikipedia.org/wiki/Data_degradation +23 external original references​

. . . +79,790 additional web search results.​

These definitions of 'bit rot' are NOT my experience of the one sector that could not be read.

It's not that the sector was read successfully but had inaccurate content.

It's that the sector could not be read at all.

But the sector could be rewritten.

And rewriting the sector then made the drive able to write and read that LBA logical block address from then on.

I have no way of knowing if the successful write, and subsequent re-read, were to a reallocated location, not at the original 'sticky' sector.

Perhaps it was a bad map, a corruption where it was looking for sector 235827ýý64, or some other misstep in the go-between, for a random example.

The original data may have been sitting somewhere, but was unfindable in sector 2358277264, but there was no map to the original data anymore.

So the drive responded to SpinRite by saying "no can do".

I'm speculating.

So maybe there was 'bit rot', but in the map chip, not in the data storage chip.

And a forced write request from SpinRite was then responded to by the drive itself re-writing the map chip first.

How would anyone know?

Chip maker Crucial says publicly at https://www.crucial.com/support/articles-faq-ssd/why-does-ssd-seem-to-be-wearing-prematurely

. . . SSD [ Solid State Drive ] wear and performance are both dependent on the nature of the workload presented as IO [ input output ] activity from the host computer, on the amount of “static” data that’s stored on the computer (or the amount of free space), and on how long data has been stored. As these variables change, performance will change, and the pace of wear will change.​

​

There are physical reasons for this. NAND [ "NOT AND", a type of flash memory, non-volatile, it can store data even when there is no power ] flash storage is organized in what SSD engineers call pages and blocks. A block of NAND flash can contain hundreds of pages, and a page contains 16kB of data, in most configurations. When a NAND block contains data, new data cannot simply be written over the present data. The block must first go through an erase step before it’s ready to receive new data. However, while NAND flash can be written a page at a time, it can only be erased a block at a time. All these complications mean that the SSD firmware is constantly managing the physical locations of stored data and rearranging data for the most efficient use of pages and blocks. This additional movement of stored data means that the amount of data physically written to the NAND flash is some multiple of the amount of data passed to the SSD from the host computer . . . [ that is, write requests always cause more write than requested ]​

More from commercial SSD maker Samsung:

https://semiconductor.samsung.com/u...ductors-101-part-3-inner-workings-of-the-ssd/​

https://semiconductor.samsung.com/u...ors-101-part-2-different-types-of-nand-flash/​

​

Flash Memory Summit 2014 Steven Hetzler, IBM:

http://smorgastor.drhetzler.com/wp-content/uploads/2014/08/SSD-Reliability.pdf​

. . . and so on.

- - - - -

Tools.

Who has tools that can look at a specific SSD in hand and report what's going on inside that SSD, chip-wise, size of data transfer, cache, blocks, pages, internal self-maintenance routines, what can be interrogated, what can get toggled on and off, what can be cleared, refreshed, whatever?

https://www.minitool.com/partition-disk/ssd-health-check.html​

http://aezay.dk/aezay/ssdz/​

https://semiengineering.com/four-steps-to-verifying-an-ssd-controller-with-emulation/​

. . . +699,000 additional web search results.​

​

Watch others dig into the chips:

https://www.pentestpartners.com/security-blog/walkthrough-investigating-an-ssd/​

. . . +2,370,000 additional web search results.​

Eavesdrop or participate in other threads:

https://www.quora.com/How-do-I-chec...-SLC-MLC-TLC-QLC-apart-from-reading-its-specs​

https://www.google.com/search?q=extensive+technical+discussion+of+ssd+testing+and+reliability​

. . . +10,300,000 additional web search results.​

- - - - -

If I have any goals here, I'd like:

to learn what is going on in my own systems, in the systems that I test, to measure, control, purchase, implement, and support accurately and appropriately, to eliminate surprises - I test ALL media with SpinRite Level 5 immediately upon purchase,​

​

to inform SpinRite 7+ development, in support of the goals above, ​

( and to know where and why SpinRite 6.1 and prior SpinRite versions are appropriate, or not, for SSDs ).​

On the one hand, folks just want to get to their data as reliably and quickly as possible.

On the other hand, end users are curious about what's going on under the hood, so to speak, and the inner workings of future SpinRite versions may depend on enhanced internal awareness of chip architectures and behaviors.

So, here I am.

Thanks.

.

 

[peterblaise]

@peterblaise "... an SSD drive will ONLY reallocate on a WRITE request, not on a READ request ..."​

@Alice "... SSDs reallocate all .. the .. time .. Only prerequisite being they can read / recover the data. SSDs function by reallocation, it's a core feature ..."​

Good point, thank you for the review and redirect.

Make that: "... In response to external read and write requests, such as from SpinRite, an SSD will ONLY reallocate on a WRITE request, not on a READ request, but, an SSD may reallocate on its own whenever the original manufacturer's programmer's whimsical internal calls decide ..."

Is that more accurate and appropriate?

In other words, on an SSD, an external read request that may experience internal errors does not in and of itself trigger a drive to perform internal LBA reallocation.

Or, on an SSD, does an external read request possibly initiate a reallocation if the drive itself 'thinks' that reallocation is appropriate in response to the read attempt?

Thanks.

.