SATA, SAS, HBAs, Backplanes, BIOS, & >2TB HDDs

[pau13z]

Ambitious, I know... However, I thought I'd ask...

I've got access to servers that have RAID controllers in HBA mode that are connected to backplanes with up to 12 x 3.5" or 24 x 2.5" HDD's which could be SAS, SATA, or NVMe, and with capacities that can exceed 20TB. The server is configured to BIOS boot, and I've hacked up the instructions on the Run SpinRite on a UEFI-only machine (Part 5B of 5 - Create your own Virtual Machine) page to get SpinRite 6.1 on a local, internal storage device, from which the system can boot, and SpinRite will run after executing spinrite.exe.

This is my first attempt at booting directly from the internal storage device to see other internal storage devices to scan. Previous attempts were done through VirtualBox, and those worked. The reason for the switch-up is that when I popped these 20TB SATA drives into the system and configured them in VirtualBox, SpinRite reported that the level 5 scan would be done in approx. 5500 hours. I'm dedicated, but I have my limits. As I understand it, running directly from the hardware should be much faster, and I was hoping to capitalise on that. Unfortunately, SpinRite reported a capacity on these disks of 2.2TB.

As I understand it from some of the reading I've done, this could relate to the use of an HBA and a backplane between the drives and SpinRite, or it could be a limitation of the BIOS on the system, not being able to pass through the full capacity.

I believe I've seen references to people using drives at or above this capacity in this forum, but I've also seen references to connecting those drives directly to SATA ports on the system board, or via NVMe to SATA bridges.

1. Have I understood these potential limitations correctly?
2. Does anyone have any thoughts on what I could do to try to get around these limitations while using the hardware in-place?
3. How many of the above limitations might be resolved in SpinRite 7? (Don't hate me for asking... guestimate possible maybe kinda ETA?)
4. Not mentioned above but curious, I found a definition for level 5 that suggested that it could, possibly, recover bad blocks? Will level 5 correct previously "bad" blocks, and if so, is that reflected in the SMART data, or is this possibly referring to "pending" errors? "This level will also mark any regions of the drive that were once previously found to be defective, but may now test reliable, as being once again available for full use."

I'm happy to provide more info if needed, and if there's any testing that I can do, I'm happy to try to help out there, I have access to a large number of similar systems of varying hardware configuration (2.5", 3.5", SAS, SATA, NVMe, varying capacities, various RAID controllers, various HBAs) at my disposal.

 

[ColbyBouma]

SpinRite does not directly support RAID or HBAs. To get the best results you'll need to connect the drives to a SATA controller.

 

[DanR]

1. Have I understood these potential limitations correctly?

Nope. See Colby's post above.

2. Does anyone have any thoughts on what I could do to try to get around these limitations while using the hardware in-place?

There are none. See Colby's post above.

3. How many of the above limitations might be resolved in SpinRite 7?

A very good question. With no answer. It looks to be a long time before Steve will get around to thinking about how he will deal with SpinRite 7 (let alone actually working on it). My best WAG at this time is two years, minimum. Very possibly longer?

4. Not mentioned above but curious, I found a definition for level 5 that suggested that it could, possibly, recover bad blocks?

So will levels 2, 3, and 4, while putting much less stress on the drive than Level 5.

Will level 5 correct previously "bad" blocks, and if so, is that reflected in the SMART data

Not likely. SMART data is a mess. There is NO standard. Drive manufacturers thus define and implement SMART in their own (sometimes proprietary) ways.

Last but not least: The 5000 hr run estimate and 2.2 TB capacity limit make it clear that SpinRite in this case was seeing that drive as a BIOS drive and accessing it via the BIOS at the very S-L-O-W BIOS I/O speed, hence the 5000 hr estimate, and the 2.2 TB capacity limit.

As Colby noted, your best bet would be to connect the drives to a SATA controller in a machine with legacy booting. This would allow SpinRite 6.1 to use its native drivers to access the drives at the max speed a controller-drive combo is capable of with no capacity limitations. That said, it would still take some time to scan 22 TB of drive space!

There are unfortunately NO shortcuts possible with SpinRite 6.1 and RAID.

 

[pau13z]

Cheers for the replies Colby & Dan

I did mention in my post that the issue could relate to the HBA/backplane, between the drives and SpinRite, wouldn't that be understanding that limitation of SpinRite not directly supporting RAID/HBA? Or are we talking about different things here?

I was thinking that the only way to work the issue with the hardware I have or a slight modification would be if I could connect the backplanes to the system board, I believe some of my systems have that ability, so I could look into that.

I don't know what else Steve has going on, but I understand life is life and it could easily be that long or longer. My only concern would be something that Steve mentioned in another post I saw on here... forgetting some of this stuff before he can finish, which of course we all hope doesn't happen. I looked through some of the technical documents Steve's published RE SpinRite and how drives work and the knowledge he has blew my mind. Steve, I tip my hat to you sir.

Yeah I see that SMART is a mess, it drives me nuts. That's why I want to run level 5 on these drives. I can get replacing a drive if it's faulty... but when the software that marks the drive as faulty or pred fail does so because it can't interpret the state of the drive or its' SMART data... that's when we've got a problem.

Seagate drives seem to be the worst from what I can see. I have several tools that will mark a Seagate drive as pred fail, including SeaTools Legacy... but you hook up the current version of SeaTools... Nah, she'll be apples mate. Which do I believe? Which do I trust? Neither... I'll trust the results from SpinRite.

 

[pau13z]

Oh something I was thinking as well, I forgot to include it in my above ramblings, and could motivate Steve (Not that he's not, I know that Steve's perfection never ends), and not saying it will happen because I have no access... but imagine if SpinRite was the product that could properly and reliably detect and potentially correct these issues and determine if a drive is actually defective or not, and resellers and vendors and OEM's around the world could save insane amounts of money by not replacing drives because they could build SpinRite into their diagnostic tools. Bulk licensing agreements

 

[peterblaise]

Any SATA3 system - old, used, cheap - will whip through a larger
drive much faster than whatever you have seen and shared as
estimates - unless the long estimates are because the drive has crap
at the front.

Nothing wrong with running a Level 2 first just to confirm, even
turning off DynaStat and ReWrite so there would be a minimum
hesitation over unrepairable discoveries, getting from end to end,
and offering a log and a map:

SPINRITE NORAMTEST SKIPVERIFY LEVEL 2 DYNASTAT 0 NOREWRITE

That said, I run a full Level 5 on all my drives, including 20+ TB, and
I'm grateful that SpinRite 6.1 found errors that triggered my return and
the vendor swapping, no questions asked.

PS - SpinRite takes no actions on S.M.A.R.T. ( read: stupid ) "errors".

- - - - -

As far as I know, @Steve Gibson never sold or licensed Flicker Free
to anyone - an example of not selling for resale, I guess.

Instead, the video adapter industry took a scolding, and then wrote
their video bioses to be much smarter from then on - nVidia is
testimony to their unrelenting competitive improvements ever since.

Some SSD on-board controllers have SpinRite-like self-repair.

There are flexible alternatives while we wait for SpinRite 7.

None are the equal of SpinRite 6.1, but all offer something in the
meantime.

Let us know what you do.

 

[pau13z]

So at least one of the systems (I presume more) can have the backplane connected directly to the system board, and the on-board controller can be set to AHCI, so unless I come across something else, I'm hoping to get hold of a few parts to change the config of this system over to be direct-connect rather than through the controller/HBA. Not sure how long it'll take to get the parts though.

In this case, I actually want to run level 5, I want to wreck these disks with I/O and block checks to verify if they're faulty and need to be replaced. I'm hoping to get something more reliable than looking at SMART tables and not being able to trust what I'm seeing.

Something else I don't think I mentioned above, it'd be great to be able to run SpinRite on multiple disks at a time, game changer in the server space.

 

[peterblaise]

S.M.A.R.T. is stupid.

I have drives that scream S.M.A.R.T. failure at boot, and then pass
SpinRite 6.1 Level 5 with no read-write data test errors.

BackBlaze brags that they remove drives on S.M.A.R.T. errors, then
they say that drives also fail without S.M.A.R.T. errors, and then they
do not report if they test those drive's that they removed on SMART
errors just to check and learn anything.

Drives are cheap, when you can get them.

Time is expensive.

So billable shops swap parts regardless, and move on.

SpinRite is for those of us of who's time is somewhere in between

The beauty of a mule cheap used SATA 3 system is that it can be a
drive test station while the main systems just keeps on a running.

I've left drives testing for months, while I do other stuff.

Multiple mules = test multiple drives at the same time.

I've had 20 SpinRite tests going at once, and I have pictures!

OK, that's one room, there are 2 more computers out of sight in that
room, and more computers in other rooms:

.

More in every room, in every walk-in closet, some with KVM keyboard
video mouse switches and displays that switch between computers,
so 1 screen can show me what's happening on 3 or more computers
running SpinRite.

I'll find a picture ot 7 laptops running SpinRite that I took on vacation -
a great opportunity for me to sit back and relax.

;-)

My 2¢.

Keep us informed on what you do - thanks.

PS - "pred fail" = "A "Pred Fail" Predictive Failure message ​

indicates that your hard drive's S.M.A.R.T. system has detected ​

imminent hardware failure. Back up your data immediately to an ​

external drive or cloud storage. The drive is likely failing due to ​

too many bad sectors and will soon be unusable, requiring ​

replacement. ​

 

[pau13z]

The drives are Seagate Exos X20's, GSmartControl (Which uses SmartMonTools/smartctl as a back-end) reports multiple flags on the drives for pred fail, so does SeaTools Legacy Edition, and HDSentinel. The current revision of SeaTools though reports no issue. So if there are OS's out there, controllers, SDS systems that in any way utilise any of those tools, it's possible that drives could be requested for replacement when there's nothing wrong with them. But before we can push back, we need to be able to show that the drive is functioning normally. Maybe I should just tell everyone to run SeaTools

 

[pau13z]

OK so lots of variables changed unfortunately, but, I got it working.

I moved to an older server with a controller that is purely an HBA, as opposed to the newer server that was running a RAID controller kind of like an HBA. It was It was configured to treat all disks as non-RAID disks. This might be "eHBA" mode? I'll dig further into it later.

Not everything is 100%, the 4TB disks are actually SAS and the 20TB disks are actually SATA, but SpinRite picks them all up as SAS, and of type "BIOS":

Checking the speed of a 20TB disk, I got around 50MB/s, if anyone can comment as to your experience RE speeds of drives of similar capacity, that would be appreciated.

Kicking off a level 5, I got an ETA of 95 days...

So I took your advice @peterblaise and dropped to a level 2, with a much more workable ETA of 5.5 days...

At some point I'd still likely need to run level 5 on these, depends on what the engineers might request, and it'll still need to run on one drive at a time, but this is progress

 

[peterblaise]

You're running through the BIOS, not SATA, so 50 MB/s is expected.

Get a SATA3 mule PC, boot from USB, and see, well, here's one of
mine on SATA3 that went 282 MB/s, about 5.6 times faster than your
BIOS:

|==========================================================================|
| ST20000NM007D-3DJ103 |
|--------------------------------------------------------------------------|
| access mode: AHCI serial number: ZVT27MN7 |
| pci bus addr: 0:31:2 firmware rev: SN01 |
| adapter vendor: Intel Corporation rotation rate: 7200 |
| vendor-device: 8086-8C02 ata/atapi spec: ATA-8 |
| bios drive: 81h drv technology: generic |
| attached port: 1 lba in use: yes, 48-bit LBA |
| port speed: SATA 3 at 6.0Gb/s |
| max driv speed: SATA 3 at 6.0Gb/s multi-word dma: -/2 (unknown md) |
| ultradma modes: 5/6 (100.0 MB/s) |
| sector count: 39,063,650,304 available pio: 4/4 (16.67 MB/s) |
| byte count: 20,000,588,955,648 |
| 4Ksec features: SMART SECURE POWR |
| transfers: 32,768 sector long ops: no : WCACH LOOKAHD hpa |
| AHCI speed: 234,187,780 byte/s feature detail: 306B7561 3069B441 |
|==========================================================================|
| Drive's measured performance before running SpinRite |
|--------------------------------------------------------------------------|
| smart polling delay: 16.659 msec |
| random sectors time: 11.834 msec |
| front of drive rate: 282.788 MB/s |
| midpoint drive rate: 239.135 MB/s |
| end of drive rate: 129.501 MB/s |
|==========================================================================|

Old PCs are quite valuable!

They save us much time.

We can only SpinRite-test one drive at a time per PC at the moment,
so if we want to test more than one drive, then we need to get more
than one SATA3 PC.

I sense a rack and KVM in the near future!

;-)

 

[AlanD]

OK so lots of variables changed unfortunately, but, I got it working.

I moved to an older server with a controller that is purely an HBA, as opposed to the newer server that was running a RAID controller kind of like an HBA. It was It was configured to treat all disks as non-RAID disks. This might be "eHBA" mode? I'll dig further into it later.

Not everything is 100%, the 4TB disks are actually SAS and the 20TB disks are actually SATA, but SpinRite picks them all up as SAS, and of type "BIOS":

View attachment 1998

Checking the speed of a 20TB disk, I got around 50MB/s, if anyone can comment as to your experience RE speeds of drives of similar capacity, that would be appreciated.

The lowish speed ( 50MB/s) is because these drives are being seen as BIOS devices, probably due to the limitation/implementation of the HBA. If you can attach them, either singly, or as multiples, directly to a SATA port on the motherboard, you should see speeds up to 500MB/s.

 

[DanR]

I moved to an older server with a controller that is purely an HBA,

You do NOT seem to be getting it! SpinRite 6.1 DOES NOT do RAID or HBA!!!

LOOK at your first two screen shots. Everything is BIOS. SpinRite is ONLY able to see/access these drives thru/via the BIOS.

Hence the very S-L-O-W speed and very L-O-N-G run times.

Your ONLY option: Move these drives individually to a system with a SATA controller.
Connect them to the SATA controller
Legacy boot the system.
This will allow SpinRite to access the drives directly using its native AHCI drivers at best possible speed.
Days of run time will become hours of run time.

You CANNOT do what you are trying to do with any RAID or HBA configuration. SpinRite does not / can not support that.

 

[pau13z]

@DanR yeah I understand it's a limitation, I believe I asked to confirm that in my original post.

I've simply been testing to see what can be achieved with little to no hardware configuration modifications.

I figured that these speeds were still horrible, hence asking for some comparative speeds (Cheers @peterblaise & @AlanD ). Now that I have those and can see the significant difference, I'll try another hardware configuration, but I'll likely need to order some parts.

Some of our systems support connecting a backplane that supports 8 HDDs directly to the system board. At that point the chipset can be configured as a RAID controller or an AHCI controller. So my next step is to get that cable, connect it to the system board, set AHCI mode, and test. But it could take a while to get that cable.

 

[pau13z]

I sense a rack and KVM in the near future!

Already got several of each! Most of the time for enterprise though we'll use remote management facilities built into the systems to connect, like the iDRAC.

So if I can get one system configured to run up to 8 drives say over a weekend, that'd be great, set and forget, but if I needed to do one individually, I'd need to see if I can reconfigure a 2nd system so I don't need to cancel any existing tests.

 

[peterblaise]

SpinRite can do each attached drive in series.

"Canceled" tests can be restarted from where they left off, see
@ColbyBouma's excellent
https://gitlab.com/GRC-Community/spinrite-6.1-wiki/-/wikis/Command-Line
Play a bit to see if you prefer restarting at a % or a particular sector.

I use a KVM or a monitor's own switch to cram as many computers
as possible into one space.

8 20+ TB drives at Level 5 at 5 days each in one computer = 40 days.

Hence, the call for additional computers.

8 20+ TB drives at Level 5 at 5 days each in 5 computers = 5 days.

;-)

 

[pau13z]

100% agree, but I'm doing this on work equipment, and while I've got plenty of these spare systems to play with, I'd need to reconfigure too many hardware wise, when getting a cable can sometimes be a hurdle haha.

But what I will do is test on my personal system more, which is why I got a copy in the first place. I got it to run previously on my personal system, I can't even remember how now, either VirtualBox or Hyper-V, but I don't remember their speeds. But I also want to keep that and this separate so I'll focus on the server and connectivity thing in this post mostly

Focusing more on that server side more though, I can say, I think with confidence, that if companies had a methodology to quickly test multiple drives using SpinRite in their servers, it could see larger adoption that way, which is why I was trying to see what could be done to get it to work under various hardware configurations, and why I threw those suggestions out there for the next release

 

[peterblaise]

@Steve Gibson hinted that Spinrite 7 ought to run in the background
as a service, find, and remedy failures, and notify the user.

Got a wish list?

 

[pau13z]

Anything above that I've tried that hasn't worked in 6.1? Haha. I know that won't happen, but working in the server space, I can see potential applications for it. Servers can utilise:

- SAS, SATA, NVMe, PCIe NVMe
- AHCI, HBA, HW RAID, SW RAID (Various OS's)
- Up to 100 (From memory, I can't recall one that can hold more but in sure some can) internal physical drives

Now imagine you're a sys admin. You order 20 new servers, each with 24 x 30 TB HDDs, and before putting them into production, you want to test that the disks are ok. How awesome would it be to be able to quickly and easily test all of those disks completely in less than a week?

Then imagine you needed to decommission those drives at some point in the future, and to recoup some cost you sell the drives. Wouldn't it be great to be able to provide a SpinRite report of how those drives tested when you finished wiping them?

And how amazing would it be as a customer to see that and gain confidence, but also ask... What's SpinRite? Then maybe get a copy for a few bucks to test other drives they have...

I know a lot of it would never happen due to how the software is designed and how long it would take to potentially even try to redesign, but if you're asking for pipe dreams, everyone would come out a winner

 

[pau13z]

Genuinely not wanting to rewrite the app and sincerely appreciating the work that's gone into SpinRite, I think that potentially the best way to go to try to achieve something like the above would be to have servers configured with flexible default configurations that can potentially cater for either configuration. Just like you can change the BIOS controller between RAID and AHCI, if a system comes with an HBA / RAID controller, then an extra system board connector and cable, along with an option to toggle between the two is available, then that would solve that issue. But even if the configuration change is possible and accepted, it might take years to implement and adopt. But still, all avenues should be investigated. I'll make enquiries.