Export thread

Everyone...

There is so much anticipation of SpinRite's and then Beyond Recall's availability that I've decided to create a work tracker for those who are interested in knowing what's going on from day to day. Although I'm unable to predict or guess when anything will happen, at least I can share where I am along the way. I'll use my main Blog for the announcement of significant milestones, and this work and progress tracking thread for more granular status updates.

I've decided not to make this tracking thread interactive because I'm afraid that it would be a huge distraction for me (I would feel an obligation to reply to posts here) that would retard my progress. So I'm going to keep this thread closed to general posting. This also means that those who wish to receive updates on my progress may use the forum's “Watch” feature for this thread without getting “spammed” by other reactions here.

While initially working on SpinRite v6.1 back in 2013, I developed the low-level hardware drivers for non-AHCI-mode IDE/ATA drives. Then, after SQRL was solidly completed, I returned to work on SpinRite and wrote drivers for AHCI-mode SATA drives. Working with the gang of terrific testers and troubleshooters over in GRC's NNTP newsgroup forums, we first got the new AHCI drivers solidly working on every piece of hardware that anyone had. Then I added read benchmarking as a means to make testing this stuff more interesting. Once that was finished, I returned to the 7-year old code from 2013, updated and verified it and then added the same sort of benchmarking to it. Also along the way, somewhere this year, I developed our own USB formatting capability since all of the next-generation products would be needing a robust ability to prepare a bootable USB device.

The two modes of hardware controller operation (non-AHCI and AHCI) are very different from one another, so I'm currently working to merge the two driver types into a single solution, while also creating a solid foundation that SpinRite and Beyond Recall will need. I suspect that I'm probably a few days away from having a fully amalgamated candidate “ReadSpeed” benchmark ready for its final round of testing, which we'll again be doing over in GRC's NNTP grc.spinrite.dev newsgroup.

Once that's finished, we'll have the “ReadSpeed” (RS.EXE) DOS program that needs to be run to perform benchmarks on everyone's machines. So then a custom version of the “InitDisk” utility will be created to automatically prepare any USB thumb drive for ReadSpeed. It will format it, install FreeDOS, and preload the various DOS programs — including RS.EXE.

Then we'll have a Windows utility that will prepare any USB drive. When that USB drive is booted, its users will be able to run ReadSpeed and a handful of other interesting system level utilities.

Onward!

It's Christmas Eve, and I have just posted the first release of ReadSpeed to GRC's new ReadSpeed page here: https://www.grc.com/readspeed.htm. The only thing left for me to work on for that page is the benchmark demo video. The one that's currently showing is not the one whose results are shown above it, and I'll be adding a voice-over in the next day or two to describe and point out items of interest as the benchmark proceeds.

But... Thanks to a great deal of work and help from the many people who hang out over in GRC's NNTP text-only newsgroups, this code has been quite well tested and pounded on. So the next question to answer is... how well does it do for everyone else?

Does it work for you? Download the ReadSpeed for Windows prep tool. Give it a USB drive whose contents you don't need. It'll wipe it and install a bootable DOS and ReadSpeed for DOS. Then boot it and have fun!!

The ReadSpeed forums are now open here... So come on over and share you results!!

Merry Christmas!!

Yes, at long last it's finally true.

10 days after ReadSpeed's release, the shakedown test of the benchmark's new hardware drivers has been a tremendous success. We have identified only one problem with a system in Denmark using an old Asrock motherboard. I've located and purchased one of them in Germany. So when it arrives, I'll see what needs to be done to get the drivers running on it, too. But aside from that one problem, as far as we know, this new technology in ReadSpeed is ready for SpinRite's use.

So, today, Sunday January 3rd, 2021, I successfully assembled SpinRite's 31 assembly language source code files and created a working DOS executable. Now I begin the work of incorporating the new drivers into SpinRite.

That work has begun.

MUCH has transpired since my previous, January 3rd, announcement that work on SpinRite v6.1 had commenced.

To plan for what SpinRite v6.1 would, and would not, be, I needed to know how I was going to solve SpinRite's imperative need to boot on UEFI systems that no longer offer a traditional BIOS fall-back. So, while beginning to work on SpinRite, I also searched for any means to add BIOS support to a system that doesn't currently have any. The firmware which boots a system is intimately tied to its hardware. That's why systems need firmware in the first place. This means that it's not possible to simply run any BIOS on whatever hardware happens to be present. So I was hoping that someone somewhere might have created a UEFI-to-BIOS translation layer which would allow for BIOS calls to be translated into their UEFI equivalent. That would allow SpinRite to load such a translation layer, boot BIOS-dependent DOS, and then reuse everything that SpinRite already is on any UEFI system. But, unfortunately, nothing like that exists and no one has ever created such a thing because the need for something like that only exists due to SpinRite's unique and continuing dependence upon the legacy of 16-bit DOS.

This provided some needed clarity. The only way to move SpinRite forward would be to finally, after 35 years, end its dependence upon the BIOS and DOS. So, I began looking around for the optimal environment to host SpinRite's future.

I'll spare you the blow-by-blow. But, for what it's worth, everything was looked at. In the end I have settled upon an extremely lightweight real time operating system for embedded applications called “On Time RTOS32.” It is a minimal OS that allows executable code to be written, tested and debugged under Windows (my preferred development environment) then packaged for operation under its own loader and environment. And, of course, it boots from either a BIOS or UEFI firmware.

Among other things, it means that SpinRite will finally need to be ported from DOS's 16-bit “segmented Real Mode” memory model environment to a 32-bit flat memory model. But this is the only way for SpinRite to grow. As a result of this research and planning, SpinRite v6.1 will be the last 16-bit BIOS/DOS-only SpinRite.

Once v6.1 is launched, I will immediately begin the process of converting SpinRite from its 16-bit segmented-memory real mode form into a flat 32-bit memory model. This will create the first 32/64-bit SpinRite v7.0 capable of booting under either UEFI or BIOS firmware. And for the sake of speed, I plan to initially leave SpinRite's old and creaky user interface as is. v7.0 will offer the same advanced support as v6.1 but with UEFI. So it will run ATA/IDE and AHCI controllers at their maximum possible speed. But since there's no BIOS to fall back on, v7.0 will not initially be able to operate upon USB or NVMe devices at all. So v6.1 will remain the only solution for those until v7.1 and v7.2. After v7.0 is released, my plan would be to immediately add support for USB, creating v7.1 (and, of course, no charge for those who have upgraded to v7.0). And once v7.1 is released I would then immediately work to add NVMe support, creating v7.2.

At this point we'll have text-mode old-looking single-tasking SpinRite v7.2 running ALL current mass storage controller interfaces at their maximum possible performance. And all of that code will be reusable by SpinRite v8.x.

The plan for v8 is just a rough sketch at this point. But it will be a move to a mouse and GUI interface, the ability to simultaneously run on any or all of a system's drives, eventual file system awareness, structure-driven data recovery, and file-based recovery and who knows what else.

As always, this is all subject to revision and refinement. But it's where we are as we start into February and it feels very right to me. It places the most functionality into SpinRite user's hands as quickly as possible, supports SpinRite's quickest migration from BIOS/DOS/16-bits -to- UEFI/BIOS 32/64 bits, while providing a clear and seamless path forward into SpinRite's future.

Work on the next SpinRite continues proceeding very nicely. The new "Flat Real Mode memory manager has been moved from ReadSpeed into SpinRite. This gives SpinRite access to 64 megabytes of RAM from within DOS real mode. The new IDE/ATA and AHCI driver code has also been moved into SpinRite and is working.

The controller and drive discovery and enumeration phase was mostly borrowed from ReadSpeed. But, ReadSpeed simply ignores any drives not connected to IDE and AHCI controllers, since that's what it was designed to test. SpinRite v6,1 needs to fall back to other access methods so that it can still run on any drives. So SpinRite needed a brand new drive & controller discovery system.

Traditionally, this happened behind the scenes while a static screen was being displayed. I decided that I wanted to surface that process to show the user what SpinRite was doing. Here's a short 25-second video showing SpinRite v6.1's process of scanning the system for controllers and drives then identifying them:


Here's another sample of the final screen. This is from my development VirtualBox system. It shows a variety of drive sizes, the presence of non-BIOS drives, how SpinRite handles NVMe and SCSI controllers, and pure BIOS drives:


In other news...

Today I received the first component of the On Time RTOS-32 system, which I wrote about above on February 1st. I will not allow myself to be distracted by it. SpinRite v6.1 comes first. But I want to make a bit of time to quickly verify that this solution will work as the platform for SpinRite v7.0. So I plan to create a quick “Hello World!” proof-of-concept app that can boot on either BIOS or UEFI. That will demonstrate that such an app can then be turned into SpinRite v7.0.

A milestone and a bullet-proof solution. It has always been the case that there's far more going on behind the scenes, unseen, with SpinRite than is apparent. That's as true for the work I've just completed as anywhere. A month ago I had a solution that would work for most users in most situations most of the time. But I encountered some users for whom it did not work. So the past month was invested in engineering a solution that would be absolutely guaranteed to work for everyone every time. SpinRite has that, now.

I also wasn't completely satisfied with the design of the first “Discovering System's Mass Storage Devices” screen shown below. The user doesn't care about PCI bus location, Vendor and Device. I care about that. They shouldn't have to. It took a few iterations for me to understand what the screen should show. Now it does:


The screen that follows this one, after the user presses the “Any” key, allows SpinRite's user to browse through these drives and examine all of their various characteristics. I'm now at work updating that series of screens to show all of the newly available data from the drives SpinRite has direct access to.

I was originally worried that users of the new SpinRite v6.1 might not notice any difference between it and v6.0. I suppose I should have known that I would not be able to keep myself from making it as good as I possibly could, and that the drive to do that would result in a clearly updated and upgraded product.

Finalizing SpinRite's operational foundation.
The 4th pre-release of SpinRite is now undergoing group testing. A readily reproducible problem has been found with Intel ATA/AHCI chipsets when operating in ATA mode (but not in AHCI mode). Since this is a new problem that previous releases never had, a solution should be readily available (I broke something that I simply need to fix.)

This release marks an important milestone for SpinRite 6.1 because its controller and drive discovery and enumeration phase now performs a full read & write data transfer confidence test using SpinRite's new I/O abstraction system which separates SpinRite's operational logic from the details of individual drive operation and requirements. In other words, once this is all working correctly — and we're close to that — the new foundation upon which SpinRite 6.1 is being rebuilt will be finished and finalized and the rest of the work is just a matter of using the existing functions that will have been proven to work solidly and reliably.

VERY encouraging performance numbers.
The 5th pre-release of SpinRite was published on October 31st. It features a largely rewritten benchmarking facility which now estimates the time SpinRite will require to perform a standard “Level 2” surface scan and analysis. And after 18 hours of quick but broad testing in GRC's newsgroup, we have learned that the main problem with Intel chipsets is fully resolved, that it appears to be much more solid than any previous release, but that its handling of BIOS-interfaced drives still needs work. (This was expected and noted in its release notes.)

In my pre-pre-release testing, the benchmark estimated that SpinRite v6.1 will perform a surface scan of...

• A 256 GB SSD AHCI-attached drive in 7.3 minutes.
• A Kingston 129 GB SSD attached to AHCI in 3.7 minutes.
• A 64 GB USB-attached thumb drive in 59.3 minutes.
• A 1 TB USB-attached solid state drive in 14.46 hours.
• An old (spinning) 40 GB Maxtor IDE (parallel cable) in 31.1 minutes.
• An old (spinning) 160 GB Seagate IDE (parallel cable) in 48.1 minutes.
• A Samsung 860 EVO SSD IDE (parallel cable) in 29.0 minutes.

Here are two screenshots posted by one of the 5th pre-release testers last night:


This person's 512GB Crucial SSD can be scanned by SRv6.1 in 15.3 minutes, and his entire massive 10TB Seagate “spinner” drive can be scanned in 15.11 hours.

So... things are looking good.

The Evolution of SpinRite's Benchmarking.
SpinRite's forthcoming 6th development pre-release has evolved further and now shows the estimated time to scan any drive upon which the benchmark has been run in a new “ScanTime” column which is visible throughout the UI once drive enumeration is completed:


I've created a 4-page PDF which walks its reader through a number of interesting aspects of what we're finding: https://www.grc.com/dev/spinrite/SpinRite-Benchmarks.pdf