It has been 4 years now, this month Steve has been working on SR 6.1! Wow time flies. How long will 7 take to develop?
4 years!
- Thread starter Mervyn Haynes
- Start date
- Watchers 6
-
Be sure to checkout “Tips & Tricks” Dear Guest Visitor → Once you register and log-in please checkout the “Tips & Tricks” page for some very handy tips!
/Steve. -
BootAble – FreeDOS boot testing freeware
To obtain direct, low-level access to a system's mass storage drives, SpinRite runs under a GRC-customized version of FreeDOS which has been modified to add compatibility with all file systems. In order to run SpinRite it must first be possible to boot FreeDOS.
GRC's “BootAble” freeware allows anyone to easily create BIOS-bootable media in order to workout and confirm the details of getting a machine to boot FreeDOS through a BIOS. Once the means of doing that has been determined, the media created by SpinRite can be booted and run in the same way.
The participants here, who have taken the time to share their knowledge and experience, their successes and some frustrations with booting their computers into FreeDOS, have created a valuable knowledgebase which will benefit everyone who follows.
You may click on the image to the right to obtain your own copy of BootAble. Then use the knowledge and experience documented here to boot your computer(s) into FreeDOS. And please do not hesitate to ask questions – nowhere else can better answers be found.
(You may permanently close this reminder with the 'X' in the upper right.)
One man, doing the work of a team of programmers, in assembly language, determined to get it right not just easy (doesn't want it erasing our data), part time, along with other projects, and a podcast. Time tables are hard to predict under those circumstances. Also, existing owners get it for free so he won't make any money on it from them. I'm a fan and user of 6.0 and looking forward to the next one, but I DEFINITELY don't want it erasing my data. I want it SOLID. Hopefully he'll be able to complete it and have it fully functioning and tested and safe. I'm also a big fan of the podcast. Much of my computer security knowledge has come from @Steve and @Leo.
May your bits be stable and your interfaces be fast.
Ron
May your bits be stable and your interfaces be fast.
Ron
Except for the time he took off to work on SQRL and the time for the podcasts, @Steve's been full-time (or more) on SpinRite. The other utilities he's developed while working on SR have been an integral part of the project.
My guess? 1 year. Minimum. Most likely longer.
The "long pole in the tent" would appear to be re-coding the 16 bit SR 6.1 code, with all of its DOS/BIOS dependencies/limitations, into pure clean 32 bit code for SR 7. I have no idea how long that might take.
I think Steve will fairly quickly have a version of the current SR 6.1 code ported into the environment of 7. (So running on the new OS and booting via UEFI too.) But I don't think 7.0 will be something someone would pay an upgrade price for if that was all it did, and so he has plans to add USB and NVMe and other options, which will make it much more valuable to users. He also has plans for new features like parallel drive operation, etc, which will also be a valuable option for some. I would expect him to release a development snapshot before the end of 2024, maybe even as soon as the middle of 2024, but it will definitely take a while to get a release of sufficient quality for his reputation.
1 year for 7! I cant see that. If 6.1 took over 4 years, I would expect 7 to take just as long. We are moving out of the DOS era and into new territory.
No, for 6.1 he mostly stopped using the BIOS and wrote his own drivers for the hardware. Assuming the new real time OS doesn't get in the way, those drivers should still be fine. What he'll need to rework is timers (if he was using the BIOS for them) and the UI input/output (since the new OS will need a new UI driver.) I didn't say he'd have a fully finished version quickly, but I think he will quickly get something basic working based on the skeleton of what he already has. Of course, only time will tell.
I've written a few small programs, nowhere near as complex and sophisticated as SR and not in assembler. Even though they are for my personal use there is always something I can improve or do better. In the end I just say it's good enough and move on rather that spend endless hours tweaking the code. Nothing I write ever touches any of my existing data so I can do that. I value my data, and fortunately for everyone so does Steve I often recall what a coworker once told a customer when he was asked how long it would take. The reply "How long does it take to catch a fish?" was his answer. I am as anxious as everyone to get 6.1 in my hands, but I am more concerned that it works as intended and I can trust it.
Comparing SR 6.1 and SR 7 development times is comparing apples to oranges. They are two quite different scenarios:
- SR 6.1 is in 16 bit DOS code with many restrictions, dependencies, problems, etc. It was built from scratch.
- SR 7 will be clean 32 bit code with NO DOS / BIOS mucking things up. It will inherit all the development work that went into SR 6.1.
Consider:
In the summer of 2020 a few months were spent developing AHCI drivers (to complement the ATA/IDE drivers developed in 2013). Along the way there was a 1 month hiatus to develop and subsequently release InitDisk v1 on GRC.
In the fall of 2020 a few months were spent developing ReadSpeed v1 and subsequently releasing it on GRC, with 4 improved versions of InitDisk (v2, v3, v4, and v5) released on GRC along the way.
At the beginning of 2021 Steve initially coded SR 6.1 from scratch. Later in 2021 he spent about 6 months re-coding SR 6.1 to "expedite eventually porting" (Steve’s words) its 16 bit DOS code to a 32 bit environment. (Note: Back then SR 7 was anticipated to be just a dual UEFI-DOS boot version of SR 6.1)
During SR 6.1 development testing, both Pre-Alpha and Alpha, MANY improvements to the aforementioned drivers were developed, as were a great many other code improvements.
NONE of this development effort needs to be duplicated for SR 7.
It is all there in SR 6.1, ready to be ported to SR 7.
Note: I do not know what is involved in porting this 16 bit DOS code into clean 32 bit code. I leave that to Steve.
I would think, however, that porting the 16 bit code should only take months as opposed to the years that went into the creation and development of the 16 bit code.
SpinRite 7.0 will need native drivers for USB and NVMe drives. However, the RTOS-32 package Steve purchased last fall (2022) contains functional USB and NVMe drivers. Steve will of course need to customize them for SpinRite’s use, but hopefully without requiring lengthy development time.
So, NO, SpinRite 7.0 should NOT take 4 years.
Rather, less. Considerably less. How much less is of course uncertain. There are always unforeseeable unknowns.
Given all of the above, I am guesstimating at least one but more likely two years. I hope I am a bit overly pessimistic. But . . .
If there are too many side projects or too many additional new features are considered for SR 7.0, then all bets are off. And it could take 4 years
!The 4 years included taking out side projects! We currently have a side project for 6.1. How many other "shiny things" will distract 7 from being developed? We also have to consider "feature creep", which will undoubtedly happen. Some time ago, I suggested having a bet on how long 6.1 would take, no one wanted to take that. So yeah, I think it will take time to get 7 out the door, I am not complaining, just making conversation.If there are too many side projects or too many additional new features are considered for SR 7.0, then all bets are off. And it could take 4 years
Right. With the move to SR 7x, all the restrictions of the 16 bit DOS-BIOS environment of SR 6.1 will be gone. The flood gates to "feature creep" will be WIDE open. Steve may succumb to "featuritis"!
Thing is a lot of the side projects, while coming out as separate things, are going to be folded into the 6.1 and 7 codebase in a good part, as sanity checks.
the easiest way to verify a device is not faking data, is to simply use a random data generator to get a block of random data, equal to the block size the device reports, less 64 bytes, and then hash the block, and get this 32 byte hash, append to the end, along with a simple list of the LBA it will be written to, and a CRC for the remaining 16 bytes of the block. then write to every LBA, a destructive write, so only going to be done after a good few "are you really sure you want to erase device xxx, description YYY". When the device returns it is full, you look to see if the number of successful LBA writes corresponds to the number of LBA blocks the device reported, and if yes, you read all the LBA blocks back, verify the CRC (lot faster than a hash, but can give false negatives for multiple errors), and then verify the LBA read corresponds with the LBA it was supposed to be written to. Errors you generate a table in memory with one byte per LBA, and start again, reading every block and comparing the hash, and set bits in the table. All zero no errors, set 1 bit for hash error, one bit for CRC error, and then when complete you make a GUI table, sized for showing all bytes of that table as a dot, and make all bytes with no error light green, low intensity, and any with errors bright red. the drives that fake will have the beginning, around 1024 LBA blocks, show as green, then a big block of red, and the last ones green again.
Destructive of data, but thorough.
the easiest way to verify a device is not faking data, is to simply use a random data generator to get a block of random data, equal to the block size the device reports, less 64 bytes, and then hash the block, and get this 32 byte hash, append to the end, along with a simple list of the LBA it will be written to, and a CRC for the remaining 16 bytes of the block. then write to every LBA, a destructive write, so only going to be done after a good few "are you really sure you want to erase device xxx, description YYY". When the device returns it is full, you look to see if the number of successful LBA writes corresponds to the number of LBA blocks the device reported, and if yes, you read all the LBA blocks back, verify the CRC (lot faster than a hash, but can give false negatives for multiple errors), and then verify the LBA read corresponds with the LBA it was supposed to be written to. Errors you generate a table in memory with one byte per LBA, and start again, reading every block and comparing the hash, and set bits in the table. All zero no errors, set 1 bit for hash error, one bit for CRC error, and then when complete you make a GUI table, sized for showing all bytes of that table as a dot, and make all bytes with no error light green, low intensity, and any with errors bright red. the drives that fake will have the beginning, around 1024 LBA blocks, show as green, then a big block of red, and the last ones green again.
Destructive of data, but thorough.
Would doing a full device encrypt and decrypt show the size difference? Possibly the decrypt wouldn't work, or would it simply encrypt the actual size and then decrypt the same? Would the software, say Veracrypt show the actual size rarher than the fake? I never had a 'fake' device to play with.
An interesting thread here, guys.
I'll offer a few points of clarification from my standpoint...
SpinRite 7 will be a full "start over from scratch" for SpinRite. This is mostly because SpinRite will be running on an entirely new windowed graphical operating system and it will have more of a feeling of being a mass storage workstation. As I currently envision it...
A (possibly long) list of the system's current drives with an overview of their characteristics will be running down the left side of the screen — like large vertical tabs on a web browser — and that list will dynamically grow and shrink as drives are added or removed. Those left-edge tabs will be large enough to allow monitoring of progress of any operations on that drive. So in addition to identifying the drive, what it's doing, if anything and a progress bar will always be visible. When a drive is selected, it obtains focus and the entire rest of the screen displays details about it with horizontal tabs running across the top allowing selection of various views of the drive's information, SMART, logs, maps, specs... everything.
I'm sure I'll lift much of the proven low-level math/time/display routines from v6.1 since they are already performing the 64-bit math that's required for 48-bit LBA drives. And, certainly, much of SpinRite's recently rewritten core will be moveable almost whole, since, as was observed above, it's all already flat-model 32-bit code. But mostly what will survive is in its entirety is everything I've just learned during the work on v6.1. This is one of the reasons do not plan to put SpinRite down and wait at all. My head is chock full of "SpinRite" right now. So I want to capture that in SpinRite's new code.
I've not allowed myself to "crack open the box" of RTOS-32, though I've been drooling to do so. So I do not yet know how much of RTOS-32's low-level driver code will survive. But I do NOT intend to rewrite anything that doesn't need it. It already has drivers for IDE, ATA, SATA, USB, and NVMe controllers. So one of our first orders of business will be to determine whether they're already able to handle everyone's systems as they are out of the box. If so, that means that I'll only need to add low-level hooks for SpinRite-specific operations. I already know that the drivers provide a very generic abstracted interface to their callers. (Read this, write that, did it work?) So I'll need to provide the means for SpinRite to obtain far more detail and talk to drives' "metadata." But SRv6.1 already does this so adding that won't be a mystery or a big deal. And I purchased the entire source code for everything.
So my plan for SR7 will be familiar to those who were with me from the start of SR6.1. I'm going to quickly bring up a simple command line interface with a scrolling text window and write a "drive enumerator" to discover every drive attached to the system. Then I'll add a benchmark to verify the readability of everything that anyone has. We'll quickly learn whether RTOS-32's existing drivers are able to handle everything that SpinRite 6.1's drives have learned to handle. And if not I'll dig into them to enhance them with what I've learned from v6.1.
At that point we'll have the basis for SR7 and I'll implement SpinRite 7's proposed Windowing system. One of the reasons I like the windowing architecture I described above is that it's completely modular and open-ended. The drive technologies that it can handle are transparent under the drive list on the left, and the functions it's able to handle per drive are also open-ended with tabs across the top. And... tabs can be added as we go along, adding additional features and capabilities to SpinRite. And, if a SpinRite 7 owner also owns Beyond Recall, the "Beyond Recall" tabs simply get added to the common SpinRite workstation.
And, finally, RTOS-32 also provides a large subset of the original Win32 API, which is the API to which I still code. The DNS Benchmark, InControl, ValiDrive, etc. are all written in the original Win32 API. And THAT means that all of the prototyping of SR7's new tabbed-windowed UI can be coded, implemented and tested by all of us under Windows without leaving Windows. This will make that aspect of the development much easier. And it also leaves open the possibility that some future SpinRite may also be dual-hosted and able to run entirely under either RTOS-32 or Windows.
So, that's my current plan. As for how long it will take?... I have no idea. But what I can promise is an interesting journey, an approach that will get something up, running and testable as soon as possible, and a result that's going to be worth working towards.
I'll offer a few points of clarification from my standpoint...
SpinRite 7 will be a full "start over from scratch" for SpinRite. This is mostly because SpinRite will be running on an entirely new windowed graphical operating system and it will have more of a feeling of being a mass storage workstation. As I currently envision it...
A (possibly long) list of the system's current drives with an overview of their characteristics will be running down the left side of the screen — like large vertical tabs on a web browser — and that list will dynamically grow and shrink as drives are added or removed. Those left-edge tabs will be large enough to allow monitoring of progress of any operations on that drive. So in addition to identifying the drive, what it's doing, if anything and a progress bar will always be visible. When a drive is selected, it obtains focus and the entire rest of the screen displays details about it with horizontal tabs running across the top allowing selection of various views of the drive's information, SMART, logs, maps, specs... everything.
I'm sure I'll lift much of the proven low-level math/time/display routines from v6.1 since they are already performing the 64-bit math that's required for 48-bit LBA drives. And, certainly, much of SpinRite's recently rewritten core will be moveable almost whole, since, as was observed above, it's all already flat-model 32-bit code. But mostly what will survive is in its entirety is everything I've just learned during the work on v6.1. This is one of the reasons do not plan to put SpinRite down and wait at all. My head is chock full of "SpinRite" right now. So I want to capture that in SpinRite's new code.
I've not allowed myself to "crack open the box" of RTOS-32, though I've been drooling to do so. So I do not yet know how much of RTOS-32's low-level driver code will survive. But I do NOT intend to rewrite anything that doesn't need it. It already has drivers for IDE, ATA, SATA, USB, and NVMe controllers. So one of our first orders of business will be to determine whether they're already able to handle everyone's systems as they are out of the box. If so, that means that I'll only need to add low-level hooks for SpinRite-specific operations. I already know that the drivers provide a very generic abstracted interface to their callers. (Read this, write that, did it work?) So I'll need to provide the means for SpinRite to obtain far more detail and talk to drives' "metadata." But SRv6.1 already does this so adding that won't be a mystery or a big deal. And I purchased the entire source code for everything.
So my plan for SR7 will be familiar to those who were with me from the start of SR6.1. I'm going to quickly bring up a simple command line interface with a scrolling text window and write a "drive enumerator" to discover every drive attached to the system. Then I'll add a benchmark to verify the readability of everything that anyone has. We'll quickly learn whether RTOS-32's existing drivers are able to handle everything that SpinRite 6.1's drives have learned to handle. And if not I'll dig into them to enhance them with what I've learned from v6.1.
At that point we'll have the basis for SR7 and I'll implement SpinRite 7's proposed Windowing system. One of the reasons I like the windowing architecture I described above is that it's completely modular and open-ended. The drive technologies that it can handle are transparent under the drive list on the left, and the functions it's able to handle per drive are also open-ended with tabs across the top. And... tabs can be added as we go along, adding additional features and capabilities to SpinRite. And, if a SpinRite 7 owner also owns Beyond Recall, the "Beyond Recall" tabs simply get added to the common SpinRite workstation.
And, finally, RTOS-32 also provides a large subset of the original Win32 API, which is the API to which I still code. The DNS Benchmark, InControl, ValiDrive, etc. are all written in the original Win32 API. And THAT means that all of the prototyping of SR7's new tabbed-windowed UI can be coded, implemented and tested by all of us under Windows without leaving Windows. This will make that aspect of the development much easier. And it also leaves open the possibility that some future SpinRite may also be dual-hosted and able to run entirely under either RTOS-32 or Windows.
So, that's my current plan. As for how long it will take?... I have no idea. But what I can promise is an interesting journey, an approach that will get something up, running and testable as soon as possible, and a result that's going to be worth working towards.
