Backing up/syncing files, how do they deal with hdd errors?

  • SpinRite v6.1 Release #3
    Guest:
    The 3rd release of SpinRite v6.1 is published and may be obtained by all SpinRite v6.0 owners at the SpinRite v6.1 Pre-Release page. (SpinRite will shortly be officially updated to v6.1 so this page will be renamed.) The primary new feature, and the reason for this release, was the discovery of memory problems in some systems that were affecting SpinRite's operation. So SpinRite now incorporates a built-in test of the system's memory. For the full story, please see this page in the "Pre-Release Announcements & Feedback" forum.
    /Steve.
  • Be sure to checkout “Tips & Tricks”
    Dear Guest Visitor → Once you register and log-in please checkout the “Tips & Tricks” page for some very handy tips!

    /Steve.
  • BootAble – FreeDOS boot testing freeware

    To obtain direct, low-level access to a system's mass storage drives, SpinRite runs under a GRC-customized version of FreeDOS which has been modified to add compatibility with all file systems. In order to run SpinRite it must first be possible to boot FreeDOS.

    GRC's “BootAble” freeware allows anyone to easily create BIOS-bootable media in order to workout and confirm the details of getting a machine to boot FreeDOS through a BIOS. Once the means of doing that has been determined, the media created by SpinRite can be booted and run in the same way.

    The participants here, who have taken the time to share their knowledge and experience, their successes and some frustrations with booting their computers into FreeDOS, have created a valuable knowledgebase which will benefit everyone who follows.

    You may click on the image to the right to obtain your own copy of BootAble. Then use the knowledge and experience documented here to boot your computer(s) into FreeDOS. And please do not hesitate to ask questions – nowhere else can better answers be found.

    (You may permanently close this reminder with the 'X' in the upper right.)

Larc

Member
May 31, 2022
5
0
The basic problem I have, is that I dont see any backup/sync software that does it in a safe way.

I've had issues where small parts of files become corrupted, which mess with portions of an image or video, but I dont know when they occured, as I dont often access the files.

Problem 1)
It seems that most programs would see the corruptions in the files and sync/backup the corruptions right?
This would also be an issue if ransomware modified your files.

saving file history can help with this issue, but...

Problem 2)
if the corruption of the files occurred in the backup, file history isn't likely to help, as i guess that the backup software doesn't keep full copies of the changed, files, more likely the diffs. so at whatever point the backup was corrupted, all subsequent versions of that file history are also corrupted.

so it seems like I'm looking for a sync/backup software that does its own error correcting code with periodic integrity checking of the backup so whenever its needed, its confidently available.

is what I'm saying accurate? i didn't find many discussions or any answers to this topic, any help would be appriciated.
 
Files at rest can suffer bit rot. I assume this is the issue you're having or considering. There are file systems that can detect, and even sometimes fix bit rot, but it requires extra overhead. These are usually applied to NAS type devices. Probably the best known for this is BTRFS (from the Unix land) and second best is probably ZFS. Even still, you need to have the device regularly run an integrity check, where it basically "reads" every file and makes sure all the checksums are still correct, invoking any self-correction if available.

For online storage with some reassurance, you can get a small (4 or 5 drive) Synology NAS for around $500 or so, but then the drives are gonna set you back another 5x200=$1000 or more, depending on your desired capacity.

Even though a NAS runs RAID, it's still NOT A BACKUP. If your backup "substrate" is not reliable, then you don't actually have any backups. You can make a backup, and then immediately run a verify, and that tells you your backup is good at that precise moment. If your media doesn't hold the data well, that doesn't guarantee your data will still be there later if you need it.

The "optimal" backup strategy is known as 3-2-1. Three copies, on two different kinds of media, one of which is offsite (or online.) The first copy can be on your device, the second copy could be on a removable HDD and the third copy might be in a cloud service, or burned to DVD and stored with a friend (or at the office.)

No matter what you do for backups, you should practice using them on occasion, so you know they're still good. A backup that has failed is not a backup, and your precious content could suffer a loss.
 
Files at rest can suffer bit rot. I assume this is the issue you're having or considering. There are file systems that can detect, and even sometimes fix bit rot, but it requires extra overhead. These are usually applied to NAS type devices. Probably the best known for this is BTRFS (from the Unix land) and second best is probably ZFS. Even still, you need to have the device regularly run an integrity check, where it basically "reads" every file and makes sure all the checksums are still correct, invoking any self-correction if available.

For online storage with some reassurance, you can get a small (4 or 5 drive) Synology NAS for around $500 or so, but then the drives are gonna set you back another 5x200=$1000 or more, depending on your desired capacity.

Even though a NAS runs RAID, it's still NOT A BACKUP. If your backup "substrate" is not reliable, then you don't actually have any backups. You can make a backup, and then immediately run a verify, and that tells you your backup is good at that precise moment. If your media doesn't hold the data well, that doesn't guarantee your data will still be there later if you need it.

The "optimal" backup strategy is known as 3-2-1. Three copies, on two different kinds of media, one of which is offsite (or online.) The first copy can be on your device, the second copy could be on a removable HDD and the third copy might be in a cloud service, or burned to DVD and stored with a friend (or at the office.)

No matter what you do for backups, you should practice using them on occasion, so you know they're still good. A backup that has failed is not a backup, and your precious content could suffer a loss.
yup, I learned the RAID lesson long ago, a raid5 card died and took down all the drives.

the issue i'm having is the 3,2,1... 2 failure modes...
1) they are copies, so if my main files become corrupted, then the extra 2 copies will copy those corruptions.
2) the corruption appears on the backup (this is what your solution mostly addresses)

does anyone worry about #1? is there some sync software that would add in ECC bits and do verification of both the local files and the backup?
 
Last edited:
I used to use Acronis True Image for backups and images (they have changed the name since). It offered periodic validations as an option. If interested please check with Acronis since my info may be a bit dated. I stopped using it for a few reasons. One was they went to a yearly subscription model and no longer offer a yearly perpetual license. I preferred buying it per year knowing each version I bought would continue working with no license issues. Perhaps just my preference on that one. Another thing was it was extremely slow starting, and at least for me needed periodic changes to keep it running as drive letters would be reassigned by Windows. It is not the simplest to set up, but it does offer many options.

The 'last straw' for my using it was when I wanted to temporarily roll back my laptop to an older image due to my mistakenly deleting a program. The restores failed. After many tries I contacted their support, ran commands and sent screen shots as they suggested. Finally I tried my 'gut feeling' and deinstalled and reinstalled Acronis which fixed everything.

My 'issues' with Acronis may not be a game stopper, and I did use it to image and restore another Windows 10 laptop many times without issue. In all fairness it may be worth looking into, and I do keep it installed. BTW it does offer protection from ransomware, and although I never used it there was some form of a blockchain protection.

Since then I tried Macrium Reflect. For a quick look it does not have a periodic backup, but I could have missed it. I can't say for sure, but one of those may be something of use.
 
I've had issues where small parts of files become corrupted, which mess with portions of an image or video, but I dont know when they occured, as I dont often access the files.

Problem 1)
It seems that most programs would see the corruptions in the files and sync/backup the corruptions right?
The first problem is that you need the ECC or whatever on the original files before they are backed up, otherwise you could be backing up garbage. When considering image/video files in particular, the corruption may only be visible to a human. The display software will not know that grass should be green not pink just because a few bits were flipped but it still passes the checksum.
 
Retrospect Backup is a mature system that I used years ago to backup clients in my business and am using now privately. It has served me well over the years on both Mac and Windows platforms. It is very comprehensive and comes in various flavors. It scales from individual clients to enterprise, including options for backing up VMs, Servers, Email accounts etc. It can backup to virtually any media. I trust it, because it has saved my bacon and that of my clients many dozens of times. It is easy to get started with it, but it does have a steep learning curve, with the Windows User's Guide coming in at 617 pages. I use the Retrospect Desktop version on Windows, that allows me to backup any local drives or NAS visible to my machine, plus the same visible to 5 more clients on supported platforms across my LAN. Apologies for the infomercial. I have screen-shotted the comparison section of the backup options and attached it. With more than 20 years of using this product, I have never had anything it said was verified as backed up, that I could not restore. Sometimes I get backup failures, and they can be tricky to diagnose. A bad driver here, or a bad USB dock cable there, or an unreliable dock power supply, or a bad disk. But Retrospect flags an error when other software does not. It is far better to know there is a problem at backup time, than restore time. Of course, I always have verification on, otherwise the loop is open.
 

Attachments

  • Retrospect.png
    Retrospect.png
    82.5 KB · Views: 159
  • Like
Reactions: CSPea