Backing up/syncing files, how do they deal with hdd errors?

[Larc]

The basic problem I have, is that I dont see any backup/sync software that does it in a safe way.

I've had issues where small parts of files become corrupted, which mess with portions of an image or video, but I dont know when they occured, as I dont often access the files.

Problem 1)
It seems that most programs would see the corruptions in the files and sync/backup the corruptions right?
This would also be an issue if ransomware modified your files.

saving file history can help with this issue, but...

Problem 2)
if the corruption of the files occurred in the backup, file history isn't likely to help, as i guess that the backup software doesn't keep full copies of the changed, files, more likely the diffs. so at whatever point the backup was corrupted, all subsequent versions of that file history are also corrupted.

so it seems like I'm looking for a sync/backup software that does its own error correcting code with periodic integrity checking of the backup so whenever its needed, its confidently available.

is what I'm saying accurate? i didn't find many discussions or any answers to this topic, any help would be appriciated.

 

[PHolder]

Files at rest can suffer bit rot. I assume this is the issue you're having or considering. There are file systems that can detect, and even sometimes fix bit rot, but it requires extra overhead. These are usually applied to NAS type devices. Probably the best known for this is BTRFS (from the Unix land) and second best is probably ZFS. Even still, you need to have the device regularly run an integrity check, where it basically "reads" every file and makes sure all the checksums are still correct, invoking any self-correction if available.

For online storage with some reassurance, you can get a small (4 or 5 drive) Synology NAS for around $500 or so, but then the drives are gonna set you back another 5x200=$1000 or more, depending on your desired capacity.

Even though a NAS runs RAID, it's still NOT A BACKUP. If your backup "substrate" is not reliable, then you don't actually have any backups. You can make a backup, and then immediately run a verify, and that tells you your backup is good at that precise moment. If your media doesn't hold the data well, that doesn't guarantee your data will still be there later if you need it.

The "optimal" backup strategy is known as 3-2-1. Three copies, on two different kinds of media, one of which is offsite (or online.) The first copy can be on your device, the second copy could be on a removable HDD and the third copy might be in a cloud service, or burned to DVD and stored with a friend (or at the office.)

No matter what you do for backups, you should practice using them on occasion, so you know they're still good. A backup that has failed is not a backup, and your precious content could suffer a loss.

 

[Larc]

Files at rest can suffer bit rot. I assume this is the issue you're having or considering. There are file systems that can detect, and even sometimes fix bit rot, but it requires extra overhead. These are usually applied to NAS type devices. Probably the best known for this is BTRFS (from the Unix land) and second best is probably ZFS. Even still, you need to have the device regularly run an integrity check, where it basically "reads" every file and makes sure all the checksums are still correct, invoking any self-correction if available.

For online storage with some reassurance, you can get a small (4 or 5 drive) Synology NAS for around $500 or so, but then the drives are gonna set you back another 5x200=$1000 or more, depending on your desired capacity.

Even though a NAS runs RAID, it's still NOT A BACKUP. If your backup "substrate" is not reliable, then you don't actually have any backups. You can make a backup, and then immediately run a verify, and that tells you your backup is good at that precise moment. If your media doesn't hold the data well, that doesn't guarantee your data will still be there later if you need it.

The "optimal" backup strategy is known as 3-2-1. Three copies, on two different kinds of media, one of which is offsite (or online.) The first copy can be on your device, the second copy could be on a removable HDD and the third copy might be in a cloud service, or burned to DVD and stored with a friend (or at the office.)

No matter what you do for backups, you should practice using them on occasion, so you know they're still good. A backup that has failed is not a backup, and your precious content could suffer a loss.

yup, I learned the RAID lesson long ago, a raid5 card died and took down all the drives.

the issue i'm having is the 3,2,1... 2 failure modes...
1) they are copies, so if my main files become corrupted, then the extra 2 copies will copy those corruptions.
2) the corruption appears on the backup (this is what your solution mostly addresses)

does anyone worry about #1? is there some sync software that would add in ECC bits and do verification of both the local files and the backup?

 

[Ralph]

I used to use Acronis True Image for backups and images (they have changed the name since). It offered periodic validations as an option. If interested please check with Acronis since my info may be a bit dated. I stopped using it for a few reasons. One was they went to a yearly subscription model and no longer offer a yearly perpetual license. I preferred buying it per year knowing each version I bought would continue working with no license issues. Perhaps just my preference on that one. Another thing was it was extremely slow starting, and at least for me needed periodic changes to keep it running as drive letters would be reassigned by Windows. It is not the simplest to set up, but it does offer many options.

The 'last straw' for my using it was when I wanted to temporarily roll back my laptop to an older image due to my mistakenly deleting a program. The restores failed. After many tries I contacted their support, ran commands and sent screen shots as they suggested. Finally I tried my 'gut feeling' and deinstalled and reinstalled Acronis which fixed everything.

My 'issues' with Acronis may not be a game stopper, and I did use it to image and restore another Windows 10 laptop many times without issue. In all fairness it may be worth looking into, and I do keep it installed. BTW it does offer protection from ransomware, and although I never used it there was some form of a blockchain protection.

Since then I tried Macrium Reflect. For a quick look it does not have a periodic backup, but I could have missed it. I can't say for sure, but one of those may be something of use.

 

[miquelfire]

Reflect does periodic backups, but if you're using the free version, it's not an option.

 

[AlanD]

I've had issues where small parts of files become corrupted, which mess with portions of an image or video, but I dont know when they occured, as I dont often access the files.

Problem 1)
It seems that most programs would see the corruptions in the files and sync/backup the corruptions right?

The first problem is that you need the ECC or whatever on the original files before they are backed up, otherwise you could be backing up garbage. When considering image/video files in particular, the corruption may only be visible to a human. The display software will not know that grass should be green not pink just because a few bits were flipped but it still passes the checksum.

 

[BigBilby]

Retrospect Backup is a mature system that I used years ago to backup clients in my business and am using now privately. It has served me well over the years on both Mac and Windows platforms. It is very comprehensive and comes in various flavors. It scales from individual clients to enterprise, including options for backing up VMs, Servers, Email accounts etc. It can backup to virtually any media. I trust it, because it has saved my bacon and that of my clients many dozens of times. It is easy to get started with it, but it does have a steep learning curve, with the Windows User's Guide coming in at 617 pages. I use the Retrospect Desktop version on Windows, that allows me to backup any local drives or NAS visible to my machine, plus the same visible to 5 more clients on supported platforms across my LAN. Apologies for the infomercial. I have screen-shotted the comparison section of the backup options and attached it. With more than 20 years of using this product, I have never had anything it said was verified as backed up, that I could not restore. Sometimes I get backup failures, and they can be tricky to diagnose. A bad driver here, or a bad USB dock cable there, or an unreliable dock power supply, or a bad disk. But Retrospect flags an error when other software does not. It is far better to know there is a problem at backup time, than restore time. Of course, I always have verification on, otherwise the loop is open.