What's new

My new backup and archive routine

Fair enough points @tack and @Dewdman42 and @whiskers

But just to review the bidding, the RAID I'm using is mirroring my backup, not my original files, so the RAID is to protect against drive failure. The "originals" are on the DAW and the PC slave computers and, although they're on an automatic backup schedule, it's once a week, hopefully long enough to discover malware etc.*

Then I take the mirrors periodically and swap them out, putting the fresh ones in the bank. So I guess I have:

1. The original files

2. The backup of those files on an HDD

3. A copy of that backup on another HDD, mirrored; and

4. A copy at the bank.

The copy at the bank is of course out of date, but still a lot better than what would be left after a fire.

I had all my computers sitting by the front door for a week recently, watching ash fall around the house, so it's not a wholly theoretical risk. For me -- suddenly -- nothing happened. Some people were a lot less lucky.

*I actually have daily backups too onto a hard drive of Digital Performer (DAW) files and my Pro Tools recordings. That's the extra-extra backup in case I delete something.

That sounds much more thorough, I'd be comfortable with that. I was just pointing out a general pet peeve of mine, not digging on your solution :) Just dont forget to verify your backups occasionally and you're good!
 
@Dewdman - I uploaded about 8tb of sample libraries in less than a week (I think) with Backblaze. It was very quick, not sure of the exact amount of time/days. That was at our place in LA which has 150mbps upload. In Georgia where we care currently it's about 10 upload, so takes waay longer.

When Crashplan changed to business orientation I looked around for another option and I think it was on ViC that I heard about Backblaze. I'd always found Crashplan a little difficult to manage and understand, but I'm not very tech savvy. Backblaze I find a lot more user friendly.

And the point about backing up locally as well as in the cloud - when my Drobo crashed recently I didn't have my Time Machine going, as we'd moved recently, and I did have to recover about 300gigs of working data from Backblaze. I think it may have taken overnight to prepare and download the stuff, not bad and saved my butt.

Next up is updating my local backup of the sample libraries. This backup stuff takes a lot of time. I'm turning into the disgruntled IT guy. :)
 
I'm calling this my backup project. I'm hoping it will be set up by the end of December. It's involving a lot of culling of files and organizing. I tend to DL everything to my desktop, prior to copying it to my studio computer or my laptop. In someways it had been a back up. I wanted to make an image of my desktop only to realize there is about 800 GB of files in the DL folder.

I probably need to combine it with the move everything on CD and DVD to hard drives project. It is way too much work.
 
Just dont forget to verify your backups occasionally and you're good!

How does one best do that? I use beyond compare, should I occasionally do a full CRC checksum compare between main drive and backup drive instead of the quick compare that gets made during regular backup? I'd have to schedule something like that down to smaller chunks because it would take too long for the whole drive at once.
 
How does one best do that? I use beyond compare, should I occasionally do a full CRC checksum compare between main drive and backup drive instead of the quick compare that gets made during regular backup? I'd have to schedule something like that down to smaller chunks because it would take too long for the whole drive at once.

from an IT perspective, I would be tempted to say attempt a restore, occasionally, but for most people that's not feasible.

Checksums/Hashes would definitely be a good idea, in theory, but really only useful if you haven't changed your source you're comparing to. As soon as you change an item in the source of whatever you backed up, the hash is going to look different. But if you hashed right after, that would be a good integrity check and provide some peace of mind.

Correct me if I'm wrong, BC will tell you if any files are missing/any deltas between two drives or images correct? But i don't think it would account for corrupted files? Been awhile since I've been directly involved in the sysadmin side of things, so others may have better ideas. Cheers!
 
Correct me if I'm wrong, BC will tell you if any files are missing/any deltas between two drives or images correct? But i don't think it would account for corrupted files? Been awhile since I've been directly involved in the sysadmin side of things, so others may have better ideas. Cheers!

BC works on "file level", no images. That's why I like it so much. I have a huge dataset (roughly 3 TB) and make changes to multiple gigabytes of files daily and I don't need any automated versioning, because I'm used to saving versions of my files manually. All I need that backup to do is make sure all files are regularly mirrored to the backup drive (ideally daily) and I disabled deletion of "orphaned" files, so that it doesn't automatically remove a file from the backup when I accidentally delete it on the main drive. A full restore would be entirely impractical and impossible to manually check if the files are all still fine. I was thinking as long as I do an occasional crc check with the file comparison (which so far I have never done addmittadly), it should protect against "data rot" on the backup drive. Is that not the case?

I do a fairly regular "drive rotation" though, to prevent total drive failure in the first place. I buy a new drive every ~2 years, use it as primary backup drive for a few months, then switch the roles of active and backup drive with the main drive. That way my main drive is always less than 2 years old and always was "battletested" for 2-3 months before it got used for "critical" work. I've based those numbers on the statistical clustering of hard drive deaths along the time axis, as demonstrated both by the big google study on hard drive longevity (released many years ago) and my own personal experience with drives.
 
I was thinking as long as I do an occasional crc check with the file comparison (which so far I have never done addmittadly), it should protect against "data rot" on the backup drive. Is that not the case?

Sounds right. DR (disaster recovery) is not really my area of expertise, but my understanding is periodically running the drives and checking the contents will help bit rot/data decay/whatever you want to call it. When running the drives they typically perform SMART checks and can mark bad sectors, etc. from being used (think chkdisk or fsck), so it's definitely better to occasionally check the contents, for sure.
 
I have unlimited space at Jottacloud. Everything gets backed up there. As cheap as Backblaze, less regulations.
 
For services offering cheap unlimited storage, it's very likely the case that your data is accessible to the storage provider. This is how they can offer unlimited storage: by relying on the fact that a good chunk of data can be deduplicated. (If everyone here uploads their copy of Monster Library X, they only need to actually store 1 copy of it.) This is invariably coupled by the requirement to use their proprietary software, rather than a third party backup program (because third party software will be able to properly encrypt the data).

I'd be quite interested to learn about exceptions to this: are there any services offering unlimited storage for around say $10USD a month that present, for example, and S3 API interface for use of third party backup software, and have been offering unlimited for more than 3 years? (The criteria of offering it for more than 3 years is important, because many unlimited storage offerings have come and gone over the years, but they tend to vanish when businesses realize the product isn't sustainable.) These are the really interesting options IMO, especially ones that perform well.

If data privacy isn't a requirement for you (and/or perhaps you have a higher priority conflicting requirement that you want to be able to access your data via web browser), then something like Backblaze Personal Backup (not B2) is hard to beat.
 
For services offering cheap unlimited storage, it's very likely the case that your data is accessible to the storage provider. This is how they can offer unlimited storage: by relying on the fact that a good chunk of data can be deduplicated. (If everyone here uploads their copy of Monster Library X, they only need to actually store 1 copy of it.) This is invariably coupled by the requirement to use their proprietary software, rather than a third party backup program (because third party software will be able to properly encrypt the data).

I'd be quite interested to learn about exceptions to this: are there any services offering unlimited storage for around say $10USD a month that present, for example, and S3 API interface for use of third party backup software, and have been offering unlimited for more than 3 years? (The criteria of offering it for more than 3 years is important, because many unlimited storage offerings have come and gone over the years, but they tend to vanish when businesses realize the product isn't sustainable.) These are the really interesting options IMO, especially ones that perform well.

If data privacy isn't a requirement for you (and/or perhaps you have a higher priority conflicting requirement that you want to be able to access your data via web browser), then something like Backblaze Personal Backup (not B2) is hard to beat.

An option is to have your own encryption software. Encrypt the data you want to store on the cloud...before you push it up to your cloud service provider. Then, no one will be looking at it. Added steps, but for things I for sure want private, I encrypt myself using Boxcryptor.

I chose Sync.com for my cloud provider because they allow me to encrypt and hold the keys, so they are not scanning my data. But, for things like taxes or other financial stuff I want secure, I just encrypt it myself anyway.
 
I chose Sync.com for my cloud provider because they allow me to encrypt and hold the keys, so they are not scanning my data.
Well, you're using their software, so you think they're not holding the keys and can't scan your data. And you have to trust that they're implementing their crypto properly, because it's proprietary and closed source and can't be audited by unbiased third parties without an expensive (and illegal in some jurisdictions) effort of reverse engineering.

If you're going through the trouble of creating encrypted local copies of all your data (and continuously reencrypting when there are changes) then, indeed, you can just point your cloud backup software at that.

Alternatively, if data privacy isn't important to one's backup strategy (which is a personal choice), that opens up plenty of options to choose from.
 
Well, you're using their software, so you think they're not holding the keys and can't scan your data. And you have to trust that they're implementing their crypto properly, because it's proprietary and closed source and can't be audited by unbiased third parties without an expensive (and illegal in some jurisdictions) effort of reverse engineering.

If you're going through the trouble of creating encrypted local copies of all your data (and continuously reencrypting when there are changes) then, indeed, you can just point your cloud backup software at that.

Alternatively, if data privacy isn't important to one's backup strategy (which is a personal choice), that opens up plenty of options to choose from.
You, I like you.
 
Throwing this in the mix.
https://www.pcloud.com/cloud-storage-pricing-plans.html
For €350 you get a 2Tb cloud-drive, you only pay once.
They also provide some solid encryption tools.
(I'm no IT-guru/tech so don't take my word for it, but it sure looks solid)

Cheers,
Anders

A single pay for 2TB that costs about the same as an external drive sounds interesting to me. Thanks for throwing it out there. How secure your data is, will always be a question on cloud sites, as Tack pointed out earlier. I still like the idea of a secondary encryption for really sensitive data.
 
This backup storage thing could end up being as expensive as hardware outboard gear.. wait.. it IS hardware outboard gear!
 
Top Bottom