Do you use RAID for backup?

[Daniel D. Teoli Jr.]

I've read lots of horror stories about things going south with RAID.  But they were the minority in the feedback. Still enough to make me warry.

I would be using it as RAID 1 which I understand is just a dual backup of the data. A few years ago, I tried buying a Sony RAID, but it did not work out and was returned to the seller. It seemed stuck on RAID 0. 

I just use single HDD and SDD for now. For permanent backup it is M-disc and Japanese Verbatim BD-R. What has been your experience with RAID?

[Heikki Repo]

I deal quite a lot with RAID arrays. However, the first and most important thing to understand is that RAID itself should never be considered to be a backup solution - instead it helps you avoid having to restore data from a backup if some member of the array fails. Depending on the array and RAID level used, the number of failures is different. Some RAID levels also give higher performance. The RAID arrays I have worked with have been either volumes for working or for backup.

So, what is your plan exactly? Are you thinking of buying a RAID-1 for a working volume or for a backup volume? Both are possible, but as I mentioned above, you should never think that if your working volume is a RAID volume you already have a backup. Always try to have three copies of the data: working volume, backup volume, off-site volume. Having also one of the backups offline (eg. LTO-tape) helps you avoid data loss if you get malware that encrypts or corrupts your data silently. Your BD-R backups seem to fit this purpose quite well.

Now then, I think RAID is very secure and good technology. Just make sure to buy from a reputable manufacturer. I think the smaller external RAID units made by G-RAIDs have been well liked in the past. I can't really recommend any brand myself, since I usually work with larger external arrays (12 or more HDDs) and smaller internal arrays.

[Phil Rhodes]

I'll reiterate what Heikki said. RAID protects you from hardware failure - to an extent. It does not protect you from human factors and simple mistakes. RAID is not backup.

LTO tape might be backup.

[Daniel D. Teoli Jr.]

On 7/12/2022 at 1:48 PM, Heikki Repo said:

I deal quite a lot with RAID arrays. However, the first and most important thing to understand is that RAID itself should never be considered to be a backup solution - instead it helps you avoid having to restore data from a backup if some member of the array fails. Depending on the array and RAID level used, the number of failures is different. Some RAID levels also give higher performance. The RAID arrays I have worked with have been either volumes for working or for backup.

So, what is your plan exactly? Are you thinking of buying a RAID-1 for a working volume or for a backup volume? Both are possible, but as I mentioned above, you should never think that if your working volume is a RAID volume you already have a backup. Always try to have three copies of the data: working volume, backup volume, off-site volume. Having also one of the backups offline (eg. LTO-tape) helps you avoid data loss if you get malware that encrypts or corrupts your data silently. Your BD-R backups seem to fit this purpose quite well.

Now then, I think RAID is very secure and good technology. Just make sure to buy from a reputable manufacturer. I think the smaller external RAID units made by G-RAIDs have been well liked in the past. I can't really recommend any brand myself, since I usually work with larger external arrays (12 or more HDDs) and smaller internal arrays.

I usually back my work in progress up to at least 2 HDDs or SDDs or a SDD and a HDD. Instead of doing it twice I figured I could do it once to a RAID with duplicate drives that made mirror copies all at once. 

For archival preservation I use M-Disc and Japanese Blue-ray discs, to a lesser degree.

[Daniel D. Teoli Jr.]

On 7/12/2022 at 4:48 PM, Phil Rhodes said:

I'll reiterate what Heikki said. RAID protects you from hardware failure - to an extent. It does not protect you from human factors and simple mistakes. RAID is not backup.

LTO tape might be backup.

 

How does that different? LTO may last longer than a HDD, but it is basically the same thing. Magnetic storage. I'm looking at a RAID as a short term storage vehicle, same as I do now with my HDD / SDD. RAID is just a pile of HDD or SDD. I'm looking at RAID as a time saver. Back up 2 drives stimulatingly instead of consecutively. Or am I looking at RAID wrong?

 

[Phil Rhodes]

If you accidentally delete something from a RAID, you're not protected.

If you have it on another storage medium, you are.

RAID is not backup.

[Tyler Purcell]

I use raids for online work only. 

I use 3.5" bare drives for 5 year backup and have a drive copy box that automatically makes copies of drives. 

So every few years, I pluck a drive out of storage, plug it into the copier, copy the data over from one to another and then store the new drive in the safe. 

Generally backups shouldn't be raids of any kind, but I do have a few older drives that have media on them, which are 2x raid 0 due to how big they are at the time. 

[Phil Rhodes]

I think a lot of people are basically doing that because there's almost no other cost-effective solution.

This is what LTO is for, but LTO is too expensive for many situations. I am constantly dismayed by the failure of the LTO people to make it more popular than it is, which would mean addressing the cost of entry in a way that would mean restructuring the whole way they market the thing.

As a technical writer I get a lot of announcements of new ideas in this area, none of which ever seem to make it to productisation.

The problem people should be aware of is that hard drives are not designed to be, and are not very good at being, a long-term archival storage medium.

[Tyler Purcell]

3 hours ago, Phil Rhodes said:

I think a lot of people are basically doing that because there's almost no other cost-effective solution.

Yep and LTO changes so frequently, it seems worthless. I have hard drives that are still working from the early 2000's and LTO's from that period are completely out of date and unworkable. So it's a big problem for sure. 

[Daniel D. Teoli Jr.]

On 7/22/2022 at 8:10 AM, Phil Rhodes said:

I think a lot of people are basically doing that because there's almost no other cost-effective solution.

This is what LTO is for, but LTO is too expensive for many situations. I am constantly dismayed by the failure of the LTO people to make it more popular than it is, which would mean addressing the cost of entry in a way that would mean restructuring the whole way they market the thing.

As a technical writer I get a lot of announcements of new ideas in this area, none of which ever seem to make it to productisation.

The problem people should be aware of is that hard drives are not designed to be, and are not very good at being, a long-term archival storage medium.

The SAS interface may cost $500 to $600 alone. They should make a secure USB port that screws in like the old cables did in the early days of computing to run a USB LTO drive. I guess the LTO makers just don't care about making it affordable. The LTO drive cost as much as 6 - 8 basic computers. 

Personally, I love M-Disc and maintain an M-Disc optical disc library. The only 3 shortcomings of M-Disc are:

1) Cost

2) Limited storage capacity.

3) Low availability.

The best the M-Disc can do is about 93GB capacity. If they could make an M-Disc that holds 300GB to 500GB, then you could have something great to archive on. 

Still, I'd like to get into tape in addition to M-Disc...if it was anywhere near affordable. 

Edited October 23, 2022 by Daniel D. Teoli Jr.

[Tyler Purcell]

On 10/22/2022 at 5:30 PM, Daniel D. Teoli Jr. said:

Personally, I love M-Disc and maintain an M-Disc optical disc library.

The idea of storing on to a single type of media and then forgetting about it, doesn't work either. Optical drives are a dead way to store media. In 15 years, when even the bus technology required to hook them up is long gone, how will you read them? I threw away my Jazz, Zip, Syquest, floppies and CDR's, because they were nearly impossible to work with. Even the formatting of them requires an older system on Mac at least. I know today, things aren't changing rapidly in the formatting department but they may. 

LTO is only a great option IF you keep the drive AND the computer for a decade and then upgrade all of it to the next standard then. 

So far, I have hard drives that have outlasted all expectations AND because they're USB/Firewire, they can be read no problem on modern systems. 

[Daniel D. Teoli Jr.]

On 10/23/2022 at 8:59 PM, Tyler Purcell said:

The idea of storing on to a single type of media and then forgetting about it, doesn't work either. Optical drives are a dead way to store media. In 15 years, when even the bus technology required to hook them up is long gone, how will you read them? I threw away my Jazz, Zip, Syquest, floppies and CDR's, because they were nearly impossible to work with. Even the formatting of them requires an older system on Mac at least. I know today, things aren't changing rapidly in the formatting department but they may. 

LTO is only a great option IF you keep the drive AND the computer for a decade and then upgrade all of it to the next standard then. 

So far, I have hard drives that have outlasted all expectations AND because they're USB/Firewire, they can be read no problem on modern systems. 

 

 

I've got CD's going back to the 1980's that still work fine Tyler. VCD's that go back to the early 2000's that still work fine. Here is an early VCD I had made back in '03. Works great. But I just keep it in the preservation Archive as a sample Tyler. And these are not even M-Disc Tyler. 

VCD's were popular between the VHS and DVD era as a bridge of sorts to downsize from Laserdiscs.

Now, I have had trouble with getting a custom computer not recognizing optical drives. So sometimes there is that problem. All that stuff you mentioned (LTO & HDD) is magnetic and subject to decay and problems. It is not archival. Engraved optical media (or if they ever come out with engraved synthetic quartz) is the old method of archival preservation when it comes to digital Tyler.

But if you are using AZO discs you got to be careful how you store them. They may only be good for a decade or two...if stored right. If stored wrong, then they can be dead in a few weeks.

Same DVD...top DVD was stored 'wrong' and has bronzing.

LTO tape looks like a nice method for additional backups, but it is too underground Tyler. The companies need to make it more mainstream and affordable. 

Edited October 26, 2022 by Daniel D. Teoli Jr.

[Tyler Purcell]

5 hours ago, Daniel D. Teoli Jr. said:

I've got CD's going back to the 1980's that still work fine Tyler.

Manufactured discs are not burned discs, they are entirely different production processes. 

The first consumer affordable burners didn't come out until 1993. 

5 hours ago, Daniel D. Teoli Jr. said:

VCD's that go back to the early 2000's that still work fine. Here is an early VCD I had made back in '03. Works great. But I just keep it in the preservation Archive as a sample Tyler. And these are not even M-Disc Tyler. 

VCD's and CD's have a wide laser, they also have a lot of error correction. BluRay's (which are M-Disc) do not have anywhere near the same error correction. The 100GB versions are 4 layers as well, meaning they have way more potential for failure. 

Burned discs work using a dye that can go bad over time. The original discs from the mid 90's seem to hold up really good. Once everyone switched over to CD burning for media in the late 90's, the quality of discs dropped off substantially. All if my personal archives prior to HD were done on DVD's and most do not playback anymore. Many never played, just duplicates, stored in cases in a safe, DVD players simply can't play them. This is a very common occurrence. I use to work at a post house who archived onto CD's/DVD"s and we found that nearly half of the archive, (which by the way was stored in a temp controlled film vault), were bad. We'd want to re-constitute avid EDL's and such, but the disc's simply wouldn't playback. We had to use data recovery software to get anything off the discs and sometimes it worked, other times it didn't. I've found the same issue with my discs as well. 

 

5 hours ago, Daniel D. Teoli Jr. said:

VCD's were popular between the VHS and DVD era as a bridge of sorts to downsize from Laserdiscs.

Yea VCD's were horrible MPEG 1 352×240 video, didn't hold a candle to the much higher bit rage DVD and even laserdisc in most cases. We never bothered with the format at any of the companies I worked for OR personally. Since most consumers had no way to playback VCD in the US until DVD players came out, it didn't really care much about the format. 

5 hours ago, Daniel D. Teoli Jr. said:

Now, I have had trouble with getting a custom computer not recognizing optical drives.

So yea, imagine reading that same media in 15 years? 

[Aapo Lettinen]

LTO is not even particularly expensive if one uses the older generation drives so that the drive itself is not costly.

The main issue of the LTO is that the device drivers are insanely picky and they may stop working if you change anything in your system, especially if updating the operating system. Additionally you can't usually do anything else with the computer when it is writing or reading LTO tapes and thus your computer is reserved for that task even couple of days in a row if you are writing large libraries with lots of tapes on them.

When I did LTO backups/archives at work I had a separate computer dedicated to only LTO use and nothing else. I basically froze that computer in time so that it was NEVER updated, NEVER connected to the internet in any way so that no auto update could screw anything up, and no one else was allowed to use the computer for anything. That was the only way to ensure that the LTO workstation worked correctly all the time, every time and there was no corruption on the tapes and no speed issues etc. when writing or reading the tapes. I had thunderbolt raid boxes with 6 hdd's each and close to 1GB/s rw speeds which I used for storing data which was going to be written to tape or read from tape. This was crucial for saving tapes because LTO has built in error correction and it cannot rewind back when it detects the written data block was faulty: it has to work full speed forward and cannot stop instantly so it just writes the same data again if the first time did not work out. Meaning that if your storage drive is slow it takes more tape to write the same data because there will be more write errors and thus more rewrites than there otherwise would be.

So a bit older LTO drive which is bought for cheap used, a separate computer which can be frozen in time to ensure that updates don't screw up the entire system, and the separate computer can't be used for anything else when writing or reading tapes.

Oh, and get a separate room or enclosure for the tape drive. They tend to be noisy, up to a point of getting  a hearing damage if you work next to it for months/years