Is LTO tape still favored by the scanning companies or are they moving over to HDD or SDD?

[Daniel D. Teoli Jr.]

Is LTO tape still favored by the scanning companies or are they moving over to HDD or SDD?

[Robin Phillips]

LTO is an archival format, and is way too slow to use for production. HDcamSR or D5 was pretty common for film transfers prior to 2008 or so, but its been HDDs since the HDcamSR factory flooded and killed the format in 2011 if I recall correctly

[Daniel D. Teoli Jr.]

I never used LTO tape, but I thought it was somewhat fast. I was thinking of LTO as an all-inclusive master archive for cine' scans. I would put scans on tape and I could 'grab and go' if I travelled to take a copy with me of the entire library in a tape or two as backup.

I will archive the scans to the optical disc library on M-Disc and a second copy on high grade, archival BD-R. But I can't travel with the disc library, as they are too bulky. So, I guess I'm looking to a backup of the optical disc library.

[Perry Paolantonio]

LTO is a backup format. For our institutional clients, we usually deliver files on a hard drive, but some also want a set on LTOs for backup. They are very inexpensive on a per-GB basis, and highly reliable, and the file copy speed is on par with a good USB3 hard drive. We use LTO 4, 7, and 8 drives, and we use the LTO8 for our internal backups. I've got LTO2 tapes around here from around 2005, and the files on those are still accessible. 

No format is permanent and anyone looking to put files on something and leave it be for 10, 20, 30, 50 years is doing it wrong. It's all about periodic migration to whatever the flavor of the day is. LTO just happens to be the best of these long term solutions right now, with a clear roadmap and decent backwards compatibility (the transition from LTO-7 to LTO-8 notwithstanding). It's had a good run and I expect it will continue to for some time. 

[Aapo Lettinen]

I often did at least one copy on LTO tapes when data managing movie/tv-series material. The main advantage of the LTO is that it does not need to be checked as often as other file based affordable-enough formats and does not need to migrated as often. Thus it saves tons of manual work which equals lots of money saved if the material needs to be stored for long. Especially on documentary stuff the material amounts can be pretty substantial and thus one wants to avoid the need to constantly migrate it to new medium every couple of years. I think the largest projects were about 150TB per project or so and the smaller ones from 40TB to 100TB.

The perfect use for LTO is when material needs to be saved more than 1.5 or 2 years reliably but a maximum of from 10 to 20 years (which necessitates migrating it to new tape once or twice if it is stored correctly). If storing media on hard drives one would need to migrate it to a new drive about 4 or 5 times if storing it for 10 years, and 20 years is not even practical anymore.

A raid-like storage server which automatically migrates material to new drives from problematic ones and constantly backs up everything is a nice solution but can be pretty expensive for large projects and consumes lots of electricity (whereas storing LTO tapes is practically free if the storage room already has approximately ideal temperature/humidity conditions)

[Tyler Purcell]

Simple answer is yes, LTO is the industry standard for long-term storage. 

Many of the commercial clients I've worked with at professional facilities require LTO backup's as part of the contract. So they'll take proxy files uploaded as temp and then get the high res on HDD's and LTO's. 

This way, they can throw the LTO's on a shelf and the HDD"s data can be ingested into their server. 

[Dennis Toeppen]

The best archival medium I am aware of is called Kodachrome.

[Dan Baxter]

It's not used for production, it's used for long-term archival storage. Cheaper than hard drives (the drives themselves are expensive, but the media is cheap per terabyte), and you don't need to read them every year like hard drives. No one uses M-Disc for professional archiving it's not a proven thing, plus the capacity is only at consumer-level quality the reason to use LTO tapes is to back-up scans or restorations at 4K or even higher like 6K or 8K 16-bit DPX which is the master copy that makes everything else (the DCP, bluray, back to 35mm and whatever else). As Dennis says the best archival format is film not digital, but if you're investing in a serious restoration you'll want an archival backup of the final restoration and probably also of the raw scans.

[Daniel D. Teoli Jr.]

On 10/18/2022 at 6:41 AM, Perry Paolantonio said:

LTO is a backup format. For our institutional clients, we usually deliver files on a hard drive, but some also want a set on LTOs for backup. They are very inexpensive on a per-GB basis, and highly reliable, and the file copy speed is on par with a good USB3 hard drive. We use LTO 4, 7, and 8 drives, and we use the LTO8 for our internal backups. I've got LTO2 tapes around here from around 2005, and the files on those are still accessible. 

No format is permanent and anyone looking to put files on something and leave it be for 10, 20, 30, 50 years is doing it wrong. It's all about periodic migration to whatever the flavor of the day is. LTO just happens to be the best of these long term solutions right now, with a clear roadmap and decent backwards compatibility (the transition from LTO-7 to LTO-8 notwithstanding). It's had a good run and I expect it will continue to for some time. 

 

Thanks Perry.

Institutional...yes it seems to be more of an enterprise thing. Maybe if they were more easily connected via USB they would have been more popular with the average data hoarder.

[Perry Paolantonio]

1 minute ago, Daniel D. Teoli Jr. said:

Institutional...yes it seems to be more of an enterprise thing. Maybe if they were more easily connected via USB they would have been more popular with the average data hoarder.

That's not the market for these, never will be, will never happen. there are thunderbolt versions, though, but you pay $1000 more for that.

Edited October 21, 2022 by Perry Paolantonio

[Daniel D. Teoli Jr.]

On 10/19/2022 at 11:39 PM, Dan Baxter said:

It's not used for production, it's used for long-term archival storage. Cheaper than hard drives (the drives themselves are expensive, but the media is cheap per terabyte), and you don't need to read them every year like hard drives. No one uses M-Disc for professional archiving it's not a proven thing, plus the capacity is only at consumer-level quality the reason to use LTO tapes is to back-up scans or restorations at 4K or even higher like 6K or 8K 16-bit DPX which is the master copy that makes everything else (the DCP, bluray, back to 35mm and whatever else). As Dennis says the best archival format is film not digital, but if you're investing in a serious restoration you'll want an archival backup of the final restoration and probably also of the raw scans.

Well, Kodachrome fades, although it was of the better fade resistant films...when it was made. If you like film for preservation, then you would need BW color separation film for color preservation.

When you use a LTO drive is it similar to using a HDD or SDD or is it more complex to back up things, delete and replace, etc?

[Daniel D. Teoli Jr.]

8 minutes ago, Perry Paolantonio said:

That's not the market for these, never will be, will never happen. there are thunderbolt versions, though, but you pay $1000 more for that.

 

Dunno. If they were cheaper, easier to connect and easy to use...why not? Is there something else holding them back from being more mainstream? What is so special that they cost $5,000?

[Daniel D. Teoli Jr.]

On 10/19/2022 at 3:13 PM, Dennis Toeppen said:

The best archival medium I am aware of is called Kodachrome.

I got some early 16mm Kodachrome film I have to scan. This was the early, early formula that faded. I think they changed it around '37, maybe '36. Well, I got the whole transition...early faded Kodachrome and some pretty decent Kodachrome from the late 30's and on to the 40's and 50's. 

Beside Kodachrome, Kodak had changed the dye formula for their dye transfer prints over the years. The dye from the 1950's was pretty poor for fade resistance in the light. By the 90's it was pretty good. Not as good as a pigment inkjet print, but better than the type C prints of that era. Cibachromes were the best for fade resistance in that era.

 Agfacolor print dye stability test - 6 months of sun vs dark storage.

I'll still stick with digital...as long as the electric is on! But if you can get physical analog backups...get em. The more backups the better.

Edited October 21, 2022 by Daniel D. Teoli Jr.

[Adrian Sierkowski]

8 minutes ago, Daniel D. Teoli Jr. said:

 

What is so special that they cost $5,000?

You're paying for the reliability, and the better QC. It is a thing that needs to be made of the best bits, so there is waste that you pay for. It's the same reason an Alexa35 is more than a BlackMagic 4.6K. They both have similar specs, but the Alexa is going to be a hell of a lot better engineered, tested, and have tighter QC.

[Robert Houllahan]

LTO gets used allot in banking and other financial record keeping and I would think medical records too, archiving film scans is just a side hustle for LTO. I think the multi gen compatibility is one factor in the expense.

Production does not really use LTO I have had very few requests for LTO from new production, it is always on drives and how quick can you get it to me, can you upload 5-10000 ft of 4K scans?

I get some people ask can you send them back on a CD? Not professionals tho.

I have an LTO5 deck and should probably get something newer.

[Dan Baxter]

6 hours ago, Daniel D. Teoli Jr. said:

Well, Kodachrome fades, although it was of the better fade resistant films...when it was made. If you like film for preservation, then you would need BW color separation film for color preservation.

Film fades, but stored well it lasts a lot longer than these tapes do. Once digital media is unreadable it's useless.

50 minutes ago, Robert Houllahan said:

I have an LTO5 deck and should probably get something newer.

Note that LTO-9 doesn't support reading LTO-7, so unless you need LTO-9 probably LTO-8 represents better value as it will cover LTO8-6 including Type-M.

[Robert Houllahan]

3 hours ago, Dan Baxter said:

Note that LTO-9 doesn't support reading LTO-7

Well that basically is the end of that formula, I thought it was 2-gen compatible as one of the primary selling points.

No digital information is neglect tolerant.

[Dan Baxter]

55 minutes ago, Robert Houllahan said:

Well that basically is the end of that formula, I thought it was 2-gen compatible as one of the primary selling points.

No digital information is neglect tolerant.

Apologies I always seem to misquote specs. LTO 1-7 can read/write the last two generations and read 3 generations down (e.g. LTO-7 can read LTO-5 but not write). LTO-7 can not read Type-M. LTO-8 and LTO-9 can read/write the last two generations only and not read 3rd generation down. LTO-8 can read/write LTO-7 Type-M. I think adding read capacity 3 generations down now adds too much to the cost of the drive.

[Perry Paolantonio]

 

9 hours ago, Robert Houllahan said:

Well that basically is the end of that formula, I thought it was 2-gen compatible as one of the primary selling points.

 

Don't believe everything you read on the internet. Not all sources are reliable. Ahem. 

Until LTO-8, the pattern was that the current version could read that version and two versions before it. It could write to the current version and the version before it. For example, LTO 6 could write to LTO5 and LTO6, and read LTO 4/5/6.

With LTO 8 they broke the pattern in order to continue with the trend of more or less doubling capacity with each generation. As such, you cannot read LTO 6 tapes in an LTO 8 drive. You can read LTO-7. You can format an LTO7 tape in an LTO8 drive to get about 9TB of storage space but that tape is then incompatible with LTO7 drives. 

LTO9 will read 8/9 and write 8/9. LTO 10 should Read 8/9/10 and write 9/10, etc. 

This is all very clearly spelled out on the LTO roadmap, and on the wikipedia page

 

[Dan Baxter]

Sigh, had Perry not blocked me he would have seen I corrected my error.

Other thing I would note is that the time between generations is increasing. When it was only 2 years then 3 generations made more sense, but you can still buy LTO-6 drives brand new and the time between generations is increasing now (4 years LTO-8 to -9 and it wasn't double capacity) so it's not surprising given the media only lasts 20-30 years when stored per spec. And they most certainly won't be manufacturing the drives that can read them in 30 years. You can still buy LTO-6 brand new, I think that's it though don't take that as gospel as I didn't check every manufacturer.

Also LTO-6 and below drives can be had for peanuts now, that said if LTO-9 makes the most financial sense for your use over LTO-8 then of course go for it. LTO-7 Type-M requires an LTO-8 drive.

[Daniel D. Teoli Jr.]

On 10/22/2022 at 7:01 AM, Dan Baxter said:

Sigh, had Perry not blocked me he would have seen I corrected my error.

Other thing I would note is that the time between generations is increasing. When it was only 2 years then 3 generations made more sense, but you can still buy LTO-6 drives brand new and the time between generations is increasing now (4 years LTO-8 to -9 and it wasn't double capacity) so it's not surprising given the media only lasts 20-30 years when stored per spec. And they most certainly won't be manufacturing the drives that can read them in 30 years. You can still buy LTO-6 brand new, I think that's it though don't take that as gospel as I didn't check every manufacturer.

Also LTO-6 and below drives can be had for peanuts now, that said if LTO-9 makes the most financial sense for your use over LTO-8 then of course go for it. LTO-7 Type-M requires an LTO-8 drive.

Perry blocked you?? Does not sound like Perry. He usually has an argument for everything, if he does not agree. Either something is right or wrong or we don't know. There should not be any shame in searching for the truth. 

I settled on SanDisk G-Drive Pro. 22TB. But once it is reformatted to Windows it drops down to 20 TB. I've never met anyone that has a LTO drive other than hearing about them here. I've never found a custom computer maker that would add the LTO interface to run a LTO drive. OK, the custom makers specialize mainly in gaming computers... but if they can't do it, who can?

 

 

Look, I'm all in favor of the LTO tape as another method of backup. But they are just too underground unless you are an IT guy or gal or zir. 

IF...they were USB plug and play and IF they were half-ass affordable (LTO-8 = $1000) I go for it. But they are just too much for the average Jor or Jane to get into. 

My goal is to put everything on M-Disc or BD-R M-Disc and high-grade BD-R as a backup to the M-Disc. But it is nice to have hard drive easy access to things instead of digging through the optical disc library. And it is good to have other digital backups as well. 

Just be careful what BD-R's you use. Some are terrible for archival work.

Blu-ray Discs …they are not all the same. – Daniel D. Teoli Jr. Archival Collection – II (home.blog)

Now the G-Drives are not cheap. But I can afford to buy one every few months if I save up for one. So far have 40TB of storage. One for general work and one for films. As time goes one, I will pick up backup drives to the main drives. But still need to test out how they hold up. HDD are good for 8 to 10 years they say. Then you have to rewrite to another part of the disc to reenergize the magnetism that holds the data. 

I've transcribed reel to reel tape going back to the 50's that still held the data. Although you can't say what was lost compared to digital. Some old audio tapes suffer from oxide shedding.

Here is what came off of 2 R/R tapes...

 

 

The tape companies should have made LTO more mainstream. They do have one USB LTO drive out there, but it is $5K. Just crazy.

Now I've got problems with external optical disc drives. Windows update killed one of my computers. After the update it won't recognize optical disc drives. Tried most ever fix I could find. Stinking Microsoft. So that shuts down that computer for half my work. Everything gets put on optical disc eventually.

Oh...one tidbit of advice. Don't encrypt of password protect your data unless you want it to die with you.

Who has a drive for this?

 

All photos: D.D.Teoli Jr.

 

 

Edited November 5, 2022 by Daniel D. Teoli Jr.

[Daniel D. Teoli Jr.]

On 10/21/2022 at 11:05 AM, Adrian Sierkowski said:

You're paying for the reliability, and the better QC. It is a thing that needs to be made of the best bits, so there is waste that you pay for. It's the same reason an Alexa35 is more than a BlackMagic 4.6K. They both have similar specs, but the Alexa is going to be a hell of a lot better engineered, tested, and have tighter QC.

 

That's fine, no argument. They got $$$ and $ cameras. But you still have a choice.

With LTO your choice is $$$ or $$$.

Edited November 5, 2022 by Daniel D. Teoli Jr.

[Aapo Lettinen]

35 minutes ago, Daniel D. Teoli Jr. said:

I've never met anyone that has a LTO drive other than hearing about them here. I've never found a custom computer maker that would add the LTO interface to run a LTO drive. OK, the custom makers specialize mainly in gaming computers... but if they can't do it, who can?

The issue with LTO drives has traditionally been that you need to choose all the kit at the same time:  The LTO drive itself, the interface card (traditionally SAS with couple of connector version options) , the possible conversion box to house the interface card if the computer does not have pci-e slots available or the system needs to be changed between computers, then the software you use for doing backups to the LTO (do you use ltfs which may not be the most reliable option for long term storage as backwards compatibility is not guaranteed if you update something, OR do you use a proprietary format which is intentionally kept the same all the time to ensure backwards compatibility but then you may need to buy new version of the software if you update the operating system. I used BRU PE for making LTO backups for the reason that it can be easily read long time in the future... but updating the operating system would have necessitated updating BRU version and purchasing new license so was not possible to update the OSX without paying like 700usd or so) .  Additionally there is the operating system which may become incompatible with the drivers or the LTO backup software if you update it and you never know what pisses off the drivers or software so the best advice is "for the love of God, NEVER update anything in the middle of the project" as it may take a week to get it back up and running again if you mess it up badly. System drive backup clones are handy in case an update messes up the LTO stuff because you can then restore the whole system easily to the state which you know is working correctly.

There may be compatibility issues between the LTO drive  <> connector like SAS <> interface card like SAS to pcie <> the possible encosure if you need to convert the pcie to for example Thunderbolt3 <> the drivers running the LTO drive, the interface card and the possible enclosure <> the operating system <> the LTO backup software .  The exact reason why people see it difficult is because there is so much variables which can go wrong and it is not immediately apparent what is compatible with which and you need to do some research before ordering anything because you may never get it working if you have the wrong parts which are not fully compatible with each other. The research will take at least couple of weeks or up to a month and then it takes from couple of days to a week to get the system up and running if all the components are compatible with each other.

My setup for writing and reading LTO5 tapes was a older external Tandberg LTO5 drive which was connected via sas cable to a pci-e sas card which was mounted inside a pci-e to thunderbolt2 enclosure. That was connected to an iMac with thunderbolt2 cable and I used BRU PE for making the backups. data was usually read from / written to (6hdd) Pegasos2R6 tb2 raid enclosures to get enough r/w speed to keep up with the LTO drive. I could relatively easily move the whole raid box between computers or swap the hdd drives inside to move the data between systems.  I normally needed at least 4 hdd's in raid to get enough read speed to save tape when writing to LTO. It was possible to use single drives too but I needed something like about 20% or 25% more tape for the same data then.

[Niels kakelveld]

SwitchActionTP1179Cut.txtTalking about QIC-80 tapes, some time ago I had a project with a bunch of awesome volunteers to digitize 1/2" mainframe computer tapes from the 60s to 80s, NASA related. Not all tapes were readable, but the bulk was and did so with only a few parity errors.

Neat huh? not bad at all.

SwitchActionTP1179Cut.txt

 

Some Pioneer 10/11 tapes with images that we were able to decode to PNG and a bunch of tapes relating to the Saturn-V rocket with IBM EBCDIC text data, what is on the latter is a mystery!

Edited November 5, 2022 by Niels kakelveld

[Aapo Lettinen]

the thing with LTO is that you save lots of manual work on long term storage because it does not need to be checked and migrated as often as most other formats. So it is well worth the time and initial expense if you want to save the data for 5 or 10 years or more.  setting up an LTO workstation surely takes some time and research but once it is setup, you only need to do some minor maintenance UNLESS YOU UPDATE SOFTWARE (which may lead to catastrophic chain reaction and you need to do everything again and possibly purchase some new hardware or restore the OS to previous version before getting it running again).

So it works reliably as long as you can keep the system as is and reserve it only for LTO use. I think the worst you can do is to try to use the same computer for video editing or gaming AND lto use because the editing software and games need constant updates but the LTO setup does not want any updates ever and you will risk running into problems then. Additionally you can't do much anything else with the computer when it's reading or writing LTO so your beloved editing computer may be unavailable for a day or more if you are doing large backup with it. Not very ideal if you need it for actual work.

Just get a separate computer for the LTO and you'll be fine