Managing Massive Amounts of Footage

Brian Rose · October 13, 2011

I'm about to embark on a massive, and I mean massive footage archiving project for my (new) employer. They possess a substantial collection of historical reenactment footage, which they license as stock footage, and want it all digitized to take their operations to the next level.

The footage spans several decades, and every imaginable format: D2,Beta SP, digibeta, DVCAM, miniDV, HDCAM, HDV, file types of every kind, super 8mm, 16mm, 35mm...

Digitizing the footage will be no problem, as we have all the inhouse facilities for tape transfers, and the film will just be a matter of telecine to pro-res.

What I'm wrestling with is just how to managed so much footage. Rough estimates are about 1000 hours of material. Given DV is around 12 gigs an hour, and you're looking at a minimum of 12 tbs, and really there will be much more than that, since a substantial chunk of that footage is HD material.

Spanning across a bunch of 2tb hard drives seems impractical, and dangerous, as each hard drive I add, the law of large numbers says one will fail, which will SUCK. Not to mention those drives are not exactly idea for long term storage. LTO struck me as another option, though that seems more suited as a backup/storage solution, whereas this material would need to be in a state that is accessible, since the point is to have this material available for license.

We also have a SAN and a fibre network, though at this point it is more of an active terminal, in that it is for live projects, and it's divided into smaller hard drives of 2-4 TB which are each dedicated to a particular editor. Total size is 22 or 24 tbs. Is expansion of the SAN an option, perhaps to establish a dedicated stock drive? Or do I look at constructing a RAID in tandem with our server?

Man, so many, many options, and in many ways I am a novice, and learning as I go. I'd sure love some input from you all, whom I'm sure have worked with large amounts of footage. How do you manage it in a way both efficient in terms of accessibility, while ensuring redundancy and safety of storage?

Best,

BR

Phil Rhodes · October 13, 2011

If this is the basis of a business, it's valuable data, so I think you should put it onto LTO (twice) in any case.

Otherwise yes keep it on a SAN or other network-accessible drive, that would be the normal approach. 12TB of storage is no longer the hundreds of thousands it used to be. If you also have it on LTO if you do decide to do something else you can just pull it back off the LTO.

Why ProRes?

Are you going to create low-res thumbnails for people to view? Is that going to go on a website?

P

John Sprung · October 13, 2011

It's best to get a complete solution from one vendor, so there's only one place to point the finger when something needs to be fixed. I haven't used them, but I've heard good things about Isilon:

http://www.isilon.com/

-- J.S.

Brian Rose · October 13, 2011

Phil,

the pro res is merely the native format (such as our converted 5D material). For online/web purposes we'd be generating motion jpeg proxies with BITC for comping.

I've been reading all day, and consulting with another techie in the office, and I think I'm leaning toward a three level approach:

Level 1: Proxies. This would be available always for use for inhouse comping, as well as for those needing stock footage

Level 2: Online originals. Factoring on cost as well as existing infrastructure, we're probably going to go with spanning the stock footage across 2TB hard drives which would be accessed when it comes time to relink clips, or provide the client with full res versions

Level 3: LTO backup. But considering hard drives DO fail, and as we add hard drives, the law of large numbers suggests a failure is increasingly likely, all the footage would be backed up to LTO 5, whose tapes roughly equal the hard drives we use (1.5 TB). We've been needing a more effective archiving solution for some time, as our current workflow has been to back up raw footage to blu-ray. Of course, we've filled two massive binders already and we're burning our way through a third. I think the environment is right that I'm going to compile a cost comparison report, to show the savings of using LTO for our archival storage.

So thanks all for your help, and if you have anything else to add, I'd be eager to read it!

Best,

BR

Phil Rhodes · October 13, 2011

Sounds reasonable, although the acid test is if your insurers are willing to underwrite it. They are likely to require that you make two sets of LTOs and hold one of them at an offsite location, which isn't difficult to do. At that point I think you're doing more than you need to.

What sort of disk arrays are you using? I'm not so sure about RAID-5, especially in big arrays. It saps performance, often ends up costing more than the extra drives required to maintain a given space or rate in RAID-10, takes forever to rebuild, and loses performance when broken. In many situations the rebuild time for a big RAID-5 is so large that the likelihood of a second drive failure during the rebuild (which will destroy the array) becomes uncomfortably high.

P