Storage Solution for 4K?

Matan Uziel · November 20, 2016

Hello,

I am trying to figure out how storage is being saved and downloaded in big productions in Hollywood? I am shooting in 4K and need a large space per production (approx 20TB) and I find it very tiring and time consuming to download from SD cards to my computer each time a memory card is out of space. Is there a storage device that I can connect to a camera so it downloads automatically, like a Cfast 2.0 multiple SSD reader or something? How exactly it is done in big movies? I'm sure they care not downloading the footage every 1 hour....

Many thanks!

Tyler Purcell · November 20, 2016

With professional shoots, the footage is ingested on set as they are shooting. This is the job of the DIT and using Cfast 2.0 cards with thunderbolt connection, a 512gb card with a thunderbolt storage, it takes around 15 - 20 minutes to download! It's super fast. They'll use two identical raid solutions and software like shotput pro, which distributes the files between two drives at once. They'll also use DaVinci and on the fly, do the low-res transcode on the fly.

Thunderbolt has been the "industry standard" for on-set DIT work since it came out a few years ago. As a consequence, most DIT's use Mac's because there is no comparable portable substitute for PC/Windows machines. USB 3 is to slow, E-Sata is to slow, Firewire is WAY to slow and almost all of the other solutions like 8GB fiber and a multitude of direct attached solutions, aren't designed for portable use.

At home my computer has 6 x 6Gbps sata ports on the motherboard. So I can run a boot drive, optical drive and 4 storage drives as a raid 0. I can put 4 x 8TB drives in there if I want and get 32TB of storage without the cost of lack of speed that's common with external storage solutions. My current system has 3 x 4TB drives, which raided together is plenty of room for a 4k Pro Res XQ feature. It's around 550MBps throughput and I have a super fast USB 3.1 card in my machine, so 512GB cards download in around 40 minutes.

The way I work is to use those little 2.5" portable drives on set. I'll download the cards of whatever camera original we use, through shotput pro, onto two 2.5" drives. I'll store the A set in my safe and the B set will go with the director. Every night after shooting, I'll run a batch transcode overnight onto my clean internal raid. It's risky to have the media out overnight, but it's a low risk compared to the alternatives. I rinse and repeat every day on set, until the drives are all full. This last show we had 10 2TB drives, 5 for the A set and 5 for the B set. By the time the show is finished, everything is already on the editing bay ready to roll, the A set will then permanently live in my safe. Since everything has already been transcoded for editing, I will then duplicate the transcoded media and put that in my safe as well. This way if anything goes wrong, it's all good. I will pull it out every once in a while as assets are added, to insure the backup and the master are identical to one another. On this last show, the director wanted a "portable" version of the transcode, so I bought an inexpensive Raid 5 ESata/USB3 array, so he could be working on stuff whilst I was.

During finishing, we simply plugin all the drives with the camera originals and re-link them in DaVinci. Then we duplicate JUST that media onto the working raid and color the film. Final output comes out of DaVinci after correction.

Unfortunately, there isn't any inexpensive automatic download system. This is because you still need some operating system to make storage work properly. So why not use a cheap computer? 80% of the time I use standard SD, SXS and QXD media. C Fast is a bit more rare, it's only on a few cameras. The other 20% of the time, it's RED media, which is expensive and kinda slow. RED has newer cards for their newer cameras but the cameras I use, the cards are pretty slow compared to CFast.

Hope that helps.

Matan Uziel · November 20, 2016

With professional shoots, the footage is ingested on set as they are shooting. This is the job of the DIT and using Cfast 2.0 cards with thunderbolt connection, a 512gb card with a thunderbolt storage, it takes around 15 - 20 minutes to download! It's super fast. They'll use two identical raid solutions and software like shotput pro, which distributes the files between two drives at once. They'll also use DaVinci and on the fly, do the low-res transcode on the fly.

Thunderbolt has been the "industry standard" for on-set DIT work since it came out a few years ago. As a consequence, most DIT's use Mac's because there is no comparable portable substitute for PC/Windows machines. USB 3 is to slow, E-Sata is to slow, Firewire is WAY to slow and almost all of the other solutions like 8GB fiber and a multitude of direct attached solutions, aren't designed for portable use.

At home my computer has 6 x 6Gbps sata ports on the motherboard. So I can run a boot drive, optical drive and 4 storage drives as a raid 0. I can put 4 x 8TB drives in there if I want and get 32TB of storage without the cost of lack of speed that's common with external storage solutions. My current system has 3 x 4TB drives, which raided together is plenty of room for a 4k Pro Res XQ feature. It's around 550MBps throughput and I have a super fast USB 3.1 card in my machine, so 512GB cards download in around 40 minutes.

The way I work is to use those little 2.5" portable drives on set. I'll download the cards of whatever camera original we use, through shotput pro, onto two 2.5" drives. I'll store the A set in my safe and the B set will go with the director. Every night after shooting, I'll run a batch transcode overnight onto my clean internal raid. It's risky to have the media out overnight, but it's a low risk compared to the alternatives. I rinse and repeat every day on set, until the drives are all full. This last show we had 10 2TB drives, 5 for the A set and 5 for the B set. By the time the show is finished, everything is already on the editing bay ready to roll, the A set will then permanently live in my safe. Since everything has already been transcoded for editing, I will then duplicate the transcoded media and put that in my safe as well. This way if anything goes wrong, it's all good. I will pull it out every once in a while as assets are added, to insure the backup and the master are identical to one another. On this last show, the director wanted a "portable" version of the transcode, so I bought an inexpensive Raid 5 ESata/USB3 array, so he could be working on stuff whilst I was.

During finishing, we simply plugin all the drives with the camera originals and re-link them in DaVinci. Then we duplicate JUST that media onto the working raid and color the film. Final output comes out of DaVinci after correction.

Unfortunately, there isn't any inexpensive automatic download system. This is because you still need some operating system to make storage work properly. So why not use a cheap computer? 80% of the time I use standard SD, SXS and QXD media. C Fast is a bit more rare, it's only on a few cameras. The other 20% of the time, it's RED media, which is expensive and kinda slow. RED has newer cards for their newer cameras but the cameras I use, the cards are pretty slow compared to CFast.

Hope that helps.

Thank you for your detailed response. So, let's say I have RED Dragon or Sony FS7 or Canon C300 Mark III. Is there a way to connect a multiple slot cfast 2.0 card reader/writer straight to the camera with a cfast 2.0 adapter? In other words, how can a camera write into multiple cards and then I download the footage without disconnecting the hardware from the camera? Thank you!

Tyler Purcell · November 20, 2016

Yea, the cameras can write as a raid zero, so both cards get the same data. You can't connect anything to a camera to record CFast that I know of.

Phil Rhodes · November 21, 2016

Long term, I guess it's worth mentioning that the only real solution is LTO tape. It's acoustically noisy, and expensive. But it works.

Tyler Purcell · November 21, 2016

Yep, LTO is currently the best method of long-term storage once you're finished with the project.

Matan Uziel · November 21, 2016

Guys, I have found this and it looks very reliable: https://www.bhphotovideo.com/c/product/1245606-REG/atoch_c2s_cfast_to_ssd.html

I am still looking for a way to connect a camera to SSD drives. I am sure there are more ways. I for one cannot believe that in Hollywood they change the cards all the time. Even if you have 2-3 DITs on set, 300TB is a lot of storage.

Opinions?

Tyler Purcell · November 21, 2016

Yea, they change cards all the time. I've been working on features and television shows in the camera department for years. I actually developed one of the original real-time replay systems for early digital cinema.

Everyone swaps cards, there is still a "loader" on digital sets, it's quite amazing.

I haven't seen that Cfast to SSD box before, that thing looks brand new.

I know a lot of top DIT's who work on huge movies and they've told me, even on bigger 4k shows, they rarely go past 100TB. Heck, this show I'm cutting now is a 4k feature shot with 2 cameras and the bulk media is only 10TB. The biggest show I've personally worked on was a bit less then 50TB of camera raw. We condensed it down to 4TB after transcoding from 4k to 1080p for editing.

So yea, I don't think it's a problem... just don't shoot 100:1 ratio! LOL :P

Aapo Lettinen · November 21, 2016

I have understood that working with the Cfast to SSD box is quite unpractical and unrealiable (too many vulnerable cables, possible data rate issues, etc.) in real world situations. I have seen some examples on web of this done with for example Ursa Mini but the users didn't recommend it for general use.

Generally one just has a laptop with some DIT software which does checksum verified transfers (like Shotput Pro or Silverstack) and backups the cards during the day whenever possible, then reuses the cards right away.

I mostly use 3.5" SATA drives for data storage both on set and in post production (on set I have two thunderbolt RAIDs so that the offloading from card is faster), if you have good enough HDD dock they are both the fastest and the most cost effective way to store large amounts of data during the production.

you crew needs of course some experience of handling them because the circuit board is exposed and vulnerable but otherwise they are the best way for short to mid term storage and for long term you can do a LTO backup of the SATA drives and then reuse them if necessary

Aapo Lettinen · November 21, 2016

I mostly use 3.5" SATA drives for data storage both on set and in post production (on set I have two thunderbolt RAIDs so that the offloading from card is faster), if you have good enough HDD dock they are both the fastest and the most cost effective way to store large amounts of data during the production.

you crew needs of course some experience of handling them because the circuit board is exposed and vulnerable but otherwise they are the best way for short to mid term storage and for long term you can do a LTO backup of the SATA drives and then reuse them if necessary

this is for about 30TB - 100TB per production, if you have productions of for example 3TB of raw material, then it might be practical to use more-expensive-per-TB-and-transfer-speed storage solutions like 2.5" external usb or thunderbolt drives. But the 3.5" drive + hdd dock solution is excellent if you are working on multiple projects at the same time and need large amounts of relatively fast storage which can be swapped quickly if needed

Phil Rhodes · November 21, 2016

I'm with you on price and performance, but I wish someone would make a cheap case for 3.5" drives the same way blackmagic did for SSDs.

Perry Paolantonio · November 21, 2016

You mean something like this? hard to beat < $3

https://www.amazon.com/3-5-Hard-Drive-Protection-BLACK/dp/B0057R18NY/ref=sr_1_3?ie=UTF8&qid=1479741886&sr=8-3&keywords=3.5%22+hard+drive+box

Or these (more expensive, look like videotape cases):

https://www.amazon.com/Plastic-Holder-Flower-Print-Rectangle/dp/B007Q92VS2/ref=sr_1_4?rps=1&ie=UTF8&qid=1479741942&sr=8-4&keywords=3.5%22+hard+drive+box&refinements=p_85%3A2470955011

Edited November 21, 2016 by Perry Paolantonio

Phil Rhodes · November 21, 2016

Oh, there you go. The second one looks good.

Perry Paolantonio · November 21, 2016

We've also used these for larger batches of client drives. Great for shipping.

https://eshop.macsales.com/item/Other%20World%20Computing/HDCTN10/?utm_source=google&utm_medium=shoppingengine&utm_campaign=googlebase&gclid=COGz2N-ZutACFctKDQod7YoMqg

Aapo Lettinen · November 21, 2016

We use this type of Chinese hdd cases, they are similar model than ICY BOX ib-ac602a (probably copied from those) but 6 times cheaper and almost similar quality, only the plastic is just a tiny bit more fragile but not a problem in most situations http://www.ebay.com/itm/3-5-Dustproof-Protection-Box-Case-For-SATA-IDE-HDD-Hard-Disk-Drive-Storage-New-/192028760914?var=491706032150&hash=item2cb5ce5b52:g:1IQAAOSwA3dYKYlS

Phil Rhodes · November 21, 2016

We've also used these for larger batches of client drives. Great for shipping.

Those are just the inner cartons used by manufacturers to ship disks. I wouldn't view that as sufficient packaging for most types of transportation, let alone with material on them!

P

Perry Paolantonio · November 21, 2016

Those are just the inner cartons used by manufacturers to ship disks. I wouldn't view that as sufficient packaging for most types of transportation, let alone with material on them!

P

Bare bulk drives (10 packs) are shipped in exactly these boxes (sometimes foam, depends on the manufacturer). Even single bare drives come in almost identical plastic holders that stabilize the drive within the box. We've been shipping drives in those single-disk holders for 15 years without a single failure.

For these bulk boxes, we put this box inside another for shipping because the cardboard itself isn't particularly sturdy.

-perry

Phil Rhodes · November 21, 2016

Yes I've seen them, I always thought they were a bit inadequate!

Perry Paolantonio · November 21, 2016

I think they're sturdier (hard drives) than we think. I've had clients send a bare drive in nothing more than a padded fedex envelope, with no static bag, and it still worked. Not that I'd recommend that or do it myself, but it happens.

-perry

Keith Walters · November 22, 2016

I think they're sturdier (hard drives) than we think. I've had clients send a bare drive in nothing more than a padded fedex envelope, with no static bag, and it still worked. Not that I'd recommend that or do it myself, but it happens.

-perry

Drives are generally only fragile while they're spinning. Once power is removed, a spring-loaded assembly pulls the read-write heads off to the edges of the magnetic discs where nothing is recorded. I've seen portable hard drives treated quite shockingly, but oddly enough it's usually only the USB connector that suffers.

On the other hand in my garage I've got some workbenches I made from the 12-layer plywood 6 foot x 3 Foot panels that they used to ship hard drives in to PC assemblers in 15 years ago. They were bolted together with these huge 3/4" black bolts, and you'd think any old crap bolts would have done for that, but these were top-of-the line products, real works of art!

The drives themselves were sandwiched in several layers of bubble wrap; they were clearly determined those drives were going to arrive in prime condition!