Jump to content

Digital "Loading"


Recommended Posts

  • Premium Member

I was asked by a guy who I have loaded film for what I know about digital loading and I had to tell him "not much." I did it for a RED job where I transferred the drives to one external drive, randomly spot checked them, and then made copies to two other drives.

 

Can anyone share the finer points? Is there a "standard" way to do all of this stuff yet? I'd like to be able to tell the guy that I've brushed up significantly enough that he doesn't go elsewhere for his digital work.

Link to comment
Share on other sites

  • Premium Member

Hey Chris. I've done enough of this to fortunately or not be able to call myself a 'Digital Loader,' which is a silly title, but sometimes you have to differentiate yourself from a DIT, unless Production lumps both jobs into one title, but I'd say there is a difference.

 

When I do "Loading" for a RED job, I manage the media - by means of doing the same as you'd do with Mags, Letter them and Build a numbering system - then doing file management backups into Folders, checking the clips, then I keep a detailed spreadsheet I combine with the 2nd AC's camera reports (which they don't always keep) to give Post/Editorial a clearly defined method of matching the Raw clips with the Scene/Take numbers, and if there were any issues or discrepancies with the footage. Then I generate TIFF files, similarly, for post and color correction. I reformat the Mags on my computer and send them back out relabeled. Sometimes productions ask for other things, but that's typically the base package, so to speak. Some productions don't want that all, so sometimes I don't do it. This workflow is for one/two day commercial jobs or low budget features. Otherwise, it'd be great to send the footage to be backed up a third time before formatting drives and cards.

 

Hope that helps,

Gus

Link to comment
Share on other sites

  • Premium Member

The critical thing about this is getting the backups verified. It's terrifyingly easy to go through a whole workflow assuming you have backups, but only find that there was a problem with them once you need them.

 

I've harped on about this endlessly on this and other fora, but while making lots of backups helps, it's really just reducing the chances of failure - to verify things, you need to actually recover the backups and check their validity, and this ideally needs to be done by a third party. Deleting orginal media is potential armageddon and should be done with the same sort of trepidation with which you'd burn the o-neg of a 35mm shoot.

 

So:

 

- Record

- Sanity check viewing - use this as an opportunity to make your paperwork notes

- Copy off the card once (A dub)

- Copy off the card again (B dub)

- Optionally, bit for bit compare A and B dubs (will save time if there's a problem at this point, but on aggregate may save time to find it later, even if it makes one incident take longer to fix)

- Make 2 sets of long-term backups from A dub (#1 set and #2 set)

- Recover long term backups and compare to B-dub

- Get in writing from production that they believe and accept that what came off the card is identical to what went to long term storage

- #1 set of long term backups goes to offsite storage

- #2 set gets used for post.

 

Critically I would never delete anything from original media without a signoff from production.

 

P

Link to comment
Share on other sites

  • Premium Member
Critically I would never delete anything from original media without a signoff from production.

 

P

 

Is that really ever going to happen, though?

 

What's the best way to compare (bit for bit, as you say) the copies and the copies from copies? Is it just going to be looking at file sizes and trusting or is there a more comprehensive method or program to do it?

Link to comment
Share on other sites

Is that really ever going to happen, though?

 

What's the best way to compare (bit for bit, as you say) the copies and the copies from copies? Is it just going to be looking at file sizes and trusting or is there a more comprehensive method or program to do it?

 

As Phil says, the key to success is verifying the transfer. Which in the case of oft chaotic on-set circumstances is easier said than done, unfortunately.

 

A piece of software called Shotput Pro allows to back up to several hard drives at once. The software allows one to choose formats, like SxS, P2, Red, Canon 5D, etc. It is $90 to buy online, and it gives you plenty of control and a log of your transfer at the end of the card back-up process. However, it is not fail proof.

 

Recently, we were transferring a bunch of P2 cards to hard drive with it, but one of them failed without the software warning us about it. It was caught before the card was reformatted, so we were safe, but we weren't on-set. But it is certainly something to be aware of.

 

On Macs, I like the Disk Utility app for creating .dmg's, it has never failed me.

Edited by Saul Rodgar
Link to comment
Share on other sites

  • Premium Member
As Phil says, the key to success is verifying the transfer. Which in the case of oft chaotic on-set circumstances is easier said than done, unfortunately.

 

A piece of software called Shotput Pro allows to back up to several hard drives at once. The software allows one to choose formats, like SxS, P2, Red, Canon 5D, etc. It is $90 to buy online, and it gives you plenty of control and a log of your transfer at the end of the card back-up process. However, it is not fail proof.

 

Recently, we were transferring a bunch of P2 cards to hard drive with it, but one of them failed without the software warning us about it. It was caught before the card was reformatted, so we were safe, but we weren't on-set. But it is certainly something to be aware of.

 

On Macs, I like the Disk Utility app for creating .dmg's, it has never failed me.

 

Shotput seems like a pretty good way to log all of those copies and to have a first check. How exactly did it miss that copy? Was it in "compare file size" or "compare file contents" mode at the time? How did you guys catch the mistake? I hate to say it but it seems like a lot of luck is involved with the average red production.

Link to comment
Share on other sites

Shotput seems like a pretty good way to log all of those copies and to have a first check. How exactly did it miss that copy? Was it in "compare file size" or "compare file contents" mode at the time? How did you guys catch the mistake? I hate to say it but it seems like a lot of luck is involved with the average red production.

 

I'm pretty sure it was in the compare file size mode. It was also set for fastest verification. It did not warn of the problem. the Log text file was fine. I caught it because, since we weren't on set, I tried transferring the files to FCP, where it gave me a "Invalid Format" warning. So, we were able to look for the card (which its always id'd by serial number on the backup), and go from there.

 

The reason it was on fastest verification mode is because it takes long enough as is, if it took any longer I would rather use Disk Utility --which does take longer--but like I said, it has never failed me.

Edited by Saul Rodgar
Link to comment
Share on other sites

  • Premium Member
I'm pretty sure it was in the compare file size mode. It was also set for fastest verification. It did not warn of the problem. the Log text file was fine. I caught it because, since we weren't on set, I tried transferring the files to FCP, where it gave me a "Invalid Format" warning. So, we were able to look for the card (which its always id'd by serial number on the backup), and go from there.

 

The reason it was on fastest verification mode is because it takes long enough as is, if it took any longer I would rather use Disk Utility --which does take longer--but like I said, it has never failed me.

 

So you would use disk utility to make a disk image rather than just copy the contents of the card/drive to a folder? Why?

Link to comment
Share on other sites

  • Premium Member
.... unless Production lumps both jobs into one title, but I'd say there is a difference.

 

I'd say there's a big difference. The loader's job is to get everything you shoot turned over to post exactly as shot. The DIT is your full time on set colorist.

 

Given that we can't afford a whole day's worth of CF cards, what we've done is to put a computer and RAID array in the darkroom on the camera truck. The loader transfers the cards to the RAID, and to a pair of removable solid state jump drives. At breakoff, one jump goes to the post facility, where they dump everything to their SAN and make LTO backups.

 

CF's get re-used as needed after they've been transferred to the RAID and jumps. Jumps get re-used after the LTO's have been confirmed.

 

The RAID is big enough for a whole episode's worth of dailies, which makes them available for review during production.

 

 

 

 

 

-- J.S.

Link to comment
Share on other sites

So you would use disk utility to make a disk image rather than just copy the contents of the card/drive to a folder? Why?

 

What I like about Disk utility is that it is a dedicated one-task-at-the-time app, unlike the Finder which does many things at once. It was specifically created by Apple to handle hard drives, form repair to creating mirror disk images. So far it has not let me down, the Finder has. If the Finder gets over whelmed by other tasks, it is likelier to botch up a file transfer than Disk Utility.

 

 

From wikipedia: A disk image is a single file or storage device containing the complete contents and structure representing a data storage medium or device, such as a hard drive, tape drives, floppy disk, CD/DVD/BD and key drive, although an image of an optical disc may be referred to as an optical disc image. A disk image is usually created by creating a complete sector-by-sector copy of the source medium and thereby perfectly replicating the structure and contents of a storage device.

Edited by Saul Rodgar
Link to comment
Share on other sites

  • Premium Member
As Phil says, the key to success is verifying the transfer. Which in the case of oft chaotic on-set circumstances is easier said than done, unfortunately.

 

 

Oh yeah. That's why I hate backing up on set. I'd much rather treat it like film and send the rushes offsite to be backed up in more controlled conditions.

 

I've always been sceptical of on set backups. The chance of interrupted power, accidental hard drive knocks, constant moving of the set, constant moving of the DIT trolly. It just screams painful. I'm even more sceptical of onset colour correction, a la DIT.

 

On a recent TV series, we simply shot to RED DRIVES instead of card. Every couple of hours of set time we swapped out the drive and sent them off to the lab (in this case Deluxe) with a runner at lunch and the end of the day.

 

Deluxe would do visual and tech checks of every print take, two sets of backups to editorial drives, backed it up to their networked storage and this was then also backed up on their LTO tape system.

 

The main chance of failure with this approach is then the camera drive itself. I feel like it's minimised by swapping it out religiously every couple of hours. Even if the drive fails, there's often a chance of recovery of most of the drive if a particular sector / take goes down.

 

The most you loose is a couple of hours of set time if the drive totally fails. More than likely you'd also know on set if that happens to the drive because the camera will report a fault as you shoot. Not a bad trade off. And i can have one less person on set. Just a loader instead of a DIT and loader.

 

jb

Link to comment
Share on other sites

  • Premium Member

Backing up not on set, but within walking distance, is sort of like what we always had with film -- getting it out of the camera mags and into metal cans. The backup station should always be running on a UPS. As cards and RAM drives become more affordable, sending them to post starts to make sense. Running mechanical drives around is a bit more scary. They're just not as rugged as solid state.

 

I, too, have seen on-set color correction not work worth a damn over and over again. Trying to make a look that way is a fool's errand. But going to tape, having a DIT to make sure what you get is within range for final timing is somewhere between a real good idea and a necessity. It's one of those temporary things that should be going away now, as the technology improves -- like the Technicolor consultants of old.

 

 

 

 

-- J.S.

Link to comment
Share on other sites

  • Premium Member
Backing up not on set, but within walking distance, is sort of like what we always had with film -- getting it out of the camera mags and into metal cans.

 

Not really backing up though is it ? You've just moving it to another container for transport. A step that does have a small degree of risk. But you still only have one instance of your content until it hits the lab and is processed to a more stable and safe state. Even then though there's still only the one instance of the content, but at least it's in a safe, stable location where conditions are more controlled.

 

 

I, too, have seen on-set color correction not work worth a damn over and over again. Trying to make a look that way is a fool's errand. But going to tape, having a DIT to make sure what you get is within range for final timing is somewhere between a real good idea and a necessity. It's one of those temporary things that should be going away now, as the technology improves -- like the Technicolor consultants of old.

 

 

I agree. I've had the best results simply taking a still and using lightroom. I'll email some notes to the colourist. Nothing else is worth the effort and seldom gives a better result.

Link to comment
Share on other sites

  • Premium Member

With film, you're just putting it in another container. With the Raid array and jump drives, we have two portable copies, and one non-portable but more reliable copy. That seems to be working so far.

 

Communication between the DP and colorist is the key here. We want always to have tests in pre-production, and the DP and colorist sit together in the timing bay and establish the look for dailies using those tests. Thereafter, notes by e-mail, voice mail, or whatever is all it takes. The colorist works for you, just like your gaffer and operators do.

 

 

 

 

-- J.S.

Link to comment
Share on other sites

  • Premium Member
With film, you're just putting it in another container. With the Raid array and jump drives, we have two portable copies, and one non-portable but more reliable copy. That seems to be working so far.

 

So 3 copies from source material made on set and verified seems to pretty much be the emerging standard?

 

I'm trying to get a bead on what is "usual" and some of the best practices because I'm starting to be asked about my prowess in loading for digital systems like the RED.

 

 

Another related topic: Anyone have a favorite backup power supply?

Link to comment
Share on other sites

  • Premium Member

Any experience with R3D Data Manager? It seems like a well thought out program for this type of thing.

 

Do any of you keep paperwork of all these file transfers, like a post-camera log so to speak?

Link to comment
Share on other sites

  • Premium Member
So 3 copies from source material made on set and verified seems to pretty much be the emerging standard?

Another related topic: Anyone have a favorite backup power supply?

 

It's what we've done. I can't say for other companies. As the comfort level goes up with time, the backup level may go down.

 

The UPS I have is an APC:

 

http://www.apcc.com/

 

 

 

 

 

-- J.S.

Link to comment
Share on other sites

  • Premium Member

I tend to agree that in an ideal world this would not be done on set, but it certainly can be done on set and, given that, it's probably a tough sell to ask someone to pay for it to happen elsewhere. I'm happy with either scenario.

 

And yes you can get signoff from production. You just need a form for each roll which has two halves: one to be filled in by you, one to be filled in by the people who read back your LTOs and verify they're good. Then all the production office has to do is be the arbiter, be the person who checks that all the numbers match up, and sign the "permission to delete" box.

 

I have used this system on the basis that I was not being paid enough to make those kinds of decisions about the camera original - I'm truly in awe of what loaders take on, for so little money.

 

P

Link to comment
Share on other sites

Any experience with R3D Data Manager? It seems like a well thought out program for this type of thing.

 

Do any of you keep paperwork of all these file transfers, like a post-camera log so to speak?

 

I'd like to chime in here. The emerging standard for all of these issues has been established by the ICG Data Handling workshop. I highly recommend this training for all L600 members.

 

For Red, R3D data manager is the standard. Make sure you use check sums when copying data from place to place to verify that it was copyed correctly. R3D data manager or shot put pro both create logs of every file transfer you make. Make sure to keep this 'reciept' if there ever is a question where/when data was copied.

 

The more copies the better. Make sure they are descrete seperate drives and live in different phyical locations. I recently day played on a Red indie feature where the digital loader had the second dive mirrored to the first drive, and the first drive got acedentally formated.. so it formated the second drive too, and production lost severals days of work. LTO tape back-ups are the best for ultimate archive.

 

Defitely go do the ICG training - two days of intensive study with the best people in the biz doing this kind of work.

Link to comment
Share on other sites

  • Premium Member
Defitely go do the ICG training - two days of intensive study with the best people in the biz doing this kind of work.

 

Yeah, been meaning to but there have been more in Chicago and New York than in LA for some reason. The one we had out here I missed because of work. <_<

Link to comment
Share on other sites

Oh yeah. That's why I hate backing up on set. I'd much rather treat it like film and send the rushes offsite to be backed up in more controlled conditions.

 

I've always been sceptical of on set backups. The chance of interrupted power, accidental hard drive knocks, constant moving of the set, constant moving of the DIT trolly. It just screams painful. I'm even more sceptical of onset colour correction, a la DIT.

 

On a recent TV series, we simply shot to RED DRIVES instead of card. Every couple of hours of set time we swapped out the drive and sent them off to the lab (in this case Deluxe) with a runner at lunch and the end of the day.

 

Deluxe would do visual and tech checks of every print take, two sets of backups to editorial drives, backed it up to their networked storage and this was then also backed up on their LTO tape system.

 

The main chance of failure with this approach is then the camera drive itself. I feel like it's minimised by swapping it out religiously every couple of hours. Even if the drive fails, there's often a chance of recovery of most of the drive if a particular sector / take goes down.

 

The most you loose is a couple of hours of set time if the drive totally fails. More than likely you'd also know on set if that happens to the drive because the camera will report a fault as you shoot. Not a bad trade off. And i can have one less person on set. Just a loader instead of a DIT and loader.

 

jb

 

 

 

Always a great idea to have multiple drives as backups. On a recent commercial, we changed the drive out each time we changed the shot.... There was plenty of time to copy the footage onto multiple drives before you needed that drive back to shoot on.

 

Corey

 

www. red-camera-san-francisco.com

www.post-production-san-francisco.com

Link to comment
Share on other sites

  • Premium Member
I'd say there's a big difference. The loader's job is to get everything you shoot turned over to post exactly as shot. The DIT is your full time on set colorist.

 

John, totally true, but a one or two day commercial operation isn't going to hire two people to do the job. At least not out here.

 

 

I'd like to chime in here. The emerging standard for all of these issues has been established by the ICG Data Handling workshop. I highly recommend this training for all L600 members.

 

For Red, R3D data manager is the standard. Make sure you use check sums when copying data from place to place to verify that it was copyed correctly. R3D data manager or shot put pro both create logs of every file transfer you make. Make sure to keep this 'reciept' if there ever is a question where/when data was copied.

 

The more copies the better. Make sure they are descrete seperate drives and live in different phyical locations. I recently day played on a Red indie feature where the digital loader had the second dive mirrored to the first drive, and the first drive got acedentally formated.. so it formated the second drive too, and production lost severals days of work. LTO tape back-ups are the best for ultimate archive.

 

Defitely go do the ICG training - two days of intensive study with the best people in the biz doing this kind of work.

 

The person who did that should never be allowed on a film set near data management ever again. What a joke.

 

I feel like taking a Seminar on this stuff is overkill. It's not rocket science. I think it just takes a careful and efficient workflow and often some creative problem-solving. I don't think you need training to do it on the level most of us are.

Link to comment
Share on other sites

  • Premium Member
I feel like taking a Seminar on this stuff is overkill. It's not rocket science. I think it just takes a careful and efficient workflow and often some creative problem-solving. I don't think you need training to do it on the level most of us are.

 

I think I'm worried about it because there's a feeling that parts of it are perhaps not within my power to control. When I'm loading film, it's all me. I either do it right or do it wrong and that's that.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...