Jump to content

Proper Data Managing on Set


Tyler Leisher

Recommended Posts

Hey all,

 

I'm working on learning to be a Digital Loader/2nd AC, and for my current job I ingest a lot of HD footage from P2 Cards and 5D/7Ds.

 

Up until now, I've just been checking the directory file size and number of files copied but I dont think thats the most efficient (or real) way of checking.

 

So, I'm curious if there are any standard industry proven ways of checking these to make sure:

- The video from the P2 was transfered to the hard drive without problems (Checksums dont match of .mxf vs .mov when ingested.)

- FIles have been copied from one drive to another

 

I know that MD5 Checksumes are normally the standard, but I'm not sure how to go about getting a checksum of an entire folder (including subfolders) on a mac.

 

Should I do a checksum of each file and then compare to the same file? Is there a faster way of generating multiple checksums?

 

I'm also curious what you guys use to backup in the field in terms of an array, are you using a Drobo or G-Raid type RAID system, or do you have a custom built machine to handle that?

 

Any other tips or advice you have for me would be great as well.

Link to comment
Share on other sites

  • Premium Member

There is no short cut to producing an MD5 sum. Its purpose is to check for single bit errors, so the computer is intrinsically required to go over every bit in the source file. It isn't quick. That's life. The only way to speed it up is to throw more hardware at it.

 

I'm not so convinced about these USB- or firewire-attached RAIDs. I'd rather have two physically separate devices, so that you can also satisfy offsite-backup requirements - or to put it another way, if building A burns to the ground, you still have backups in building B.

 

Should I do a checksum of each file and then compare to the same file?

Copy the file off the card twice, produce two checksums, and if they're identical you're good. If not, delete both files and retrieve them from the card again, twice. Repeat until you've got a match or until it's failed a reasonable number of times.

 

Here's what slightly perturbs me about this. I'm sure I'd be sneered at, as being entirely underqualified to do this kind of work personally, and yet I constantly get phone calls and emails about it from "real" film ACs asking questions exactly like this. Is nobody running some sort of course?

 

Anyway, I wrote an article on it on my website: http://philrhodes.com/write_wrangle.xhtml

 

P

Link to comment
Share on other sites

  • Premium Member

Most MD5 tools work on files, but you really want one per file anyway - so you can then put that hash into the filename, and anyone can check the integrity of the file again at any time in the future, by redoing the hash and comparing it with the one in the filename.

 

P

Link to comment
Share on other sites

  • 2 months later...
  • Premium Member

If you're using a mac I would highly recommend you learn how to use the terminal; Using just a plain old mac with osx (or PC with linux) you can copy safely and quickly using Rsync then create an MD5 checksum and compare it using diff. Sounds a little daunting, but if you're serious about data managing, then you better learn it.

Link to comment
Share on other sites

  • Premium Member

Hey all,

 

I'm working on learning to be a Digital Loader/2nd AC,

 

Am I being a Luddite thinking they are completely separate grades? I'm sick to death of seeing 1st AC's trying to organise a shoot single handedly whilst the 'Digital Loader' stares at his Mac Book Pro. Either you wrangle data or you're a 2nd AC.....no?

Link to comment
Share on other sites

  • Premium Member

I don't think that being a data wrangler should be any more difficult than being a conventional loader - really it should involve less of your time and more machine time, which can happen unattended. When I've written software designed to work in these scenarios, anyway, that's the assumption I've worked on. Copying a few files around, which is what we're really talking about here, should not be a full time job, so my feeling is that if you would not have had a separate loader and second, you should really be able to do without a separate wrangler and second.

 

That said the practical reality is that some current cameras and their support software can be much more attention-seeking than perhaps they should be. There's also the factor that a lot of film-oriented ACs want to sell themselves as computer people when perhaps that's a bit ambitious/

 

Phil's Two Cents: Play by ear. Depends on the camera system.

 

P

Link to comment
Share on other sites

  • Premium Member

I'm also curious what you guys use to backup in the field ....

 

We have one show that's taking a radical approach, not backing up in the field at all. They just put the cards from the camera in a pellican case with the camera reports, and at the end of the day, the whole thing goes to the post house. Just like exposed film, only easier.... So far, it's working.

 

 

 

 

 

-- J.S.

Link to comment
Share on other sites

We have one show that's taking a radical approach, not backing up in the field at all. They just put the cards from the camera in a pellican case with the camera reports, and at the end of the day, the whole thing goes to the post house. Just like exposed film, only easier.... So far, it's working.

 

 

 

 

 

-- J.S.

 

 

I remember reading that Fincher did that on "the social network" I can't see that as being a very viable option these days, as I normally blow through 20+ CF cards /day. I recently did a a 3 Camera RED shoot where we shot 500GB/day, plus I was transcoding on set. This was a very "full time" job.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...