Jump to content
Tyler Leisher

Proper Data Managing on Set

Recommended Posts

Hey all,

 

I'm working on learning to be a Digital Loader/2nd AC, and for my current job I ingest a lot of HD footage from P2 Cards and 5D/7Ds.

 

Up until now, I've just been checking the directory file size and number of files copied but I dont think thats the most efficient (or real) way of checking.

 

So, I'm curious if there are any standard industry proven ways of checking these to make sure:

- The video from the P2 was transfered to the hard drive without problems (Checksums dont match of .mxf vs .mov when ingested.)

- FIles have been copied from one drive to another

 

I know that MD5 Checksumes are normally the standard, but I'm not sure how to go about getting a checksum of an entire folder (including subfolders) on a mac.

 

Should I do a checksum of each file and then compare to the same file? Is there a faster way of generating multiple checksums?

 

I'm also curious what you guys use to backup in the field in terms of an array, are you using a Drobo or G-Raid type RAID system, or do you have a custom built machine to handle that?

 

Any other tips or advice you have for me would be great as well.

Share this post


Link to post
Share on other sites

There is no short cut to producing an MD5 sum. Its purpose is to check for single bit errors, so the computer is intrinsically required to go over every bit in the source file. It isn't quick. That's life. The only way to speed it up is to throw more hardware at it.

 

I'm not so convinced about these USB- or firewire-attached RAIDs. I'd rather have two physically separate devices, so that you can also satisfy offsite-backup requirements - or to put it another way, if building A burns to the ground, you still have backups in building B.

 

Should I do a checksum of each file and then compare to the same file?

Copy the file off the card twice, produce two checksums, and if they're identical you're good. If not, delete both files and retrieve them from the card again, twice. Repeat until you've got a match or until it's failed a reasonable number of times.

 

Here's what slightly perturbs me about this. I'm sure I'd be sneered at, as being entirely underqualified to do this kind of work personally, and yet I constantly get phone calls and emails about it from "real" film ACs asking questions exactly like this. Is nobody running some sort of course?

 

Anyway, I wrote an article on it on my website: http://philrhodes.com/write_wrangle.xhtml

 

P

Share this post


Link to post
Share on other sites

Most MD5 tools work on files, but you really want one per file anyway - so you can then put that hash into the filename, and anyone can check the integrity of the file again at any time in the future, by redoing the hash and comparing it with the one in the filename.

 

P

Share this post


Link to post
Share on other sites

If you're using a mac I would highly recommend you learn how to use the terminal; Using just a plain old mac with osx (or PC with linux) you can copy safely and quickly using Rsync then create an MD5 checksum and compare it using diff. Sounds a little daunting, but if you're serious about data managing, then you better learn it.

Share this post


Link to post
Share on other sites

Hey all,

 

I'm working on learning to be a Digital Loader/2nd AC,

 

Am I being a Luddite thinking they are completely separate grades? I'm sick to death of seeing 1st AC's trying to organise a shoot single handedly whilst the 'Digital Loader' stares at his Mac Book Pro. Either you wrangle data or you're a 2nd AC.....no?

Share this post


Link to post
Share on other sites

I don't think that being a data wrangler should be any more difficult than being a conventional loader - really it should involve less of your time and more machine time, which can happen unattended. When I've written software designed to work in these scenarios, anyway, that's the assumption I've worked on. Copying a few files around, which is what we're really talking about here, should not be a full time job, so my feeling is that if you would not have had a separate loader and second, you should really be able to do without a separate wrangler and second.

 

That said the practical reality is that some current cameras and their support software can be much more attention-seeking than perhaps they should be. There's also the factor that a lot of film-oriented ACs want to sell themselves as computer people when perhaps that's a bit ambitious/

 

Phil's Two Cents: Play by ear. Depends on the camera system.

 

P

Share this post


Link to post
Share on other sites

I'm also curious what you guys use to backup in the field ....

 

We have one show that's taking a radical approach, not backing up in the field at all. They just put the cards from the camera in a pellican case with the camera reports, and at the end of the day, the whole thing goes to the post house. Just like exposed film, only easier.... So far, it's working.

 

 

 

 

 

-- J.S.

Share this post


Link to post
Share on other sites

We have one show that's taking a radical approach, not backing up in the field at all. They just put the cards from the camera in a pellican case with the camera reports, and at the end of the day, the whole thing goes to the post house. Just like exposed film, only easier.... So far, it's working.

 

 

 

 

 

-- J.S.

 

 

I remember reading that Fincher did that on "the social network" I can't see that as being a very viable option these days, as I normally blow through 20+ CF cards /day. I recently did a a 3 Camera RED shoot where we shot 500GB/day, plus I was transcoding on set. This was a very "full time" job.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



  • Glidecam



    Metropolis Post



    G-Force Grips



    Ritter Battery



    Paralinx LLC



    Wooden Camera



    Visual Products



    New Pro Video - New and Used Equipment



    Tai Audio



    FJS International



    Gamma Ray Digital Inc



    Abel Cine



    Just Cinema Gear



    Media Blackout - Custom Cables and AKS



    Rig Wheels Passport



    CineLab



    Broadcast Solutions Inc



    Serious Gear


    Cinematography Books and Gear
×
×
  • Create New...