Red Workflow

John Sprung · March 11, 2009

... the level of reliability to which digital acquisition is being held is vastly beyond the reliability of photochemical media.

Digital is sort of like the one and only new guy on a very long established crew, under intense scrutiny from everybody. Until it builds up a track record like film and tape have, we'll choose to err on the side of caution. Kinda like we stretch black tape around film magazines that really don't need it.

I think we're really saying pretty much the same thing: Look at it at the earliest possible point. If a reasonable picture is there, protect it with a checksum for the rest of its life. MD5 may be the black camera tape of the digital world. ;-)

-- J.S.

Keith Walters · March 14, 2009

It's worth pointing out, in the interests of not scaring people, that the level of reliability to which digital acquisition is being held is vastly beyond the reliability of photochemical media.

P

There seems to considerable confusion of terminology here. Most of the Great Unwashed seem hopelessly shackled to some nebulous notion of: “Film vs Digital”, having securely and unshakeably espoused the notion that the term “digital” is entirely interchangeable with “electronic capture,” whereas “film” is irretrievably locked away in the Phantom Zone of “Analog”.

Earth to Fanboys: At risk of further deterioration of an already well-worn record, ALL image capture devices in current use are 100% Analog. At which point the analog signal is digitized and all subsequent massaging carried out with the illegitimate descendants of Charles Babbage’s particular error of judgement, depends on the particular technology used.

You can still make perfectly good films using the much-maligned “photochemical” processes all the way. You could, if you wished, even eschew the use of Mr Hollerith’s vile descendants for the editing process, and simply hand cut a work print the old-fashioned way, (by the light of a kerosene lantern if you’ve spent all your money on the production and thus neglected to make timely compensation to whichever clone of Thomas Edison’s ne’r-do-well issue you have chosen to patronize).

OR you could shoot on film, and still take advantage of Mr Fox-Talbert’s excellent and genetically flawless descendent to quietly and effortlessly transmogrify the brutal extremes of photonic flux encountered in the workaday mundanity of the average film location, into an admittedly inaccurate but most definitely not unreasonable and certainly not unpleasing facsimile of the original tonal range, but which can then be easily coped with by that toothless and illiterate but still necessary peasant living on the edge of the estate: The. Film. Scanner.

OR, those who have little or no notion or regard or purse for excellence, can simply cut to the chase and basically stick a lens directly in front of the silicon film scanner, which is in essence, what a “digital” camera actually consists of.

Storage is where total confusion unfortunately reigns. We now seem to have a world arbitrarily split such that "analog" apparently refers only to photochemical storage and its particular issues, while “digital” refers to computer-type storage, ie flash memory and/or Hard Disk drives and all their problems.

Digital Tape appears to be the idiot bastard son that nobody talks about any more…

Let’s look at the advantages of the three systems

Of the three basic ways of storing pictures, photographic film has the longest proven track record. There are surviving examples of 35mm movie film that was shot in the19th century, and with care, a modern film scanner can extract remarkably good images from them. With a bit of judicious post-production they can yield images of considerably higher quality than anyone would have seen at the time they were shot. (The vast majority of “vintage” 35mm footage you are likely to see is unfortunately most unlikely to be anywhere near a first generation scan – it can be a real shock when you first see a 1930s film scanned from the original negative).

And we still have this original and viewable 19th century footage, in spite of the fact that the original film stock was made under unbelievably primitive manufacturing conditions, with no thought for (or even concept of) archival storage, and more often than not, stored for decades under conditions not even vaguely resembling an ideal archival environment.

One significant downside of film as an archival medium is that any copies made from the original negative can only ever “photographs” of the original, with an unavoidable amount of image degradation. This is one major advantage digital tape has over film: as long as the dreaded error correction “Algorithms” are working correctly and the tape is still in reasonable condition, 100% data recovery is the norm, which means that a digital copy of a digital tape can be an absolutely identical copy, and a 100th generation copy can look identical to a first generation digital copy. (Incidentally the same applies whether the original image came from the analog pickup device in an electronic camera, or was scanned from film, but like the little men who operate Bunraku puppets, this particular fart in the Church-of-The-Fifteenth-Coming-of-Celluloid’s-Angel-Of-Death is apparently meant to be ignored).

Otherwise film has many attractive features:

1 it is a universal format. There would be very few places in the world where you could not get film scanned to just about any video format.

2. It is technologically transparent. If some alien archaeologist were to stumble on a roll of 35mm film, it would be immediately obvious what it was, and any reasonably advanced society would soon be able to build a device capable of displaying the images.

3. All the information required to extract a particular frame of film is contained in that frame. A film scanner or projector does not need to first extract decoding information from one part of the film before it can decode another.

Earlier digital videotape formats share many of the advantages of film in that each frame (or field) is like a separate JPEG file, so the tape cassette can wound forward or backward to any position, inserted into the machine and it will play immediately. Indeed, you could completely remove the tape from its cassette, and as long as you could reliably feed it through the VTR’s tape guide, the tape going past the heads would play perfectly, regardless of what happens to the rest of it. The supply end could be dragging through a pool of burning gasoline while the takeup end could be disappearing down the gullet of a hungry cassowary or dragged to the bottom of a murky stagnant pond by a nesting platypus. Until the degraded bits actually tried to go past the playback heads, the playback machine would remain blissfully unaware that anything unusual was going on.

In this respect such formats (DV Mini-DV etc) have the same advantage as film – each frame or field is a complete and separate entity. If part of the tape gets damaged, the undamaged parts can still be recovered.

Now alas, we come to more recent storage formats based on hard disks, optical disks, flash memory etc.

The biggest problem with this sort of storage is that the same media used to store the image data is also used to store the information required to retrieve it. A splendid example is the recordable DVD. The Table of Contents only occupies a minute part of its storage capacity but without it, the disc is completely useless. If you have ever experienced the delight of spending several hours transferring old Video-8 home movies onto a DVD, only to have the disc “coaster-ed” because the DVD recorder glitched during the finalizing process, you will realize what a stupidly fragile system it really is. I would NEVER recommend anybody buy a DVD-based video camera, because I’ve seen so many sad examples of people’s once-in-a-lifetime home movies that never made it through the finalizing process. For my money, Mini-DV is still a far better format for home video, but hardly anybody makes them anymore.

Flash memory is not really much better.

Flash memory depends on the storing of tiny electrostatic charges inside microscopically thin layers of quartz. With the original EPROM technology on which Flash is based, post fabrication packaging was a relatively simple affair: if there were 524,288 storage cells on the chip (ie a 64K Byte EPROM), every single one had to work, otherwise the chip was rejected.

To save wastage, some manufacturers would “cheat” by offering a “32K” version, which basically was a 64K chip where one of the two banks of 32K had one or more faulty cells and was not hooked up.

Flash memory expands on that idea.

Simply hard-wiring the chip to fence off the faulty segments is not a practical proposition once you start moving into the multi-megabyte range. The manufacturing error rate becomes so high that is simply not possible to make a chip with enough good segments. Instead, modern Flash memory is deliberately made with a considerable amount of spare cell capacity, and a large amount of on-board data wrangling “intelligence”.

In the initial manufacturing process, each flash chip does an “audit” of all the memory cells it contains, and makes a note of all the ones that are non-functional. This information is stored in a separately reserved area of flash memory, which also contains the operating system for the flash chip’s on-board computer.

With an old fashioned memory chip, data was written to or read from whatever location the device accessing it said it was to be read from or written to. With Flash memory, the memory chip’s on-board microprocessor makes all the decisions about the locations in which data will be stored, all the time consulting its list which ones are not useable. If the microprocessor makes a mistake, for any of a variety of reasons, and corrupts the directory data, there is then simply so practical means of recovering the stored files. When you buy a particular flash memory module, you have no way of knowing how many faulty segments it has. The more of these there are, the more likley it is that the “housekeeping” software is going to trip over its own feet when asked to store a particularly onerous burst of data.

This is analagous to the system used with Hard Disk drives, and in fact the Compact Flash cards used by the RED and many professional stills cameras, have exactly the same connector pin layout as the IDE interface connection used by IDE drives. With the right adaptor you could in theory connect one of those directly to an IDE connector, and it would work as a disk drive.

A Hard Disc has a thing called the File Allocation Table (FAT) which is a directory of what files have been written to the drive, and where all the pieces of each file are kept. If the FAT gets corrupted, for most practical purposes the disc becomes useless. In the old MS-DOS days, “brute force” recovery of files without a FAT was possible, but what you would be left with was basically a jigsaw puzzle of 512-byte segments, of possibly hundreds of different files. In those days the number of bytes in word processing files was roughly similar to their character count, so it was often a practical proposition to wade through the mess and retrieve all the segments of your document and paste them back together.

With the complexity and size of modern word processing files and particularly with images, this is simply not possible, regardless of what you may have seen on CSI and similar shows.

These are the real problems besetting both Hard Disc storage and Flash Memory. Too much of the process is dependent on fiddly little microprocessors running from a not-terribly-well-regulated power supply rails, in a sometimes quite hostile environment.

A digital tape is simply a strip of plastic with a magnetic coating. Film is a strip of plastic with a gelatin coating. Certainly it is easy to damage both media, but it is well within the capabilities of even the most inept operator to NOT damage it.

Flash memory (and to a lesser extent Hard Disks) has a mind of its own. No matter how good or experienced or careful you are, if it decides not to work, it will simply not work! All the talk of multiple backups and workarounds and so on, is hardly reassuring. Most people would be of the opinion that it should not happen in the first place.

This, more than anything else is the reason mainstream acceptance of the RED is taking so long.

Tape just works. So does film. Flash and Hard Disk ... usually work. :rolleyes:

Christopher Adams · May 25, 2009

If it hasn't been mentioned before this I hear does a good job and will let you automate

the process to copy to multiple locations keeping data checks going.

http://www.r3ddata.com/

CJ Adams

Simplexity Digital Post

http://simplexitydigital.com

Satsuki Murashige · May 26, 2009

There seems to considerable confusion of terminology here.

Thanks Keith, that was a fun read. Work is a bit slow at the moment, eh? ;)

Oh great and wise moderator, please sticky this thread for posterity.

Keith Walters · May 26, 2009

Thanks Keith, that was a fun read. Work is a bit slow at the moment, eh? ;)

Oh great and wise moderator, please sticky this thread for posterity.

No, work was a bit slow about 6 weeks ago (note the date :lol: )

Keith Walters · May 26, 2009

Sorry, make that 10 weeks ago :rolleyes:

Satsuki Murashige · May 27, 2009

Sorry, make that 10 weeks ago :rolleyes:

Oh, hehe. Well I'm glad you're busy now. Just had my two busiest weeks of the year so far myself, hope it keeps up.

Aaron Moorhead · May 27, 2009

As I start to nail down DIT/wrangler gigs, this stuff is exceptionally useful. I particularly like detailed workflow explanations. Great stuff, keep it coming!

Conrad Hunziker III · May 27, 2009

Sorry for the late reply, Ive just noticed this thread.

I wrote R3D Data Manager for this very situation. It copies your files to multiple destinations with a single click. It will then verify each and evry copy is a valid copy by using checksums. And it will store those checksums in a way so that at any point in the future you will know that the copied data exactly matches your camera original.

R3D Data Manager also does batch quicktime dwonconverts and a variety of other tasks. I wrote R3D Data Manager to make it easy to use on set, but it can be used in every stage of production.

Id encourage you to visit the website for more info : http://www.r3ddata.com

Joe Giambrone · May 30, 2009

Some great advice here. Don't try and be the data wrangler and the AC simultaneously. Something will suffer, with the potential for catastrophic loss. Do look into the software that makes multiple backups. Good luck.

Tony Brown · June 13, 2009

Use a DIT. If the production get away with it this time they wont next. Chances are they dont have a clue what they are asking you to do.

Its in all our interests (ESPECIALLY Productions) that this new workflow is handled correctly and continues to be in a professional environment

Just my opinion......

Phil Rhodes · June 13, 2009

The problem with all of this, in my very direct experience, is that you can be slipshod and flaky and you will continue to get away with it for a very long time.

The trick from a DIT's perspective, beyond trying to ensure it doesn't happen (which you probably won't be allowed to do) is to insulate yourself as much as possible from the inevitable ordure/ventilator collision when it does go wrong.

P

Keith Walters · June 14, 2009

The problem with all of this, in my very direct experience, is that you can be slipshod and flaky and you will continue to get away with it for a very long time.

The trick from a DIT's perspective, beyond trying to ensure it doesn't happen (which you probably won't be allowed to do) is to insulate yourself as much as possible from the inevitable ordure/ventilator collision when it does go wrong.

P

Just so, and there is a very fine line to be followed here.

If, by means of persistent persuasion, nagging, threats of physical violence, arranged assasanations etc etc you succeed in achieving a pristine result, the trouble then is that nobody ever realizes that anything could have gone wrong, (well nobody important anyway) and you are made to look like a pedantic/neurotic prat.

In my 10 years at a famous film equipment rental company, I eventually learned the art of anticipating potential problems and devising effective solutions (if possible), but then saying nowt until the know-it-alls had totally f*cked up.

Believe me, it takes a VERY long time for production companies to realize that with certain people on set, in the main, jobs just tend to go smoothly, at least in their department.

Edited June 14, 2009 by Keith Walters

Noel Villers · July 13, 2009

I'd be fractionally, though probably paranoically, concerned that there isn't much bit for bit verification going on; the LTO 4s are five copy operations (CF -> RAID -> Jump -> SAN -> LTO) away from camera original, and there is no way of making sure that nothing's been even fractionally broken in that time. This could be fixed by putting MD5 checksums on things.
P

I highly recommend the application R3D Manager --> http://www.r3ddata.com <-- especially since the advent of v5.0 that fixes what was IMO its only weakness --> checksum creation on the actual recorded media. It has a set of highly automized functions that can securely manage the entire data management workflow from on-set backup through post. It generates a checksum for each r3d file that can then be checked for consistency every time an additional copy is made. Additionally it creates a log of exactly when and where every file was copied. Version 5.0 also has a set of new features including the ability to format medias ready for the camera, interface the RedUndead data recovery line inteface program as well generate rendered files although I prefer the absolutely wonderful freeware program (MAC only) Clipfinder --> http://www.daun.ch/software <-- for this purpose.

regards,

nv

Von Thomas · July 14, 2009

In the past tens years I've been involved in digital capture, I've heard my position as digital tech on a still shoot could be done by a photo assistant, or the clients office assistant. True the duplication process is not rocket science, but you have to know there is a rocket, and that rocket might malfunction, and if/when it does, who's going to fix it?

A good DIT, RED Tech will be able to troubleshoot to the extent, no one every knows there was a problem in the first place.

So, don't downplay that position. That position is handling (protecting) everything you are on set for days or weeks to capture. If they screw up because you though a PA could do it, how much money will they have cost the production? Nobody likes to pay insurance either.

People get lucky on shoots everyday not having a DIT, just as there as uninsured motorist driving every day, one day, luck will run out.

Von