Jump to content

16mm 4K scan edit workflow for digital and film out 23.98 or 24p?


Brian Rose

Recommended Posts

Hey all,

 

I'm getting closer to beginning shooting and editing of my new documentary! I'm shooting Super 16 BW and am getting a 4K scan. I'm having a little trouble nailing down my workflow, having done some earlier tests with transcodes and relinking to DPX files, and the results were rather wonky, with cut points changing when I relink, etc, etc. I suspect I made a mistake with the frame rate, and wanted to pose this general question to you all about workflow.

 

My goal is to have a 4K master, for theaters and for future resilience as UHD blu-ray and UHD streaming gains further traction. I'd also like, if I successfully raise the money, to produce at least one 35mm print from the 4K DI so I can have that option as well.

 

My plan is to get 4K DPX files, and for editing cut using 4K prores proxy transcodes. What I'm uncertain about is, just what framerate do I edit with and transcode to, given the stated goals for distribution? Do I use 24p or 23.976?

 

Your help and expertise will be deeply appreciated!

 

BR

Link to comment
Share on other sites

  • Premium Member

Why not get Pro Res XQ? Its a full 12 bit RGB format and most transfer houses can deliver it. The great thing about XQ is that Avid, Premiere and Final Cut X use it natively. So you don't need to do any conversion, drag, drop and edit.

 

You'd edit at 24fps for theatrical and going back to print.

 

Are you shooting B&W negative or reversal?

 

Sounds like a cool project, I'd like to know more about it.

Link to comment
Share on other sites

Why not get Pro Res XQ? Its a full 12 bit RGB format and most transfer houses can deliver it. The great thing about XQ is that Avid, Premiere and Final Cut X use it natively. So you don't need to do any conversion, drag, drop and edit.

 

You'd edit at 24fps for theatrical and going back to print.

 

Are you shooting B&W negative or reversal?

 

Sounds like a cool project, I'd like to know more about it.

Thanks! I may give that Prores a try. I like to get DPX from the lab because it's nice to have the film is the "rawest, most uncompressed" state possible, and because the lab charges extra to do the conversion. But on your advice I may try converting the DPX to prores XQ.

 

I'm shooting BW negative - Orwo 400 since I'll be shooting in some tough locations, with low lighting, plus some night shooting. Had a lot of great results pushing up to 2 stops with that stock.

 

The story itself is an incredible one. It's the culmination of three years of research. It's about a student who, while on a class trip, vanished without a trace. No clues, no indication if even a crime took place. If I can pull it off, it will be like In Cold Blood...minus the solution.

Link to comment
Share on other sites

  • Premium Member

Very cool!

 

DPX is 10 bit BTW and it's a very old codec. Pro Res XQ is 12 bit and pretty much lossless.

 

With B&W, the bit depth becomes more important and honestly, if you do 12 bit, I'd go for it.

 

Come monday, you'll get your ear talked off by the "scanner" guys on here. :D

Link to comment
Share on other sites

There is nothing to prevent you from working in DPX 16 bit. If you need to change one single shot in the middle of a 2 hour program, DPX suddenly becomes much more easy to work with than any Quicktme codec. Just rerender the shot and you are done. That said, you need a fairly heavy RAID to playback DPX in real time without proxies.

Link to comment
Share on other sites

Hey all,

 

I'm getting closer to beginning shooting and editing of my new documentary! I'm shooting Super 16 BW and am getting a 4K scan. I'm having a little trouble nailing down my workflow, having done some earlier tests with transcodes and relinking to DPX files, and the results were rather wonky, with cut points changing when I relink, etc, etc. I suspect I made a mistake with the frame rate, and wanted to pose this general question to you all about workflow.

 

My goal is to have a 4K master, for theaters and for future resilience as UHD blu-ray and UHD streaming gains further traction. I'd also like, if I successfully raise the money, to produce at least one 35mm print from the 4K DI so I can have that option as well.

 

My plan is to get 4K DPX files, and for editing cut using 4K prores proxy transcodes. What I'm uncertain about is, just what framerate do I edit with and transcode to, given the stated goals for distribution? Do I use 24p or 23.976?

 

Your help and expertise will be deeply appreciated!

 

BR

What frame rate did you shoot in the camera? If 24fps, stay with that. If 23.98, stay with that. If your timeline doesn't match your camera FPS, you will loose sound sync every 501 frames I believe. And your sound recording should also match your camera FPS as well...

 

After you edit and mix the sound masters, you can change the frame rate for playback with a tiny shift in running time. DCP for theaters will be 24fps. Home video, in the USA, will be 23.98. I usually shoot 23.98, edit, 23.98, sound mix at 23.98 and convert to 24fps only for the DCP for theaters. Our post production sound mixer strongly prefers 23.98 for his workflow.

Link to comment
Share on other sites

On my scanner I can do 8, 10, 12 and 16 bit per channel DPX, but most applications I have won't recognize 12bit version. It is indeed open format, SMPTE approved, which means you are not vendor-locked into one particular workflow. Try combining Windows or Linux with a full Quicktime workflow.

Link to comment
Share on other sites

Very cool!

 

DPX is 10 bit BTW and it's a very old codec. Pro Res XQ is 12 bit and pretty much lossless.

 

Putting on my pedantic hat here: DPX isn't a "codec" in the sense ProRes is. [Edit: Gah! Rob beat me to it!]

 

That said, If the film is color negative, there is basically no difference between 10bit DPX Log files and 16bit linear. That's just the nature of log-encoded data. You don't really gain anything from the extra bits unless you're working with linear-encoded files. ProRes is generally linear (it can be log though), so if you're comparing 10bit linear DPX to 12 bit linear ProRes, then yes, you're getting a bit more with a 12 bit file, but not that much.

 

To the OP: we scan 16mm directly to ProRes or DPX, no charge for conversion because it's output by the scanner directly to either (or both) formats. There's definitely a lot more convenience with ProRes, but if I was planning on a filmout, I'd stick with DPX to avoid the compression applied by the ProRes codec. It's minimal (really minimal), but it's there. Hard drives are cheap, so there's really no reason not to scan to an uncompressed format like DPX if you want that.

 

It's easy enough to make a set of proxy files in ProRes at a lower res, to go along with a DPX sequence for editing, and then simply relink with the DPX sequence later in the editing or grading tool of your choice. Best of both worlds..

Edited by Perry Paolantonio
Link to comment
Share on other sites

DPX is not a Codec, it is an uncompressed 32bit or 50bit RGB file format which is open.

I feel like working with DPX, TIFF or any codec free sequence file to be a step forward in my work flow over any codec. Maybe because i'm a Windows only guy, tired of Apple based codecs dominating the scene, tired of the codec boondoggle all together. My one experience with ProRes was too noisy, and I find codec free files to be a much smoother work flow. DPX and TIFF drop into resolve with no fuss.

Link to comment
Share on other sites

I feel like working with DPX, TIFF or any codec free sequence file to be a step forward in my work flow over any codec. Maybe because i'm a Windows only guy, tired of Apple based codecs dominating the scene, tired of the codec boondoggle all together. My one experience with ProRes was too noisy, and I find codec free files to be a much smoother work flow. DPX and TIFF drop into resolve with no fuss.

 

Can you define "noise?" -- there's no noise in the analog sense (random high frequency information) in a ProRes file. I seriously doubt that ProRes compression was visible unless you were using a lower quality (more compressed) flavor like LT or Proxy. ProRes HQ or higher is indistinguishable from uncompressed, including DPX.

Link to comment
Share on other sites

  • Premium Member

Yep, there was a recent test done where the filmmakers went out with an Alexa and shot both RAW and Pro Res XQ. The difference was indistinguishable. They pushed it, pulled it, did all sorts of funky things to the image in order to break it up and nothing happened, both were rock solid.

 

Now I heard the Quicktime Pro Res decoder on PC's wasn't quite to the level on the macs. I'm not a windows guy and I do post production for a living. The WHOLE INDUSTRY is Pro Res and now that more and more cameras are shooting in that codec, it's become more popular as a complete workflow codec then it ever was before.

Link to comment
Share on other sites

  • Site Sponsor

Yeah ProRes444 is very good and XQ is even better, that said you cannot decode 444XQ on a PC as far as I know. IMO it will be a long time before the film and tv industry moves away from ProRes it's just too bad that Apple is making junk computers these days.

Link to comment
Share on other sites

Now I heard the Quicktime Pro Res decoder on PC's wasn't quite to the level on the macs.

 

It's the same as in Quicktime 7 on the mac. Quicktime X is 64bit so the playback performance of higher resolution files is better there, but that's not available on Windows. QT7 is 32bit, so it's got some limitations. It's the same engine as in Quicktime 7 on the mac, less the ability to make ProRes files.

Link to comment
Share on other sites

Yeah ProRes444 is very good and XQ is even better, that said you cannot decode 444XQ on a PC as far as I know. IMO it will be a long time before the film and tv industry moves away from ProRes it's just too bad that Apple is making junk computers these days.

 

Scratch Play can do it and that's free (and 64bit). Apparently, it's just a matter of changing some metadata (the fourcc code) in the file, and then you can do it in Quicktime player as well. But it's a little kludgy and kind of a pain. There doesn't seem to be any technical reason why it won't work, but Apple didn't update Quicktime 7 player to support it without this little hack.

Link to comment
Share on other sites

 

Can you define "noise?" -- there's no noise in the analog sense (random high frequency information) in a ProRes file. I seriously doubt that ProRes compression was visible unless you were using a lower quality (more compressed) flavor like LT or Proxy. ProRes HQ or higher is indistinguishable from uncompressed, including DPX.

Interstitial plasma? on a few bright shots.. but what it was inherent to is hard to pinpoint. it was a shadow scan of some S8 so maybe more due to the colorist than the file? But wasn't really my point. My point is I found ProRes a bit more cumbersome to deal with. I guess I just don't get why an Apple based codec is becoming the current standard? There are already enough headaches on various compatibility issues between Windows, Mac, codecs, hardware, media players, edit software ect... and I feel like image sequence files cut through a lot of workflow headaches for once. At least that's my perspective as a basement novice, between incoming and outgoing files.

Link to comment
Share on other sites

What you're describing as noise definitely isn't something that's inherent to the ProRes format, which frankly is one of the best available codecs. That had to have been in the transfer somewhere.

 

I've been using Quicktime almost since it was released, and for years before ProRes came out, we had to use Uncompressed 10bit files, which were massive for HD. For 2k and 4k, they're just totally impractical. ProRes has been around for at least 10 years, I think, and in its HQ form, it's equivalent to 10bit uncompressed files, at roughly 1/10 the size. It became (past tense) the standard because of this and because playing it on both Mac and Windows has been possible for some time. There are indeed issues with some applications and how they handle things, but those are mostly a thing of the past. We have been using ProRes for years as our primary containerized media format, and it has vastly simplified and sped up work around here, without sacrificing quality. We are about 90% Windows here.

 

I can see what you mean in that removing media layers from the equation makes the signal path simpler. But I have to disagree about the workflow argument. We do a ton of work with DPX. Our ScanStation can scan to this format directly, and our Northlight can only make DPX files. We use DPX in Resolve as our main media format, and we use DPX in our restoration system as well. But it's an utter nightmare to work with, especially with long-form projects.

 

Just to give you an idea:

 

We are currently wrapping up a feature film project. We scanned it on the northlight at 4k, we graded it, we are doing the restoration right now. Deliverables to the client are the original log scans (DPX), the graded files (DPX), the graded & restored files (DPX), as well as about a dozen variations in Quicktime (HD, 4k, 23.98, 24, 25fps, a couple different aspect ratio mattes, etc).

 

Each version of the film in DPX is just under 5TB. To copy one reel of the film to a hard drive (7200RPM SATA 6gbps disk in a dock) takes 2-3 hours. We then remove that drive and move it to the next workstation (say, from resolve to the restoration system). That then gets copied to the RAID in the restoration system. 2-3 more hours. Now we've spent an entire work day moving the files from one machine to another, before we get any work done. To do the entire film is a couple days of just copying files, renders aside.

 

When we're done, one reels of film (as DPX) will fit on one LTO5 tape. I'm pretty sure we'll be able to fit all of the ProRes versions (10-12 files) on one, maybe two LTOs, depending on what they decide to order.

 

DPX is a wildly impractical format. The only way it works smoothly is if you have a massive and incredibly fast centralized storage system that uses something like 10-40GB Ethernet, Infiniband or FibreChannel. That's expensive stuff (though we're working on a cheap 20-40Gbps Infiniband SAN that I hope to get wrapped up in June). It's also clunky, because you really can't do this over standard networking protocols like SMB, which would be the easiest way to go. Instead you have to use iSCSI, which isn't complicated, but it means only one workstation can access the files at any given time, unless you've spent tens of thousands of dollars on an enterprise-level SAN with proper management software.

 

 

 

Link to comment
Share on other sites

Interesting info Perry- I didn't realize ProRes came in smaller packages. The file I got was 130GB for 400ft S8 uncompressed I presume, it was 4 years ago. But yes, large files are an issue for me, even though I have never gone above a TIFF. Im grading and editing 250ft S8 i scanned for someone now, 100GB proxy as a TIFF. Most of the files I deliver are from small volumes of film but the files are usually 60-100GB. But I was curious as how pro facilities handle transferring large volumes... which like you said, involves more expensive gear.

Link to comment
Share on other sites

Interesting info Perry- I didn't realize ProRes came in smaller packages. The file I got was 130GB for 400ft S8 uncompressed I presume, it was 4 years ago. But yes, large files are an issue for me, even though I have never gone above a TIFF. Im grading and editing 250ft S8 i scanned for someone now, 100GB proxy as a TIFF. Most of the files I deliver are from small volumes of film but the files are usually 60-100GB. But I was curious as how pro facilities handle transferring large volumes... which like you said, involves more expensive gear.

 

ProRes has several flavors, each with different levels of compression. They're all compressed to some degree. but then again, so are most (if not all) of the HD and SD tape formats, even the ones most people consider to be "uncompressed."

 

TIFF is a nightmare to work with and I'm not aware of anyone using it regularly for editing or even grading at the professional level. It's always linear, so to get the full color bandwidth you have to work with 16bit TIFFs. There is no 10bit TIFF standard, only 8bit, which would be inadvisable because the vast majority of the color data would be missing. DPX is at least more widely supported by some editing systems, but even then you still have to have massive disk arrays to hold and move that much data. It's not a common format at all for editing, just for grading and restoration, and for VFX work. Few edit systems really handle it well.

 

On the other hand, containerized formats like Quicktime, make it simple to keep audio and video together and in sync, and to work with files that are much more reasonable in size.

Link to comment
Share on other sites

  • Premium Member

It's the same as in Quicktime 7 on the mac. Quicktime X is 64bit so the playback performance of higher resolution files is better there, but that's not available on Windows. QT7 is 32bit, so it's got some limitations. It's the same engine as in Quicktime 7 on the mac, less the ability to make ProRes files.

Ohh interesting, so that makes more sense to me.

Link to comment
Share on other sites

  • Premium Member

I guess I just don't get why an Apple based codec is becoming the current standard?

Nobody else has stepped up to the plate. All the other high-quality methods of storage are either extremely proprietary; RED code, Arri Raw, Cinema DNG, etc... or not high quality enough; Intraframe MPEG 2 and .H265.

 

Pro Res has some huge benefits. The biggest one is not using the GPU for decode. Pro Res is actually multi-threaded, so it uses the CPU for decoding. This allows the load to be spread across multiple cores and keeps the GPU available for real-time effects and of course, the GUI of your editing workstation.

 

Pro Res is also an intraframe codec, based on wavelet technology, more similar to J2k then MPEG. It's also an old codec, it's been around since the late 90's and it's only gotten better over the years. It's not very efficient, but efficiency isn't necessarily important in filmmaking. The most important thing is quality and Pro Res has offerings from 8 bit proxy all the way up to 12 bit 444 RGB. No other codec has those offerings.

 

There are already enough headaches on various compatibility issues between Windows, Mac, codecs, hardware, media players, edit software ect... and I feel like image sequence files cut through a lot of workflow headaches for once. At least that's my perspective as a basement novice, between incoming and outgoing files.

Image sequences can't be played back on your clients laptop. They can't be edited on standard editing software without transcode either. Plus, image sequences which are in folders cause another problem, which relates to how many individual files your drive can deal with. Once you get over a certain threshold, drives get SLOWER when they're constantly having to access small individual files. So if everyone worked in image sequences which were in folders for instance, it would be a problem for most people who don't have fast storage solutions which may not suffer from speed degrade. I've done extensive research and testing for the companies I do post production support for and it's clear to them and to me, single files are better then multiple in a sequence.

 

The solution however is to compact those frames into a single file. That's kinda what Pro Res does since it's an intraframe codec. The down side is if one or more of the frames are damaged, the whole file is corrupt. This COULD be fixed however, there are open wrapper versions of Pro Res available, I've used them in the past and they work great for fixing corrupt files. However, the normal one we all use is closed wrapper, which is unfortunate.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...