Jump to content

Questions about DPX Workflow and What to Expect from a DPX Film Transfer


David Fitch

Recommended Posts

As I prepare for an upcoming S16 transfer, I have the option of getting an HD telecine or a UHD film scan. In the end, I'd want a ProRes file for editing in Premiere Pro, but if I go the scan route, I'd also have the DPX sequence along with a ProRes file rendered from the individual DPX frames. As I'm familiar with the concept of individual DPX frames from film scans, I'm not well versed when it comes to working with them, so hopefully someone can help with some of my questions.

 

I don't have an exceptionally high powered editing system (I'm editing on a MacBook Pro with Premiere Pro CS6), so is there much of an advantage to me in having the individual DPX sequence files, and is there much I can do with the DPX sequence in Premiere Pro? My understanding is that the individual DPX frames are around 10 MB each, so I'd imagine that you would need a fairly powerful system to be able to edit directly or perform color grading with the individual DPX frames. I think using the ProRes file is much more in line with my system's capabilities, so I'm just curious what, if anything I could do in Premiere Pro with the DPX files. Even if I can't really use the DPX sequence for real-time editing, is it still possible to perform color correction on a DPX sequence in Premiere Pro, or is DPX color grading typically performed elsewhere?

 

Also, since DPX files are essentially a "flat" scan of the original camera negative, should I expect a DPX sequence to be pretty flat in terms of color saturation and low in contrast, even if properly exposed, with the expectation that some type of color correction will be performed on the resulting image sequence? And, if that's the case, would a one-light ProRes file rendered from the individual frames essentially have the same flat-ish color and contrast as the original DPX sequence?

 

I have a lot to learn about the DPX workflow, so any help would be appreciated. Thanks!

Link to comment
Share on other sites

  • Premium Member

First of all, I recommend getting the UHD transfer as you call it. Will it really be UHD or 2k for S16 though? Usually S16 is scanned at 2k.

 

Your editing system does not have to be "high-powered" to handle DPX. You do need a fast data connection to play back at 24fps though. Don't get hung up on this, 24fps playback of your scan is likely unnecessary.

 

When you get the scans back, they will be flat. Low contrast and low saturation. Premiere can interpret DPX image sequences as a singular movie. You'll want to bring the DPX into a timeline to encode a ProRes editing proxy. Given your computer setup, I would recommend making 2 prores files - a 2k "online" ProRes4444 file that you'll color correct later, and a 1080 or 720 422 editing proxy. Don't touch the color of the 4444 file, but add some contrast and saturation to the low quality editing proxy so you can tell what you're looking at.

 

When you're done editing, you'll replace the editing files in your timeline and color correct the 2k version. I'd recommend color correcting in software like DaVinci Resolve, which is free, albeit with a bit of a learning curve.

 

But here's why you should bother going through the trouble of the DPX. The scanner they use for the DPX sequence is almost certainly a different scanner than their telecine setup. And telecine scanners can't interpret the entire tonal range captured on your negative. DPX will preserve more information (even when encoded into prores) and give you more creative freedom with color after you edit instead of making these choices before. And, down the road, if you decide to revisit this project and have access to a more robust setup, you can return to your DPX master and have all the information possible to play with.

Link to comment
Share on other sites

Thanks for the reply, Dan. You've answered many of my questions and provided some excellent information for a DPX newbie like me :)

 

I do have a couple of follow-up questions. If I receive a DPX sequence, I'm assuming it will be a collection of many individual .dpx files...one for each frame of film. How do I go about importing this into Premiere in preparation for generating a 2K ProRes4444 file? I'm assuming (incorrectly, perhaps) that I'll have to drag the frames onto the timeline, but perhaps there's another method of doing this.

 

I didn't realize that DaVinci Resolve is available as a free download, so I'll definitely check it out. It looks like, at least for Mac, that only the "Lite" version is free, however I'd imagine that it should be more than sufficient for someone like me just getting started with color grading.

 

You did mention color correcting the ProRes4444 file in Resolve. Would there be enough data / tonal range from the original DPX sequence preserved in the albeit high resolution, but more compressed ProRes4444 file that would make the ProRes file suitable for color correction?

 

Also, this is more of a general question about DPX files and something one probably needs to grasp in order to understand the significance of a DPX sequence or a negative scan in general, but what is inherent or integrated into a DPX file that makes it so suitable for color correction? I've seen a few stills of flat DPX scans, and I was surprised at the lack of contrast and color saturation, almost as if they had been over-exposed. That said, what is it about a DPX file that allows you to pull good contrast, saturation, and an overall good image from an image that, on the surface, looks so lousy?

Link to comment
Share on other sites

  • Premium Member

There is nothing intrinsically different about DPX other than that it is uncompressed (and it supports various film-industry metadata, but that may not be that relevant to you). In practice they tend to be used for images stored with higher precision than something like a PNG file you'd find on the internet - storing an image a 10 bits per pixel, for instance, is common, and means that each of the RGB channels is stored with 1024 levels of brightness, rather than the 256 of an 8-bit scan.

 

This makes it practical to store much lower-contrast images. You'll put contrast back into them in the grade, but because there's finer encoding of luminance, you don't get ugly artifacts. Of course this can become a circular issue, since you only had the low-con scan done because you had 10 bits available in the first place so... anyway. Ultimately, you get more information to do more things with in the grade.

 

The very low-con, odd-looking scans you describe may well have been logarithmically encoded. This is (usually) a luminance encoding that ensures each f-stop of exposure information occupies something like the same number of digital values. This is not how most displays expect the information to be supplied so the image looks extremely flat and washed-out. Again, this is mainly useful with 10-bit encoding, otherwise the large contrast corrections required to make it look normal on a normal display become problematic. Resolve supports various log encodings; liaise with the scanner people to ensure they're giving you something you can handle properly (in the worst case scenario, you can often fix it by hand, but better not to).

 

The difference between high-res prores and DPX is likely to be vanishingly insignificant, especially if you're shooting 16mm to begin with. Prores is compressed, very much like a JPEG. DPX isn't, very much like a BMP or PNG. But the prores compression is fairly light.

 

If you want to get a bit more techy, you can convert the DPX frames to prores quicktimes en masse using the free tool ffmpeg, and a commandline such as:

ffmpeg -r 24000/100 -i inputfile.0%05d.dpx -qscale 9 -probesize 5000000 -vcodec prores_ks -profile 3 -acodec pcm_s24le -vendor ap10 prores.mov

This will convert DPX files with names such as inputfile.00001.dpx (and subsequent files) to prores.mov. Alter the filenames to suit. Alter the %05d to howevermany digits you have in your DPX file names. Adjust -r to a suitable framerate. Adjust -qscale up to maybe 13 if you need to save a bit of space. This will give you Prores HQ. I think you can add -pix_fmt yuv4444p10le to the command line if you want 4444, but I haven't needed it personally.

 

P

  • Like 1
Link to comment
Share on other sites

  • Premium Member

^ What Phil said haha

 

ffmpeg is a great tool, though if you're intimidated by that command line stuff, Premiere will display an image sequence as a single file in the import dialogue. Or sometimes you just select the first image and check a box that says "image sequence" and it will bring the whole sequence and display it as one entry in your media browser. I have CS5 so I imagine CS6 is even easier to deal with.

 

DaVinci Resolve Lite has only 2 limitations: No noise reduction and no 3D tools. Every other feature is available, including UHD output now. Just don't expect any customer support if you have trouble with it - they don't care if you're struggling with something you got for free.

Link to comment
Share on other sites

I would recommend 2K instead of UHD.

 

Last time I scanned S16 it was to FullHD (2011), which was more than enough resolution given the lenses used maxed out at around 1200-1600 pixels in sharpness (superspeeds used around T1.3-5.6). It was shot on Vision 3, T200 and D250 so with better lenses UHD might have been worth it, but then it cost like 3-4 times more to do 4K, and we would struggle in post even with 2K which was twice the cost from FullHD and only 10% more resolution.

 

I took the DPXs into after effects and made ProRes files for editing, in FullHD. But you can just as well use premiere nowdays.

 

Are you scanning and storing them in Log or Linear? I did Log and then used the cineon-effect in After Effects (premiere has it too) and once finished editing just re-linked and graded in AE.

 

You are looking at around 1TB per hour of footage. When I scanned last time I was moving between MACs and PCs, and USB3 wasn't really out yet, so I had to stick to USB2, which was very slow and the bottleneck for production. Just transferring the scanned footage took a 24 hours (and then it isn't backed up)... With USB3, that should be around 3-4 times faster, since my disks stopped around 20MB/sec, and nowdays they go about 60-80MB/sec on USB3 or Thunderbolt. So expect 1-3 days of just transfer/backups including rendering out proxies.

Edited by Karl Eklund
Link to comment
Share on other sites

  • 5 years later...

I've been reading this thread with great interest. I'm knee-deep in this exact process and have lot of helpful info based on what I've learned along the way. But your initial questions go way back, so this is most likely old news. Let me know if you want to hear of my process scanning 16mm at 4K, dpx files, using Resolve workflow.

  • Like 1
Link to comment
Share on other sites

  • Site Sponsor

What scanner are you planning on getting the film scanned on? Any professional "real" scanner can make DPX frames in proper 10bit or 16bit depth in Cineon-Log.

Additionally any scan house should be able to make you ProRes4444 either directly from the scanner or as a no cost addition to the scan.

True RGB scanners like the Scannity, Spirit 2K 4K Arriscan will make DPX frames and then the lab or post house should convert those to ProRes444 as part of the service.

Newer scanners like Scan Station 5K or 6.5K, Xena 4K 6.5K  Blackmagic-Cintel (scans S16mm at sub 2K though) can make ProRes directly in the scanner.

For a UHD-4K scan of S16mm you would want that from a really good machine like a Scannity or a newer Scan Station or Xena with the 6.5K sensor.

Link to comment
Share on other sites

On 5/31/2020 at 11:56 AM, John Schmidt said:

I've been reading this thread with great interest. I'm knee-deep in this exact process and have lot of helpful info based on what I've learned along the way. But your initial questions go way back, so this is most likely old news. Let me know if you want to hear of my process scanning 16mm at 4K, dpx files, using Resolve workflow.

I would be interested to know how much of an improvement there would be with a dpx work flow vs ProRes 444. 

I recently had some stuff scanned to 2k at Kodak Atlanta. I received a drive full of DPX files, I then converted those DPX files to 2k ProRes 444 for editing. When I was finished with the edit, I handed off a flattened version of that ProRes file for the final color grade.

For what it's worth, I found the Scan Station straight to ProRes to be a bit sharper and it seemed to handle under exposed stuff better.

Edited by Ben Ericson
Link to comment
Share on other sites

  • 1 month later...
  • Premium Member

The ProRez files will be compressed and have a lut applied, whereas the DPX files, if they are true log files, will have none of that applied and be much more like a RAW file.  Admittedly the compression on ProRez files is good, but any sort of LUT imposed on your image PRIOR to grading can be destructive to your target color space by clipping or limiting the response.

In any event, it's best to appy LUTs and color transforms at the END of your node tree or pipeline to avoid this clipping and possible distortion of the color range.

If you are using DaVinci Resolve, even with a slower computer, you can either use proxies or optimized footage and cut/grade in a lower resolution format.  When you are finished, reconfigure the project to output 4K and let it render. 

That being said, there can be instances of where your graphics card can choke on the very large files IF you use a lot of nodes and noise reduction.  I have found times that 2K files worked acceptably in Resolve, only to have the GPU error-out when trying to render @ 4K.

It's just something you have to test; like running cameras tests before you do a shoot.  You wouldn't shoot a feature without a proven camera, so why attempt post with an untested computer?

 

  • Like 1
Link to comment
Share on other sites

  • Premium Member

I would venture 4444 is compressed just enough to make working with the files much faster, but retains the proper color space without impact on color operations.

Uncompressed DPX files are quite taxing on playback;  4444 less so...

Link to comment
Share on other sites

  • 1 month later...
On 7/22/2020 at 7:02 AM, Frank Wylie said:

I would venture 4444 is compressed just enough to make working with the files much faster, but retains the proper color space without impact on color operations.

Uncompressed DPX files are quite taxing on playback;  4444 less so...

This is true, however it is very easy and cheap to build a system that plays back 2k DPX in real time for finishing.  UHD or 4k DCI is MUCH more demanding, but HD or 2k is no biggie.

  • Upvote 1
Link to comment
Share on other sites

  • Premium Member
On 8/26/2020 at 4:30 AM, Chris Burke said:

This is true, however it is very easy and cheap to build a system that plays back 2k DPX in real time for finishing.  UHD or 4k DCI is MUCH more demanding, but HD or 2k is no biggie.

Yeah - my last post on this thread was in 2014 and at that time it was a bit more expensive to get 2k DPX playback going.  In 2020, building a 2k DPX ready system is way more affordable.

That said, I still find that 12 bit ProRes 4444 footage provides all the latitude I need 99% of the time (for log footage).  And as Mr Mullen pointed out, you can get (or generate yourself) ProRes 4444 files with no LUT applied at all - just a tiny bit of compression.  I think one could argue that if you're scanning 16mm at 4k, you're probably oversampling enough that when you downres ProRes 4k to 2k you'll get a file that's visually identical to something originating as DPX.

And then you'll compress to h264 4:2:0 lol

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...