Working with DPX Files

Scott Pickering · March 25, 2015

I was under the impression a DPX file was a video format with low compression, high quality, etc. But what I didn't know was every single frame is saved on its own and you have to stitch them all together as one big file. First off what is the best program to do this with, while making sure each frame is loaded in the proper order, and having the program set the frames per second at what you want it to be set to? I could see there being tens of thousands of files in a folder and you have to make sure they are all saved in the correct order. Also when loaded into the program, how does one make one master edit on all the frames, to keep consistency the same with every frame?

And finally- with what program can one edit the original 5K frame and crop out the portion of the frame you want shown on screen? Reason I ask is the Super 8 transfer Im having done shows outside the Super 8 frame and I need to crop that portion out. I want to master to 4K, so this is why I had 5K scans done.

Phil Rhodes · March 25, 2015

You could do this in almost any nonlinear edit software. Most of them support file-per-frame formats and will give you a check box (as in Premiere) if you select the first frame of the sequence, thereafter treating the entire sequence as one clip. You may need to set a frame rate manually, although that is sometimes interpretable from the DPX files themselves. Usually the ordering is controlled by filename.

If the cropping setup you need is fairly consistent - and one would hope it is - I think you could probably do it all in one step in something like the free software tool ffmpeg, which could take all your DPX files, crop them appropriately, and output ProRes in a single (fairly longwinded) operation. Give or take a bit of scripting, you could end up with a fairly automated approach that might be able to maintain appropriate metadata and so on.

5K DPX does seem a bit excessive for super8. Actually, it seems a lot excessive.

Scott Pickering · March 25, 2015

I want to master to 4K, so this is why I chose a 5K scan. Even if the image quality isn't there, it won't have to interpolate if I do a 5K scan to master to 4K.

I checked Premiere and it doesn't allow custom frame rates. I need 18 fps and it only does 15 or 24. Nothing in between.

Perry Paolantonio · March 25, 2015

Hi Scott,

I haven't used Premiere in years, but in After Effects, when you bring in an image sequence it shows up in your bin as if it's a single file. If you right click on the file there's an option labeled "Interpret Footage." there you can set the frame rate. You might see if there's something similar in Premiere - Adobe is usually pretty good about re-using features from application to application. If you can do this, then that's where you'd set your frame rate for the incoming DPX sequence, which will appear in the user interface as if it was any other movie format.

If you bring that into a 24fps timeline, you'd have to either repeat frames (like a step print process would have done back in the day), or interpolate up to 24. I'd avoid interpolation, but that's just my preference.

On the crop - your scans were done with overscan. They can be scanned directly at 4k with a slight crop just inside the frame lines, if that's what you want. Saves you a step and gets you smaller files to work with.

Phil - you'd be surprised how much is actually on the film. 5k is not a format we scan to very often, except for archives that want to capture the full frame plus perfs and film edges, but there's subtle detail there that's lost in a 2k or HD scan.

-perry

Scott Pickering · March 25, 2015

Hi Perry,

Yes cropping the framing to just the edge of the frame would help me a lot. I have no use for the overscan framing. Also as we discussed, I might get you to save this file as a ProRes4444 file instead. Would save me a lot of grief in editing.

Scott Pickering · March 25, 2015

Perry,

Send me the DPX files as you have already done. I figured out After Effects to do the DPX conversion, and then Premiere to work with the files from there. It all works and it makes sense to me now. So send me the invoice for the work you did.

One last question. When working in a program like Premiere, is there a limit to how large a DPX files you can have loaded into memory at one time? Since my films that I will be sending you are a minimum of 13 minutes in length, would that many DPX files overload the memory and crash the computer? One of my other films is over 45 mins in length, so I'd wonder if that too would crash the computer with that many DPX files loaded? My computer only has 32GB of memory.

Edited March 25, 2015 by Scott Pickering

Phil Rhodes · March 25, 2015

It won't load the whole sequence at once.

P

Perry Paolantonio · March 25, 2015

Memory isn't as much the problem as disk space is. Your 50' reel is just under 450GB as 5k, 16bit DPX. ~115MB/frame. So the main issue will be disk space, not RAM. As Phil said, it's not loading the whole movie into RAM, just a few frames at a time usually.

5k DPX won't play back smoothly off external portable drives, you'd need a RAID capable of over 3GB/second in order to play it in real time. The usual workflow for files this large is to make smaller proxy files (you could do this from After Effects, at something like 2k, where you'd need speed more on the order of 500MB/second for DPX, or just a fraction of that for something like ProRes) and edit/grade those proxies. Then when you're finished, relink the media in your timeline up to the 5k version, set up your final crop, and render out the final version at 4k.

-perry

Kenny N Suleimanagich · March 25, 2015

DaVinci Resolve is also an excellent program for handling a DPX workflow. Very easy to create proxies and export to codecs as needed.

Bruce Greene · March 25, 2015

If you have adobe premiere CC, then you also have access to speedgrade. Open your .dpx sequences there and crop to 4k (to avoid scaling) and render to prores if you'd like for editing.

You can even scale to HD prores for editing, and this might be alll you need for a master possibly

Robert Houllahan · March 25, 2015

Loading large frame size DPX files has more to do with the VRAM on the GPU than it does CPU or system RAM, the DPX frames are totally Uncompressed, and especially if they are 16bit linear (not sure why 10-bit log is not being used) typically a GTX-Titan with 6G of VRAM or the new Titan-X with 12G would be the card for 4K + work.

I would load them into Resolve or Speedgrade and make 4K proRes444 files as others have said.

Scott Pickering · March 25, 2015

I have about 1050 feet of Super 8 I intend on getting transferred to 16 bit DPX at 5K rez. Im planning on buying the Seagate 10 TB hard drive when it comes available this year, to store those 3 films onto. I will keep the original drives I send to Perry for archiving, in case the 10 TB drive gives out. I also plan on getting a second 10 TB drive with a copy of those films to put into a safety deposit box. So I should have 2 backups (and one master) of that 1050 feet of film. Would that main drive be all I need to work with the DPX files? I will also have an 8TB drive as my main data files drive for storage for all my other files. Currently they are sitting on 2 3TB drives. I can master that DPX film files onto the 8TB drive as a Pro Res 4444 file.

How does one make a 4K/ UHD Pro Res 4444 file on Premiere? Or should I use Resolve for that instead? I still haven't been able to get Resolve to do any importing without using the exported Premiere file to load into it. Can you start a project in Resolve without having an exported Premiere file loaded into it? How do you load DPX files into Resolve?

Phil Rhodes · March 25, 2015

the DPX frames are totally Uncompressed

To be completely fair, so's everything else by the time it hits the GPU.

Perry Paolantonio · March 26, 2015

How does one make a 4K/ UHD Pro Res 4444 file on Premiere? Or should I use Resolve for that instead? I still haven't been able to get Resolve to do any importing without using the exported Premiere file to load into it. Can you start a project in Resolve without having an exported Premiere file loaded into it? How do you load DPX files into Resolve?

Since you're on Windows, making a ProRes file is a bit more complicated (reading ProRes isn't an issue, but making them is less straightforward than on a mac). Resolve, for instance, can't export a ProRes file because Blackmagic doesn't offer that as an option. If we're not scanning directly to ProRes (our scanner can do that, even though it's Windows based), we use ffmpeg to do conversions - not user friendly but it works. There are some limitations, however. Premiere might export it out, though - I'd be surprised if Adobe doesn't include this (you'd need to have Quicktime installed first, of course).

Resolve (at least version 11) can import your files on Windows. We have two systems: the full version and a light version, and both will take the 5k file (though the Lite version will limit your output to UHD - you need the full version to round trip 5k->5k or to output true 4k. In resolve, just point it to the folder of DPX files, and it appears as if it was a single file.

Just remember - these files *won't* play back with speeds near real time at that resolution, as DPX, off of your external drives. On the RAID in our Resolve system, 16bit 5k files play at about 5-6fps. This is just playback with no color correction or other effects - so the low speed is almost entirely due to disk speed. That's on a RAID that can easily move data at 1.5GB/second. If you're playing back off of an external drive I'd expect slower speeds.

5k ProRes 4444 might play back ok if you have a fast PC - the disk speed won't be as much of an issue, but ProRes is CPU-bound and 5k 4444 files requires a fairly hefty CPU to decode in real time.

-perry

Robert Houllahan · March 26, 2015

I only mention uncompressed Phil because of the huge data rate for file I/O.

16bit is exponentially larger and requires higher performance disk systems also IMO 16bit DPX is unnecessary and has no advantage compared to 10-bit LOG because almost no scanner sensor is more than 12-bits, certainly no color sensor is.

The 5K DPX frame is basically equivalent to a 15K Bayer "Raw" frame being actual RGB instead of a monochrome 5K which then is demosaiced so more GPU VRAM the better for Resolve due to the large frame size, otherwise you will get GPU "out of ram" errors in Resolve.

I would make 5K ProRes 444 and dump the DPX frames as they are huge but YMMV.

Phil Rhodes · March 26, 2015

I only mention uncompressed Phil because of the huge data rate for file I/O.

No argument there.

For what it's worth, which may not be much since I haven't done this for ages and this is off the top of my head, you can go DPX to ProRes, on any platform, with a commandline something like this:

ffmpeg -f image2 -i file.0%05d.dpx -r 25 -s 1920x1080  -c:v prores -profile:v 3 output.mov

This will read file.00001.dpx (and so on) and give you 1080p25 prores.

You can crop it on the way with something like this:

ffmpeg -f image2 -i file.0%05d.dpx -r 25 -vf "crop=3840:2160:0:0" -s 1920x1080  -c:v prores -profile:v 3 output.mov

The crop filter takes four numbers:

- Output height

- Output width

- Horizontal offset (from the left)

- Vertical offset (from the top)

Thus the commandline above would take the quad-HD image area at the top left of the frame, scale it down to 1080p, and give you a prores of the result. It's pretty handy and by experimenting with the numbers you can probably achieve something that will do what you want in a single bound. Alter "profile:v" to 0, 1, or 2 for other types of ProRes and lower bitrates.

Take care about strange colour and luminance issues, regardless of what software you use. Getting this sort of thing to look exactly right is alarmingly hard, sometimes. Additional commandline options may be required if you need specific colour matrix behaviour.

P

Scott Pickering · March 26, 2015

I'd like to stay in Premiere if I could to do all the work needed. Funny thing is I don't see any way to render or encode/ export to 4K in Premiere. All I can see if 1080P for the highest definition. I thought Premiere supported UHD?

Phil Rhodes · March 26, 2015

To be completely fair, you may still encounter weird colour and luminance bugs, sadly.

Normally you can configure output settings manually in Premiere if you need to, but I don't use CC, so I may be behind the times. Some output codecs may have limited resolution capability.

P