Jump to content

Problem with DPX to ProRes 422 workflow on a Filmfabriek HDS+


Guy Burns

Recommended Posts

I have just had some test scans done into ProRes 422, from an 8mm home movie (my reference film), prior to going ahead with having an historic 8mm amateur film scanned into ProRes 4444. The historic film is from the late 1960s, and documents a scientific survey of the now-flooded Lake Pedder in Tasmania. So I want to get the scan right.

Over the years, I have had the test film scanned on a RetroScan, Muller HDS+, and Lasergraphics ScanStation. So I’m familiar with how the scan should look.

The present machine is a Filmfabriek HDS+, scanning at 2.5k. For the test, I asked for a raw scan into ProRes 422, with no corrections of any kind. The raw output from the scanner is DPX. The operator, using a PC, converts to ProRes 422 via Quicktime.

I was very disappointed with the test. Viewing the RGB Parade in Premiere, the blacks seemed to be clipped at 20%. That ugly straight line effect that looks so unnatural. And it couldn’t be graded to approach the look of any of the previous scans.

I brought the problem to the attention of the machine’s owner. He consulted his DPX files and said he could see no problem. He first suggestion was that compressed ProRes 422 was causing the problem. We couldn’t agree on that because from my point of view, 422 is 10-bit and a pretty good codec. After a few emails and phone calls, he then suggested that he revert to his usual workflow and alter the levels of the DPX at his end before converting to ProRes 422, so that the image range was from about 5% to 95%.

He sent me the new file and it looked pretty good.

He now thinks the problem is with his workflow, and so do I. Somewhere in the conversion from DPX to 422. But we’re not sure.

Ques 1
Is the quality of ProRes 422 such that there would be a significant difference between it and DPX at the low end, at around the 20% level?

Ques 2
Or is it more likely that the conversion is being done incorrectly?

Ques 3
Why would changing the levels of the DPX file by a relatively small amount (originally 20 -75%, later 5-95%) make such a big difference to the 422 outcome?

A 6-second snippet of the original 422, and the new level-adjusted 422 can be downloaded here.

Any suggestions most appreciated.

 

Link to comment
Share on other sites

  • Premium Member

There are a thousand ways to get this wrong and end up clipping or cropping brightness ranges, and it's very hard to figure out exactly how without examining the fine details of exactly how this work was done. No, going from DPX to ProRes should not intrinsically cause the problem you describe and yes, I would suspect that the problem is connected with the settings of the software in use.

Can you make a few of the DPX files available? If I were to do this I'd look into using something like ffmpeg which provides a lot of very fine granular control over how things are handled, albeit at the cost of creating something that can best be described as ProRes compatible (though generally fine.)

P

Link to comment
Share on other sites

Thanks for the response, Phil.

This problem has started me thinking that maybe I should be asking for the scan to be delivered in DPX. The scanner outputs DPX as 10-bit log, but I don't think Premiere will like a log file.

I'm going to ask the operator for 10 seconds of the DPX files so that I can test them, and I'll also upload them so that anyone who is interested can play around and make suggestions.

 

Link to comment
Share on other sites

  • Premium Member

If you're getting it scanned, at least get and archive the DPX. You'll never get anything cleaner; that's your master. It's huge, but if this is a matter of historical record, my feeling is that it is the most archival format you will find.

There are a lot of places in which brightness offsets can get in. The problem may be at the scanning facility, or on your workstation, or both. So, get those DPX files, then post back here and we'll figure out how to do this properly. If you can upload them somewhere (or have the scanning facility upload them) then post a link here, I'll have a look at it and we can develop a workflow you can use yourself.

P

Link to comment
Share on other sites

This seems to me an issue of data levels vs video levels.

If the .dpx files are good, then the ProRes 422 is the issue.

ProRes 422 is usually interpreted as video levels, while .dpx is usually full range data levels.

In your grading software, you might try assigning data levels to your ProRes 422 clips.  In resolve, go to "clip attributes" and change the assignment from "auto" to "data" levels and see if that fixes the issue.

Link to comment
Share on other sites

Two comments from the operator. For my scans, "colour grade" means levels adjustment only.

  1. "My workflow is colour grade the captured dpx files in Resolve, export to Uncompressed QuickTime 10 bit RGB files (as the scanner software is on PC I can’t directly export to ProRes hence the uncompressed transcoding), then convert the uncompressed to ProRes via Adobe Media Encoder."
  2. "I am still confident the issue is that the chroma subsampling in ProRes422 (being 4:2:2 vs PR4444's 4:4:4) is noticeably ill-equiped to translate the sparse black and shadow detail of colour reversal film in a flat scan."

I have yet to see the DPX files. They should be uploaded soon.

Link to comment
Share on other sites

  • Premium Member
3 hours ago, Guy Burns said:
  1. "I am still confident the issue is that the chroma subsampling in ProRes422 (being 4:2:2 vs PR4444's 4:4:4) is noticeably ill-equiped to translate the sparse black and shadow detail of colour reversal film in a flat scan."

I have yet to see the DPX files. They should be uploaded soon.

Chroma subsampling is irrelevant to the representation of shadow detail. 

Having looked at this stuff briefly in Resolve the actual scanned image is sitting up with black at probably around code value 192, which is around 20%, and which is vastly off. I notice the ProRes image area outside the film scan is actually black, so I would suspect that this is possibly connected with the way the scanner was set up; this is a much bigger offset than could be explained by a data versus video levels issue at any point in the chain.

It's common practice to make sure the scan gets into all the shadow details and it's common to have to grade film scans down a bit in the shadows to achieve an appropriate black level, but I'd have said this was a bit extreme. It could have been exacerbated by additional mistakes with data versus video levels, but again, the file itself has black in it.

If I were grading from the DPX files I'd just correct it in the grade, to be honest, and not bother too much about it. It'll cost you 20% of your available picture code values but it's hardly the ends of invention especially for super-8 stuff that's noisy as hell anyway. I wouldn't want to grade it from this, compressed version, though. Crunching all that contrast back into it is likely to make the ProRes 422 compression more objectionable than it needs to be.

I suspect the proper solution to this, based on the assumptions I've made which may be wrong and the information I have which isn't complete, is that a reasonable solution is to go back to the DPX, set a more appropriate black level, and re-export. The ideal solution would be to rescan it with a more appropriate black level.

There is an outside chance this could represent several generations of video versus data luminance levels mistakes in a row but it depends exactly how everything's set up.

P

Link to comment
Share on other sites

Daniel   In hindsight, maybe I should have asked for the scan to be output to Tiff. But I thought I read somewhere that Premiere (I'm running CS6) doesn't import Tiff at full resolution, only 8-bit. That's why I chose DPX for the test scan.

Shootout   As for the scanner shoot out, even though I have a dozen or so scans of the same reel from various machines at various settings, the shootout wouldn't be between machines, but between how the operators use their machines. So it would mostly be a test of operators.

The most pleasing scans of my 8mm test film have not come from the Lasergraphics ScanStation, but from the Filmfabriek -- because the operator of the Filmfabriek has taken the most care. The ScanStation (costing five times the price of the Filmfabriek?) gave me overly-sharpened images and blown highlights. That's not the machine, that's the operator.

As with all machines (cameras, scanners, power tools) -- above a certain price level, the quality of the output does not depend on the machine, but the operator. That's why I want to encourage this young bloke in Melbourne, a one-man band with a Filmfabriek, to do the best he can with his new toy. I can ring him and he answers the phone. I email and get a prompt response. If there's a problem, as with this ProRes 422 thing, he's wants to get on top of it: "I have investigated this further and have narrowed the issue down to using the ProRes422 codec." Whether or not that's the real issue, is yet to be determined.

DPX to ProRes   I want to work out where this DPX to ProRes 422 problem lies. I'm new to DPX, but have been reading and thinking about it a lot. I'll make some comments below to test my understanding of DPX. No need for detailed correction if any of them are not quite true, but if any of them are definitely wrong, please say so.

  1. DPX is a file type that tries to capture the intensity of light in a way similar to human vision: sensitive in low light (more bits required to capture different levels), not so sensitive in bright light (less bits required).
  2. This variation in bit-intensity, reduces the number of bits required to capture a range of light intensity.
  3. If this intensity-data is displayed unaltered on a computer monitor, the image will not appear as it did in real life. It will be washed out.
  4. That doesn't really matter, because the image will be altered anyway to achieve a certain look (colour, contrast, sharpness…). It is not necessary to start with an accurate portrayal of the original if the image is going to be altered.
  5. However, if you do want an accurate portrayal of the original image (say, because it's an historic image), you've got a problem. How do you get the original image back from the distorted intensity-data? You need to know the maths of how to do the reversal.
  6. There are certain file types that will capture intensity-data in a way that will display images correctly on a computer monitor: Tiff, ProRes…
  7. A conversion from DPX to any ProRes flavour should appear virtually identical when viewed on screen.

 

Questions

1. What is still unclear to me is how does a scanner operator convert DPX to ProRes accurately? When Rob Houllan uses Resolve to convert the DPX output of his various machines to ProRes, who provided him with the reversal maths? The manufacturer of the Xena, the ScanStation, the Spirit, the Arrilaser?

2. Does he have four different LUTs (not that I know what they are), that he enters into Resolve?

3. Or is the reversal maths somehow encoded into the DPX itself, as metadata?

4. If I do happen to get hold of the DPX files for my scans, and assuming I want to see images on my iMac screen that look the same as on film, will that be possible in Premiere? In Resolve?

 

Link to comment
Share on other sites

  • Premium Member
9 minutes ago, Guy Burns said:

DPX to ProRes   I want to work out where this DPX to ProRes 422 problem lies. I'm new to DPX, but have been reading and thinking about it a lot. I'll make some comments below to test my understanding of DPX. No need for detailed correction if any of them are not quite true, but if any of them are definitely wrong, please say so.

You are overthinking this. I'm not convinced it is a DPX to ProRes problem. I think (I don't know) that it's just been scanned really sat-up. If this is who I think it is, he knows what he's doing, and there might be a reason why that is.

P

 

Link to comment
Share on other sites

David   No particular reason that I wanted to use ProRes except it seems to be a common delivery file. Before this slight problem turned up, I didn't even know what DPX was. But now, after testing the DPX files in Premiere, they certainly slows things down on my system, so I'll be sticking with ProRes.

Downloads

• The DPX files can be downloaded in a zip, here. There are about 100 of them, 1.8GB. Or you can download just the first 10 here.

• The corresponding ProRes 422 files can be downloaded here. There are two ProRes versions in the zip -- one is direct from the DPX, called Test Excerpt (original); the other has been level-corrected by the operator before transcoding to ProRes, called Test Excerpt (graded).

My attempts at grading   I've loaded the three files into Premiere, lined them up, and tried adjusting the DPX files so that they appear similar to Test Excerpt (graded). I can't do it. Using RGB Curves, there's no way I can achieve the same look. The problem is in the shadows. No matter how much gain I apply, there is always a certain amount of straight line bottoms as viewed in the RGB Parade.

So, I've made the decision to let the operator adjust levels, and deliver to me in ProRes 4444. I'm also going to ask him for one more scan of the test file, this time to Tiff. He doesn't mind me doing all this testing, cos I'm paying top dollar.

Anyone Else?   If anyone wants to experiment with the files, I'd be most interested to find out if you can turn the DPX files into something that looks similar to Test Excerpt (graded). And I'd like to know how you did it.

 

Edited by Guy Burns
Extra download added
Link to comment
Share on other sites

  • Premium Member
7 hours ago, Robert Houllahan said:

Those DPX frames are clipped in the shadows (Looking at them in Resolve scopes) specifically the red channel and the shadow levels are elevated.

The scanner was not setup right IMO.

clipped.JPG.2afb6712b87eb20c63170ede8f65a11c.JPG

I guess the film was shot on reversal stock, right? That might explain the crushing of the shadows or the HDS+ scanner not having enough dynamic range to see details in the shadows of a reversal film

Edited by David Sekanina
additional question
Link to comment
Share on other sites

Thanks for the comments. The test film is Super 8, positive.

I find it difficult to accept that the scanner may not have been set up correctly. However, after he adjusts levels in Resolve, the operator is able to deliver scans in ProRes 4444 that, to my eye, appear good quality.

Someone else is having a look at this problem for me using the latest Premiere. I run CS6, and he had me almost convinced that it was my system at fault. Something about CS6 and 8-bit. Maybe it is, but I'll wait till he gets back to me before I think about moving to a newer Premiere.

This young fellow with his new Filmfabriek is really trying to do the best he can. It's hard to believe the problem is with his scanning.

 

QUESTION    I always specify to operators that I want manual exposure (clear film to be max RGB, unexposed film to be min RGB, or other settings they think gives the best result); slowest recommended scan speed; and no auto corrections of any kind (sharpness, contrast, colour). Yet too often the scans have been problematic.

I've ended up with about a dozen scans of my test film because I always ask for (and pay for) tests at various scanner settings before giving the go ahead on the real films. One operator told me that scanning at his normal speed gave perfect results, yet my test at half speed was sharper (Muller HDS+, UK). That wet scans offer the same quality as dry scans, yet the wet scans had obvious colour differences (UK again, and Filmfabriek had to step in and upgrade their software). That film white will be scanned as maximum RGB, but comes back obviously blown (ScanStation, Denmark).

Is scanning at high quality on an expensive machine difficult to do? Is there something I'm missing?

Link to comment
Share on other sites

  • Premium Member

Here's a waveform of an ungraded negative scan, 10 bit DPX with some dark, almost black areas. The bottom isn't clipped (sitting on a line) like in your DPX Waveform. Only on the far left and right of the Waveform, you have a line at the bottom, which represent the mask (absolute black) in the scan:

wave.JPG.061296fbf3914574c02b7c87f6a2f505.JPG

 

Link to comment
Share on other sites

Thanks, David, for the waveform. I'm new to DPX, so every little bit of extra info helps in my understanding.

And to repeat something from a previous post of mine: for anyone who has downloaded the files, I'd be most interested to find out if you can turn the DPX into something that looks similar to Test Excerpt (graded). The operator can do it in Resolve, supposedly with a simple levels adjustment. I'd like to know if it really is that easy, or if there is fancier grading involved (possibly a workflow suggested by Filmfabriek?)

 

Link to comment
Share on other sites

  • Site Sponsor

When i say the scanner wasn't setup right I think the LED lamp level wasn't bright enough and not balanced and then the operator used the gpu grade tools in the scanner software to elevate the shadows.

They possibly did this to avoid clipping the hilites, maybe due to the lack of DR in that sensor?

 

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...