Jump to content

Is it possible to get wide gamut color out of Scanstation scans (color primaries wider than sRGB/REC709 locations)??


Larry Baum
 Share

Recommended Posts

12 minutes ago, Robert Houllahan said:

The data is stored in log format, directly corresponding to density of the original negative. Since the scanned material is likely a negative, the data can be said to be "gamma with log encoding".

There is no suggestion in the posted description of cineon encoding that positives (as distinct from negatives) would be encoded in linear rather than log.

Be it the density of negatives or the density of positives, density is defined as the log of reciripical transmittance.

Why log encoding?

Because, per bit, log encoding is more accurate than linear encoding.

There is a claim that positives are encoded in terms of linear values. This may very well be true, but there is nothing in the posted description of cineon to suggest this is the case. So why post it?

DPX (based on Cineon) supports both linear and log encoding (among others) so its not out of the question that DPX encoding of positives might be done in linear format. 

Either way it would be good to know what the actual practice is. By way of analogy, it would be good to know whether a list of numbers representing distances were provided in millimeters or inches.

 

 

 

 

 

 

Link to comment
Share on other sites

  • Site Sponsor
12 hours ago, Carl Looper said:

There is no suggestion in the posted description of cineon encoding that positives (as distinct from negatives) would be encoded in linear rather than log.

Actually it is pretty clear that negatives are scanned with (gamma cineon log) and that prints do not have that curve applied.

IDK if you think you are somehow magically reinventing the wheel here but since Kodak's Cineon data scanning was invented in the early 1990's hundreds (or thousands) of billions of feet of film has been scanned with 99.99997% of all prints and other positives like Reversals being scanned as linear data with the 00.000038% having been scanned (somehow) as Log because some scanner op made a mistake.

There is nothing new here this ground has been covered and re-covered and run over by artillery horses then model Ts then Chevys then Land Cruisers and tanks then horses again and a few HumVees and then a pack of dogs then followed by crows and a herd of cats.

This conversation is moot.

Link to comment
Share on other sites

  • 1 month later...

Sorry I've been out of my own thread for a while.

I've received more files (my original samples were all 5K and my new ones were mostly full 6.5K with a few 5K for additional comparison). I also got a little bit of an IT8 chart scanned.

In the end it seems that full 6.5K scans from it ARE stored in some wide gamut format (contrary to the totally incorrect metadata which claims something similar to REC709 primaries). It's something weird though. Vaguely like P3, but definitely not P3. It means that you need to painstakingly feed it calibration charts and try to build a LUT or manually adjust primary locations by eye until the data matches what is on the 35mm prints. Kind of weird, the files really should have metadata that lists what primaries they are stored as! What is the point of making someone spend endless hours trying to say tune up adjustments to the primaries in Adobe ACR compared to what the metadata falsely claims they are? Or spend time trying to build some program to make a custom LUT out of chart data and so on. Sure you may need to fine tune to adjust how the scanner sees a certain stock or this or that but we are not talking about that but simply finding out what format the data itself is stored in reference too. Yeah, DSLR makers don't give you the proper metadata for the primaries of the RAW output of the cameras which is silly too but at least there the big stills processing software companies all test the sensors and figure it out and provide a RAW interpretation file for each camera model. It doesn't seem any of them do that for things as esoteric as motion picture film scanners.

Now whether or not these stored in reference to some custom wide gamut format 6.5K scans actually store any wide gamut colors or the engine actually clips them down first and then stores the data is not yet known for sure yet. So far it seems a little suspicious that things seem to stay within REC709 triangle for the most part (and what doesn't might possible just be getting pushed beyond by grading choices). But then I may not have the proper stuff scanned yet, maybe the DI were done in REC709 before printout to film, not sure.

So whether the scans can ever actually ever contain any colors beyond REC709 or not, as motion picture print film certainly can, not yet sure, but the data format CAN hold such colors though, contrary to my initial claims, BUT that is only because the new scans are full 6.5K scans and these seem to use a different reference for the data than any sub-6.5K scans from the Scanstation do.

So, as just mentioned, on the other hand any scans done under the native 6.5K seem to NOT be stored in a wide gamut format, but some regular gamut format. It's vaguely like REC709 gamut, the saturation is about the same, but the primaries seem a bit off, they almost seem to have the same hue as for the full scans and yet the saturation remapped to be something like REC709/sRGB, kind of odd, not sure if that is a bug or not. It definitely doesn't seem capable of storing any colors beyond REC709 (beyond maybe a little twist) in the data format used for the 5K and under scans (which is what my original samples were done in).

So it is seeming like I was correct and not correct. The scans 5K and under, which my original demo samples were, do NOT appear to allow for wide gamut color and seem remapped to REC709 max saturation (although with an odd twist to certain colors, almost like they left the primaries hues in some wide gamut format). But it seems that once you do full 6.5K scans that suddenly the data is stored in reference to a different color gamut, one that is definitely stored with saturation levels that seem more like something wide gamut, maybe vaguely along the lines of something could allow for P3-ish levels or just a bit less max saturation. Tentatively, the blue primary seems quite different from P3 though in particular.

 

 

In the end it seems that full 6.5K scans from it ARE stored in some wide gamut format (contrary to the totally incorrect metadata). It's something weird though. Vaguely like P3, but definitely not P3. It means that you need to painstakingly feed it calibration charts and try to build a LUT or manually adjust primary locations by eye until the data matches what is on the 35mm prints. Kind of weird, the files really should have metadata that lists what primaries they are stored as. Whether they actually ever contain any colors beyond REC709 or not, not yet sure, the data format can hold such colors though. On the other hand any scans done under the native 6.5K seem to NOT be stored in a wide gamut format, but some regular gamut format. It's vaguely like REC709 gamut, the saturation is about the same, but the primaries seem a bit off, they almost seem to have the same hue as for the full scans and yet the saturation remapped to be something like REC709/sRGB, kind of odd, not sure if that is a bug or not. It definitely doesn't seem capable of storing any colors beyond REC709 (beyond maybe a little twist) in the data format used for the 5K and under scans (which is what my original samples were done in).
Link to comment
Share on other sites

More simply since my last post I have received some 6.5K scans in addition to my initial 5K demo scans:

It seems that 6.5K scan mode has data stored in reference to some wide gamut format that probably could hold most colors film is capable of while any scans it does below 6.5K seem to be full re-mapped to a non-wide gamut format that seems like it would have to clip some colors film is capable of.

It seems the metadata for color gamut is totally utterly wrong for ALL format outputs from the Scanstation at 6.5K and somewhat wrong for ALL output formats from Scanstation under 6.5K.

Sub 6.5K stuff will load into Resolve automatically as REC709 and look kinda OK, but on close examination some colors are twisted, reds go orange, you get wrong relatively color balance, it can NOT be correct by color temp or tint or most normal color controls, you need to do more advanced adjustments like adjusting primary locations or use custom LUTs. It certainly seems like it would clip or remap down wider gamut colors that film can produce. The data does not seem to be set up to be able to handle wide gamut colors.

6.5K stuff loads in Resolve and ACR looking all dull and weird, since it is treated as something close to REC709 gamut but it is not at all REC709 gamut, metadata utterly wrong. To get proper colors it seems you must do extensive calibration adjustments to Adobe ACR primary locations or feed custom built LUTs into other programs so the data can be interpreted in the format that it is actually stored in which seems to be some custom moderately wide gamut format. These 6.5K scans might contain all colors 35mm film has or might not, the data certainly allows for it, whether it does or still gets clipped, not sure yet.

 

Link to comment
Share on other sites

opps OK even more simply hah:

6.5K scans actually do seem to be stored in a format that allows for wide gamut colors

any scan under 6.5K does NOT seem to be stored in a format that allows for wide gamut colors (all my sample scans when I was first posting here were under 6.5K)

it is not yet know whether the 6.5 scans actually deliver any wide gamut colors, although the data format would make it possible

the metadata for color gamut seems to be somewhat wrong for all files scanned under 6.5K and utterly wrong for all files scanned at 6.5K, it tells you totally wrong info for programs on how to interpretate the colors so you need to RADICALLY custom adjust the primary locations in programs like ACR or use color charts and special software to build up LUTs, not sure why they don't give proper metadata

 

 

Link to comment
Share on other sites

4 hours ago, Larry Baum said:

6.5K scans actually do seem to be stored in a format that allows for wide gamut colors

any scan under 6.5K does NOT seem to be stored in a format that allows for wide gamut colors (all my sample scans when I was first posting here were under 6.5K)

I have avoided this thread because, quite frankly, I think you're looking for a problem that doesn't exist and you have yet to post a single image demonstrating the supposed problem.

But I need to correct this:

The ScanStation's resolution modes have nothing to do with color. They're unrelated. The "modes" in a 6.5k scanstation refer only to the pixel dimensions (and subsequently the speed at which the scanner runs, because lower resolutions mean less data).

Inside the scanner, the camera and lens move closer to or farther away from the film, depending on the mode. In a lower res mode, the only difference from a high res mode is that instead of using the entire sensor it's using a crop, and can run at a faster speed. That is, in 2.5k mode, it's using a 2.5k crop of the 6.5k sensor and is capable of running at up to 60fps. In 6.5k mode, the entire sensor is used because the camera and lens are in a different position, filling the sensor with the image of the film, and this runs at a lower speed because - more data. 

The internal processing of the data, the output file formats, all of that is identical regardless of the resolution. The only difference between the various resolution modes is the pixel count and the speed at which the machine can run.

Now, if you're comparing DIFFERENT SCANNERS - say an original 2.5k ScanStation (which used a CCD camera), or a 5k ScanStation (which used a 5k CMOSIS CMOS camera), or a variant with a 4k camera (which I think is still Sony IMX like the 6.5k but a different model), or an Archivist (which technically isn't a ScanStation and uses a different model Sony IMX camera), then potentially you might see some color differences. But those are different machines, not different modes on the same machine. 

Edited by Perry Paolantonio
Link to comment
Share on other sites

On 1/19/2023 at 9:11 AM, Perry Paolantonio said:

I have avoided this thread because, quite frankly, I think you're looking for a problem that doesn't exist and you have yet to post a single image demonstrating the supposed problem.

But I need to correct this:

The ScanStation's resolution modes have nothing to do with color. They're unrelated. The "modes" in a 6.5k scanstation refer only to the pixel dimensions (and subsequently the speed at which the scanner runs, because lower resolutions mean less data).

The internal processing of the data, the output file formats, all of that is identical regardless of the resolution. The only difference between the various resolution modes is the pixel count and the speed at which the machine can run.

 

Well all I can say is the 5k DPX and DNG file samples I got appear to be in a different gamut than the ones I got in 6.5K. All done on the same machine with the same settings as far as I've been told. The 6.5K ones load into ACR or Resolve with noticeably less saturation than the 5K ones. I'm not running the machine myself, but that is what I can say based upon the files provided. I mean maybe something got toggled somewhere somehow. Again all I can go on is the files I got and what I was told. I'm not running the machine and was not there when stuff was scanned. The only thing I can think of is that to produce lower than 6.5K they might need to de-Bayer first (depends upon how it creates lower res) and once they are doing that, maybe they do more processing? Although if what you say is how it generates lower res then that would not seem to be the case, it shouldn't need to de-Bayer first.

The metadata for all of them seems to be wrong. They are all stored with a forward color matrix that would make sense for something very close to REC709 gamut. Applying that to 5K files I got looks kind of correct, saturation looks normal, but you do see that there is a weird shift in some colors, red is too orange, some other colors don't balance relative to others (nothing that color temp or tint fix, they can fix one color but then send the other off), maybe subtle at first glance, but quite real in the end. Anyway, the fact that REC709 gamut applied to interpret the 5K files looks remotely normal at all would kinda imply the data is not in native scanner wide gamut but has been pre-processed. Applying the metadata or REC709 to the 6.5K files I got doesn't look correct at all, same as for 5K but also with saturation much too low. If you do large adjustments to Adobe ACR Calibration panel to adjust primary locations you can eventually get it to start doing stuff that looks normal.

Anyway, how do you explain the forward matrix in the metadata being so similar to a forward matrix for REC709? Either it's completely wrong (which actually does seem to be the case for the 6.5K files) but in that case why do they put bogus metadata and not metadata that properly describes the gamut the data is stored in reference to? or it's correct and the data is not in native scanner gamut but processed down (As seems, sort of (not sure about the color twist), to be the case for the 5K files I got).

 

 

 

Link to comment
Share on other sites

I can find and show this demo at least to show the metadata for 6.5K files being wrong part at least.

The 6.5K metadata says to process files like this as default load:

52633876262_9c70b0042e_b.jpg

If major changes are applied to the near REC709 gamut by altering the primaries to something like some custom wide gamut mode then people can get it to load more correctly like:

52634874858_03336055e1_b.jpg

 

With the 5K samples I got it would be like the above only the first image would have about the same saturation as the second as a default load, but with the color tones still like you see in the first image and because of that that is why I was questioning if it was storing native wide gamut or not, since they seemed to load fully saturated when interpreted as REC709 gamut. The 6.5K files I got don't seem to load fully saturated when interpreted as REC709 gamut.

 

 

Edited by Larry Baum
Link to comment
Share on other sites

Anyway all I can go by is what I received and what I was told and how the files are loading on my machine in ACR (and seemingly also Resolve, although I didn't use this too much). Maybe I was told incorrect stuff or something got toggled or there is a bug in ACR or something. All I can say is what I was told and what the metadata in the files and how they are acting. I do agree it would seem fairly strange for 6.5K files to be stored in reference to a different gamut than 5K ones (especially if it didn't have to de-Bayer and downscale to 5K but it seems like it just uses a smaller part of the sensor so that does make it quite strange). I don't know why it should be at all.

The 5K (and 6.5K) files have metadata suggesting they are nearly REC709 gamut (although this appears to be wrong for 6.5K files at least, unless the difference is not due to some ACR/Resolve software bug that is for some reason making them load differently) and the 5K samples I got acted like REC709 in processing so that is why I was asking about how you get wide gamut out of this scanner. Now the 6.5K files I got later behaved differently. It does seems strange, but they do. As a bit of a sanity check I was give a tiny new set with 6.5K and 5K and in the new set the two sizes once again behaved differently. Maybe it's a bug in ACR or Resolve. Maybe the scanner somehow hit something different each time for 5k and not for 6.5k. Maybe the scanstation is doing more stuff to the lower file sizes. I dunno. If they truly are both in the same format, than there is some bug in some software on my end somewhere along the line. And then there is the question is the metadata correct or not and is the way they are loading as 5K or as 6.5K on my system correct. In one case, the scanner would not be producing wide gamut output but clipping to REC709. In the other case, at the least, the data output would be stored in some moderately wide gamut format. Loading the 5k samples into RAW software that was told to ignore metadata for color gamut and just pretend the data is stored in reference to REC709/sRGB primaries made them load looking fairly normal (do that to wide gamut format data and it looks all washed out). I didn't try that for the 6.5K files.

Edited by Larry Baum
Link to comment
Share on other sites

On 1/19/2023 at 9:11 AM, Perry Paolantonio said:

I have avoided this thread because, quite frankly, I think you're looking for a problem that doesn't exist and you have yet to post a single image demonstrating the supposed problem.

But I need to correct this:

The ScanStation's resolution modes have nothing to do with color. They're unrelated. The "modes" in a 6.5k scanstation refer only to the pixel dimensions (and subsequently the speed at which the scanner runs, because lower resolutions mean less data).

Inside the scanner, the camera and lens move closer to or farther away from the film, depending on the mode. In a lower res mode, the only difference from a high res mode is that instead of using the entire sensor it's using a crop, and can run at a faster speed. That is, in 2.5k mode, it's using a 2.5k crop of the 6.5k sensor and is capable of running at up to 60fps. In 6.5k mode, the entire sensor is used because the camera and lens are in a different position, filling the sensor with the image of the film, and this runs at a lower speed because - more data. 

The internal processing of the data, the output file formats, all of that is identical regardless of the resolution. The only difference between the various resolution modes is the pixel count and the speed at which the machine can run.

Now, if you're comparing DIFFERENT SCANNERS - say an original 2.5k ScanStation (which used a CCD camera), or a 5k ScanStation (which used a 5k CMOSIS CMOS camera), or a variant with a 4k camera (which I think is still Sony IMX like the 6.5k but a different model), or an Archivist (which technically isn't a ScanStation and uses a different model Sony IMX camera), then potentially you might see some color differences. But those are different machines, not different modes on the same machine. 

Looking at the files more it does seem like the 5k and 6.5k actually were not done exactly the same way. The 5k were exposed a bit more with a slightly different color balance used it seems (although the metadata seems to claim the LED balance was the same so not sure, maybe the color balance difference is because they were processed differently by scanner software at some stage or maybe the metadata doesn't report a difference in how the WB was actually really set or maybe ACR uses a twisted profile or maybe who knows what was done with the early 5K quick demo samples). So until I get some more samples done same time, exact same way for 100% sure, I suppose ignore my last few posts.

That said, if I raise the whites to make them look more or less the same exposure wise and set white balance on same white part to get white balance the same, the 6.5K one needs a huge +38 saturation to look similar (tint and colors are still a touch different). Just messing with color temp and tint sliders in ACR, for this one particular image, it doesn't seem to affect saturation too much at all though. Although perhaps the scanner sensor and such is very sensitive to the balance of it's LED lights and how it sees film and maybe it records a lot less saturation if it sends a bit less light through or something and maybe even for this color it would do that even if that color doesn't act like that in ACR? I wouldn't think it should make THIS much of a difference from a bit of an exposure difference and slight, nothing huge, seeming shift in white balance on the scanner to the images overall saturation, but maybe?

Anyway, since it seems they actually were not scanned the exact same way, I guess it is better to ignore what I said until I can verify it happens when the conditions truly were exactly the same for the scans. I'll try to get a sample 5K and 6.5K frame absolutely done the exact same way at the same time and see what happens. Maybe something was set weirdly or vastly different the 5k demos frommuch earlier on. Again, it's hard to say not running the machine myself.

At the least though there do seem to be the somewhat strange things:

1. metadata suggests the data is more or less REC709 gamut which one would not think should be the case for native scanner output, of course the metadata could be and probably is wrong, but why put in incorrect metadata? data should really be stored in reference to something correctly given so programs know how to interpret it (now sure the data may still need some calibration adjustment after that, but the base starting point should at least be correct) Or if the metadata is correct, how can native RAW from the scanner end up being so ultra close to REC709 gamut? (side note: at least with the way the 5K samples were scanned they sure seem to work and act like they have REC709/sRGB-like data in terms of how saturated the primaries are (although with whatever was done with the 6.5K that doesn't seem to be the case, so it's all strange, trying to adjust exposure to equalize didn't match saturation at all, but ACR might do funny stuff with color engine, etc. etc.))

2. how come if the DNG are true RAW files it puts the display rather than scene referred flag in the metadata and tells programs to treat them closer to JPGs and TIFFs than real RAW files (for instance ACR gives a points slider for color temp instead of the Kelvin scale, etc.), you'd think true RAW scanner output should be scene referred flag like it is for DSLR RAW images or for most stills film scanner RAWs

 

Link to comment
Share on other sites

I'm starting to think those initial few quick 5K samples I got maybe had the LED balance on the scanner set in some odd way. Either that or on the new 6.5K files.

The thing that makes it tricky is the darn metadata claims same As Shot WB and scene and LED temp for both! But looking into things like colors in sprocket holes that does not actually seem to be the case.

So it seems the metadata is likely not reliable at all (which isn't a great thing). And I'm not sure that things actually were scanned using same settings as I thought, so at this point it's hard to know exactly what is going on beyond that the metadata appears likely a false mess for the outputs from Scanstation and that things might actually have been scanned in different ways each time contrary to what I thought, the quick demo frames and bits and the real scans at full res. Anyway, I need to get more info at this point to figure out what the real deal regarding anything going on is. All I can say is the early 5k samples I got behave totally differently (actually closer to how the metadata says they should, but that would be weird if it was supposed to be like that since native scanner data should not be REC709-gamut-like, so maybe something was set weird and it was just a coincidence that it worked closer to the metadata for gamut and the metadata is set wrong) than all new 6.5K samples and scans which all act virtually identically and that the metadata does not let programs give the proper colors for either set, especially not for the 6.5K stuff although doing a lot of manual adjusting of primaries and so on you can get them all looking correct.

 

 

Edited by Larry Baum
Link to comment
Share on other sites

Man, I'm starting to think that the initial little batch of a few quick 5K demos I got, that something was set all strangely with them which added a lot to the general confusion along with the seemingly generally bogus metadata in the files. I found another 5K little set that seems to act more like my 6.5K final scans.

Who knows at this point after all the confusion, but I suspect it may all come down in the end to:

5K and 6.5K and so on probably will all act the same (which would make more sense)

Scanstation probably does scan out data in some moderately wider than REC709 gamut format, perhaps something a trace smaller than P3 (still not 100% sure though and you'd think the true RAW data might be even wider than that, not that wider than that would be needed for film)

metadata put out by the scanstation is almost all completely bogus and best ignored, but this also means you need to do some complex type corrections, adjusting primary locations and such (still not 100% sure, but seems likely)

 

Link to comment
Share on other sites

Anyway I need to wait to get another set of files to really tell what is going on. It's hard with potentially bogus metadata and then not being 100% sure exactly what was done and set for each scan.

Some things are still not adding up. And the 5K early samples still ended up with relatively similar color balance in the main image area so you wouldn't think anything should have been so radically different in saturation, etc. (sprocket holes have extremely different color temp, optical track area has only moderately different color temp, might just be sprocket clipping after adjustments applied for the way this system can expose each channel on its own, so may not mean so much and would make you think that the 5K set should not act so differently than the other sets)

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...