12-bit Linear RAW == 10-bit Log DPX?

[Peter Moretti]

I have read that 12-bit linear data can be stored accurately as 10-bit log data. I don't see how this is possible.

 

A 12-bit linear scale has 4,095 gradations of equal value. A 10-bit log scale has 1,023 gradations of non-equal value. In the log scale, each gradation is twice the value of the previous gradation. A Log scale work well for representing sound and light intensity because it closely maps how we perceive these stimuli.

 

But as for re-mapping 12-bit linear to 10-bit log w/o any loss or error introduction? This seem impossible to me.

 

I am posting this ? because a fair amount of Red workflows perform a RAW to DPX conversion as an intermediary step.

[Phil Rhodes]

When people say things like that, they're usually referring to the fact that the natural logarithm of 1024 equally-spaced 12-bit code values happen to fall on preexisting 10-bit code values, so you don't get any quantisation error.

 

I should point out that this is largely moot anyway because few cameras (and most certainly not red) actually have 12, 10, or often even 8 bits of precision due to the noise floor of the imager, so I shouldn't worry about it too much. The purpose of more than 8 bits is to allow room for colour correction, not to store real camera data.

 

a fair amount of Red workflows perform a RAW to DPX conversion as an intermediary step.

 

...and thereby lose a good chunk of the cost advantage of shooting red, but hey, that's How Things Are Done and who are we to argue!

 

P

[John Sprung]

The other thought behind this notion has to do with how we see brightness differences. A just noticeable difference in brightnesses is a change of about 1%. So, linear uses too many codes in the high end and starves the shadows. Log comes closer to using codes where your visual system needs them. Obviously they're not mathematically the same, but they sort of look just as good -- at least that's the idea. (Yes, vision isn't exactly logarithmic, but it's a lot closer to log than to linear.)

 

 

 

 

-- J.S.

[Dirk DeJonghe]

Kodak kind of invented the log format when they developed Cineon; the characteristic curve of film was linear encoded to 10 bits (1024 levels of grey per color), but since the exposure axis of this curve is already log, we can speak about a log encoding.

What matters is that you have much more precision in the shadows where it matters and less in the superwhites where linear formats waste too much bandwidth. Very recommended reading on this subject (also for DoP's) 'Digital Compositing for film and video' by Steve Wright. Best explanation of log/linear I have seen.

[Brian Drysdale]

David Newman from Cineform discusses log v linear in his blog.

 

http://cineform.blogspot.com/search?update...;max-results=34

 

There was some online debate when RED were deciding which way to go.

[Peter Moretti]

When people say things like that, they're usually referring to the fact that the natural logarithm of 1024 equally-spaced 12-bit code values happen to fall on preexisting 10-bit code values, so you don't get any quantisation error.

...

 

Phil, I know what a natural log is, it's the exponent by which e is raised to equal a given value. And I know that 2^10 = 1024 and that 2^12 = 4096. But could you please explain your statement more? From a purely mathematical point of view, I don't see how you can go from 4096 to 1024 and not lose precision or introduce error regardless of what the scales are. Now the error may not be visually significant.

 

 

I should point out that this is largely moot anyway because few cameras (and most certainly not red) actually have 12, 10, or often even 8 bits of precision due to the noise floor of the imager, so I shouldn't worry about it too much. The purpose of more than 8 bits is to allow room for colour correction, not to store real camera data.

...

 

Is this really the case? I would think that the dynamic range of the sensor would play an important role in determining how many bits are needed. Of course I want 8 bits to be enough.

 

Thanks much.

[Phil Rhodes]

Hang on, I'm talking drivel about natural log curves. Sorry. There are all kinds of curves in use in imaging for this sort of thing. Some of them - and as I've already proved, we're dancing on the edge of my level of knowledge here - are such that log to lin conversions between bit depths will always output an integer - that is, the 1024 code values of 10-bit log happen to fall exactly on 12-bit code values, so you don't lose anything to rounding errors. Someone like - and may whoever's in charge be merciful upon me for suggesting this - someone like Graeme Nattress would probably be able to explain this better. Most systems just seem to choose a curve that makes them look good, witness the oddities used by SI on the 2K camera, as detailed in the previously linked forum post.

 

As to bit depth against dynamic range - yes, exactly, but the point is that the useful dynamic range of most video cameras is determined by the noise floor. I can detect 9 stops of dynamic range on a JVC GY-HD250, which would be outstanding if the bottom two and a half weren't buried in snow.

 

P

[Keith Walters]

...and thereby lose a good chunk of the cost advantage of shooting red, but hey, that's How Things Are Done and who are we to argue!

P

MR RHODES!!

Once again I am appalled, appalled by your criminally lamentable lack of vision, your dastardly insensitivity to the sensibilities of the struggling artist struggling desperately, and so often, futilely against the Plutocratic might of the Sony’s, the Panasonic’s and all the other sad filth-ridden Satan-espousing perpetrators of “The System” that forever, forever plots and schemes to deny them their right, their absolute God-Damn God-given RIGHT to shoot, produce and … DISPLAY whatever lies at the end of the path their own particular cinematic Muse has set them irrevocably upon!!

 

Would we now have 100 billion web pages, each one packed with the quintessential essence of excellence, all that is worthy of knowing, knowledge that has been won through the distillation of the intellectual sweat and blood of the finest, most agile and fertile young minds that our species has produced, if febrile exsanguinate critics such as your good self held sway?!

 

Would my local supermarket carry 27 different brands of yoghurt? Eight different brands of butter? 10 different brands of margarine? Nine different brands of milk? 12 different brands of biscuits, Nine different brands of bread? The list just goes on.

 

How DARE you sir!! I shudder to think of the kind of sad colourless world we would live in if everyone heeded the bitter deprecations that appear to spring reflexively and uncontrollably from the cracked and purulent lips of dried-up, visionless, joy-corroding curmudgeons such as yourself, sir!

 

Aaargh! Words fail me ….

[Keith Walters]

As to bit depth against dynamic range - yes, exactly, but the point is that the useful dynamic range of most video cameras is determined by the noise floor. I can detect 9 stops of dynamic range on a JVC GY-HD250, which would be outstanding if the bottom two and a half weren't buried in snow.

 

P

I'm afraid this is one of the most refractory misconceptions about modern video cameras.

 

The noise floor is largely determined by the limitations of the hybrid digital/analog circuitry used in the CCD or CMOS sensor BEFORE it ever gets anywhere near an analog-to-digital converter.

 

A silicon photocell can only charge up to about 0.6Volts. After that it starts to leak internally, which is perfectly analogous to an overflowing cup. So without clipping, the brightest parts of the image can only ever produce a voltage of 0.6 Volts, or 600 millivolts.

 

10 stops is about 1,000 to 1, so a 10-stop tonal range (in other words, the variation on brightness between the brightest and darkest parts of an image) would be producing about 600 microvolts in the most dimly-lit photocells. The problem is, getting these analog charges off the chip and into the processing circuitry requires digital switching voltages thousands of times the amplitude of the signals themselves. The minutest amount of phase jitter in the clock signal will severely contaminate low-level signals, to the point where anything much below 600 microvolts becomes unusable. Clock jitter is produced by the vibration of the silicon atoms; there is simply no known way of preventing this.

 

If it was just a matter of using an analog preamplifier to change the gamma curve to make better use of the ADC, then that’s what manufacturers would do, at least in the case of CCD sensors. As it is, there is virtually no advantage in doing this, since the CCD sensors introduce far more noise than the ADC.

 

I doesn’t matter in the slightest what you do the video signal AFTER the damage has been done; the noise is there, and that’s it. Oh yes, you can do all sorts of tricks to disguise the noise, but you then inevitably run into the perennial problem that a computer can’t really tell the difference between random picture detail and random introduced noise. Just as it can't always tell what is meant to be sharp and what is meant to be soft.

Edited June 7, 2009 by Keith Walters

[Keith Walters]

Someone like - and may whoever's in charge be merciful upon me for suggesting this - someone like Graeme Nattress would probably be able to explain this better.

Why don't you just open an account over on Reduser and ask him yourself? :lol:

Just a minute ... where's my stopwatch?

[Scott Fritzshall]

Converting linear 12bit to log 10bit is not mathematically lossless; you do in fact lose data. The thing is that you lose brightness data in the bright areas of the image- where the eye doesn't really perceive brightness differences very well- and keep more brightness data in the darker parts- where the eye is very perceptive about changes in brightness. In the real world, .dpx files have been used without anyone really telling the difference for a long time, so it's generally not of any practical concern. At the same time, storage and processing has improved over the last decade to the point where this sort of thing shouldn't really be necessary anymore. I haven't involved myself in RED workflows too much so far, but I don't know why you wouldn't just want to skip this whole problem and just use 16bit .exr files, which are able to contain all of the information that the RAW file has. They're totally linear as well, which is nice once you figure out the proper workflow for something that's truly linear.

Edited June 7, 2009 by Scott Fritzshall

[Phil Rhodes]

The noise floor is largely determined by the limitations of the hybrid digital/analog circuitry used in the CCD or CMOS sensor BEFORE it ever gets anywhere near an analog-to-digital converter.

 

Yes, of course. How does this disagree with the idea that useful dynamic range is determined by noise floor?

 

You have your 600mV photocell, you have a voltage below that at which noise becomes an unacceptable percentage of the signal. That's your dynamic range, and yes, of course, exactly what an unacceptable amount of noise is a matter of opinion.

 

joy-corroding curmudgeon

 

Excellent - I'm gonna have that printed on a T-shirt!

 

P

[Keith Walters]

Yes, of course. How does this disagree with the idea that useful dynamic range is determined by noise floor?

Not at all. I was simply pointing out the futility of attempting to separate the average chucklehead from his cherished notion that all that is necessary to raise the performance of yer average $300 handycam to celluloid-like perspicuity is to slap in a ADC wiv' more bits.

 

'n megahertzes and 'megabytes 'n algorithms 'n ... stuff ...

Yeah, thassit, more algorivms...

 

I suppose I entertain a faint hope that if I keep saying this over and over and over again, after several years (or decades) maybe one or two people will notice that my predictions bear a vague resemblance to reality.

[Peter Moretti]

...

 

A silicon photocell can only charge up to about 0.6Volts. After that it starts to leak internally, which is perfectly analogous to an overflowing cup. So without clipping, the brightest parts of the image can only ever produce a voltage of 0.6 Volts, or 600 millivolts.

...

 

Keith, is this the case for both CCD and CMOS sensors? I ask b/c I've read that CCD's tend to have greater dynamic range than CMOS. But if both are limited to .6V, then I don't really understand where the CCD's dynamic range advantage comes from.

 

Thanks much.

[Phil Rhodes]

Not to tread on Keith's toes, but the actual light sensing equipment is a photodiode in both cases; it's the same fundamental technology. The reason CCDs may seem to have higher dynamic range is that, since there's rarely much if any circuitry on the substrate other than photodiodes and charge control transistors, the individual cells can be bigger. This is termed a higher fill factor, and means that for a sensor of a given resolution and physical size there is more actual sensor on a CCD, since there's no space taken up by support electronics.

 

This means that each subpixel can store more electrons, a greater well charge capacity, meaning that the difference between full and empty is larger.

 

Actually this is a case in point regarding noise floor, since a larger well capacity doesn't necessarily (or even usually) give you a larger voltage, since the silicon is still an 0.6V device. Nor, actually, does it give you a lower dark current, since the dark current is influenced by heat and radiation, both of which are more or less proportional to the area of the detector. What it does is give you a much bigger light current, so the effective noisefloor of the thing is much lower, extending the effective dynamic range. This also decreases overall light sensitivity, since you now need more photons to drive each well to saturation, which is presumably why video cameras with a comparatively large ratio of resolution to sensor size (such as 1080p 2/3" broadcast TV cameras, or Red) tend to be effectively faster than a video camera with a similar-resolution sensor that's quite big, such as a D-21, but with poorer dynamic range.

 

P

[Keith Walters]

Keith, is this the case for both CCD and CMOS sensors? I ask b/c I've read that CCD's tend to have greater dynamic range than CMOS. But if both are limited to .6V, then I don't really understand where the CCD's dynamic range advantage comes from.

 

Thanks much.

Yes, as Phil points out, the actual light gathering/conversion/storage mechanism on the sensor itself is identical, whether it's CMOS, CCD, or even the original 1960s MOS technology.

 

Also, silicon photosensors are extremely efficient at photon capture, far more so than even the best hypersensitized astronomical photographic film.

 

Unfortunately, the horsefly in the ointment has always been that that while silicon sensors are exquisitely good at capturing the images, getting those microscopic electrical charges off the chip and into the amplifier/processing circuitry without introducing noise has proven exquisitely difficult. A good analogy is trying to pick up spilled salt grains with barbeque tongs.

 

However, the main reason CMOS sensors are so inherently noisy is the sheer amount of on-chip digital signal processing they require. Not only does that leave precious little room for actual light gathering, it's rather like trying to listen to a distant AM radio station in a room full of computers!

Not only do CCD sensors have more silicon available for photon capture, their support electronics is much simpler, and it all runs at a single clock speed, so introduced clock hash is easier to filter out.

 

Sony have developed a new CMOS technology where the photocells are separately constructed over the top of the processing circuitry, giving them near 100% photon capture with the need for microlenses, but that only gives something less than 2 stops improvement, certainly worthwhile, but hardly a paradigm shift.

 

Digital still cameras can produce consderably better dynamic range, but that's because the data can be clocked off the chip much more slowly - seconds instead of 24ths of a second. Unfortunately as the clock speed in increased, the noise introduced increases exponentially, not linearly.

[Jim Jannard]

Yes, as Phil points out, the actual light gathering/conversion/storage mechanism on the sensor itself is identical, whether it's CMOS, CCD, or even the original 1960s MOS technology.

 

Also, silicon photosensors are extremely efficient at photon capture, far more so than even the best hypersensitized astronomical photographic film.

 

Unfortunately, the horsefly in the ointment has always been that that while silicon sensors are exquisitely good at capturing the images, getting those microscopic electrical charges off the chip and into the amplifier/processing circuitry without introducing noise has proven exquisitely difficult. A good analogy is trying to pick up spilled salt grains with barbeque tongs.

 

However, the main reason CMOS sensors are so inherently noisy is the sheer amount of on-chip digital signal processing they require. Not only does that leave precious little room for actual light gathering, it's rather like trying to listen to a distant AM radio station in a room full of computers!

Not only do CCD sensors have more silicon available for photon capture, their support electronics is much simpler, and it all runs at a single clock speed, so introduced clock hash is easier to filter out.

 

Sony have developed a new CMOS technology where the photocells are separately constructed over the top of the processing circuitry, giving them near 100% photon capture with the need for microlenses, but that only gives something less than 2 stops improvement, certainly worthwhile, but hardly a paradigm shift.

 

Digital still cameras can produce consderably better dynamic range, but that's because the data can be clocked off the chip much more slowly - seconds instead of 24ths of a second. Unfortunately as the clock speed in increased, the noise introduced increases exponentially, not linearly.

 

You and Phil post like you actually think you know what you are talking about...

 

Jim

[Tom Lowe]

I will take the knowledge of Graeme Nattress over James Murdoch any day.

[Keith Walters]

You and Phil post like you actually think you know what you are talking about...

 

Jim

 

What? No! we're just making all this up as we go along.

I can't speak for Phil, but I personally only post here so I can recycle old 19th century vaudeville gags.

You have to admit we haven't done too badly; you're the only one who's noticed so far.

I think I've even had Graeme Nattress fooled occasionally.

 

Well ... Jan von Krogh may have shot us down in flames, but, nobody is ever entirely sure what he is talking about, so the jury is still out on that one. Same with that Portuguese person with the multiple personality disorder and the über-variable English skills.

 

And by the way, does Red Ray-

Oh never mind... :rolleyes:

[Keith Walters]

I will take the knowledge of Graeme Nattress over James Murdoch any day.

Didn't take you long :rolleyes::lol:

Ironic that you invoke the name of someone you assume is a phony Scotsman, in a classic "No True Scotsman" statement :lol:

[Keith Walters]

Sony have developed a new CMOS technology where the photocells are separately constructed over the top of the processing circuitry, giving them near 100% photon capture with the need for microlenses,

That was supposed to be "without the need" :(

[Emanuel A Guedes]

What? No! we're just making all this up as we go along.

I can't speak for Phil, but I personally only post here so I can recycle old 19th century vaudeville gags.

You have to admit we haven't done too badly; you're the only one who's noticed so far.

I think I've even had Graeme Nattress fooled occasionally.

 

Well ... Jan von Krogh may have shot us down in flames, but, nobody is ever entirely sure what he is talking about, so the jury is still out on that one. Same with that Portuguese person with the multiple personality disorder and the über-variable English skills.

 

And by the way, does Red Ray-

Oh never mind... :rolleyes:

 

Keith,

 

Since I've noticed you have a successive triple post since this morning in Europe -- before that there will be another thread closed once again, I'm afraid you are in risk to become to speak alone to yourself one day. It wouldn't be pleasant to see.

 

Who do you exactly miss? This portuguese here? Because since we are a way more than 10,000,000 allover the world, it is just a feeling that it is an uncalled-for engaging this 'person' here.

 

Or do you miss anyone from the infamous filmmaker's gang joke by any chance?

 

'cause whenever someone makes mention of 'James/Jim Murdoch' character you don't forget to attack the usual RED supporters at cinematography.com from day one (prepare yourself Tom ;-) not even forgetting Jan's real name now. Do not enroll me in your past mess, OK? Would it work if I'd ask you with a formal please? Sigh. :lol:

 

I thought 'the über-variable English skills' could add something more in behalf of the real world to a clever read that I am sure you are, despite your usual not beautiful hate(s) over this RED sub-forum. By the way, once you've finally decided to add your own picture to your avatar, why don't you join us at reduser.net where I have mine?

 

E.

[Emanuel A Guedes]

(...)

 

I thought 'the über-variable English skills' could add something more in behalf of the real world to a clever read that I am sure you are,

 

(...)

Once you probably don't speak my native language, it should be your interlocutor to write in yours -- I stand corrected:

 

«to a clever reader that I am sure you are»

 

or

 

«to a clever read that I am sure you did»

 

Similar idea, anyway. I hope. ;)

[Stuart Brereton]

You and Phil post like you actually think you know what you are talking about...

 

Jim

 

Jim, both Keith and Phil have been talking non-specifically about both CCD and CMOS sensors. Neither of them have said that the engineering difficulties they are describing are specific to the Red One. Phil has gone so far as to admit that Graeme Nattress would be able to explain the issues better than he can.

 

So, given that neither of them are criticizing your product, if you are going to disparage their explanations, you should at least offer your own.

 

As owner of a major camera manufacturer, this sniping is a little below you, I would think.

[Scott Fritzshall]

What the hell happened here???