Question about chroma subsampling, bayer sensor patterns and RAW.

[Eamon Colbert]

The past week I have tasked myself with better understanding camera sensors and chroma subsampling and have run into some points of confusion, and was hoping someone could clarify for me. So I understand the bayer sensor patterns allows photosites to measure the number of photons produced by one particular wavelength of light: either red green or blue. I understand RAW is a non debayered image where the Red green and blue color channels info is recorded with little modification. Where I begin to get confused is with chroma subsampling. I understand chroma subsampling is denoted as Y'CbCr where Y'= luma= the weighted sum of the R,G, and B color channels. I also understand that for luma engineers chose a signal that is 21% red, 72% green, and 7% blue. I also understand that for chroma subsampling there is no Cg as it is inferred using the other channels (luma and chroma). Where I begin to get confused is how the green channel is calculated and how this relates to RAW images and the bayer sensor pattern. So images that use chroma subsampling are never actually recording the green channel just calculating it? do RAW images utilize chroma subsampling or do they record Cg? What are green photosites actually measuring/ how are they used? and then the last point of confusion for me was with the signal percentage for Y'. Is the  Y' signal always composed of the same percentage values for the different color channels (72, 21, 7), or do the overall percentages of the signal change based on the different signals of each photosite? for example if a red and a green and a blue photosite all register 100 photons the luma value will be 100, but if for example the red photosite has a photon reading  higher than the other 2 photosites will it now proportionally represent a larger percentage of Y'? Sorry if this is confusing but I'm having a hard time wrapping my head around these concepts

[David Mullen ASC]

With raw recording, there are no red, green, and blue channels yet -- all that is created in debayering the mosaic later.  All there is really is in a raw file is luminance information per photosite.  A raw image is monochrome, color is something that has to be derived from it.

I believe chroma subsampling is derived from RGB channels, so I assume it goes RAW --> (debayer) --> RGB --> Y'CbCr.  But I don't know the particulars of the resolution scaling since a Bayer pattern is only filtered 50% green, 25% red, 25% blue.  In other words, if it is a 4K bayer sensor, is raw first converted to temporarily to uncompressed 4K per color channel before anything else is done?  I suspect that's not necessary... does the 4K raw conversion first create 2K green, 1K blue, and 1K red?  I don't know.

[Saikat Chattopadhyay]

Although I am from Electronics Background but I am not sure about my answer: 

A digital sensor might not measure any wave property i.e. wavelength or frequency . Tools like Spectroradiometers can but they cost more than mid range cine cams. Sensor has photosites(in short diodes) and 2,4 or more photosites make a pixel. Photosites convert photon energy (light energy) to electric and voltage is measured in sensor . That measured voltage is sent to ADC and other components . 

RAW data doesn't have any chroma subsampling and workflow is same as David mentioned because there is no chroma component before bayer pattern. However, RAW can have compression just to save space.

In Y'CbCr/YUV the CbCr components are 'Chroma Difference Blue' and 'Chroma Difference Red' ,which actually means how much they differ from the green component while keeping the luma same. However,I don't have any idea about how much chroma component is there in luma or how it is calculated ,I just know in subsampling the luma component is kept aside and YUV Broadcast model was first coined to keep BW and color TV models running at same time.

Hope this helps,corrections will be appreciated.

[Phil Rhodes]

On 3/7/2021 at 8:42 PM, David Mullen ASC said:

With raw recording, there are no red, green, and blue channels yet -- all that is created in debayering the mosaic later.  All there is really is in a raw file is luminance information per photosite.  A raw image is monochrome, color is something that has to be derived from it.

I believe chroma subsampling is derived from RGB channels, so I assume it goes RAW --> (debayer) --> RGB --> Y'CbCr.  But I don't know the particulars of the resolution scaling since a Bayer pattern is only filtered 50% green, 25% red, 25% blue.  In other words, if it is a 4K bayer sensor, is raw first converted to temporarily to uncompressed 4K per color channel before anything else is done?  I suspect that's not necessary... does the 4K raw conversion first create 2K green, 1K blue, and 1K red?  I don't know.

I guess it's possible someone might get a bit clever and try to go more directly from raw sensor data to YUV if that's what was required, but my suspicion would be that it's invariably demosaiced to cosited RGB first, then subsampled. Otherwise you'd end up with all sorts of complicated problems to solve around ensuring the subsampled pixels are cosited properly, and of course some cameras support RGB formats.