HTP (highlight tone priority) and DUAL_ISO? OK? No-no?

[l_d_allan]

My understanding is that DUAL_ISO works best with the "base ISO" at 100, and the "recovery ISO" at 200 to 1600.

My observation is that using HTP can provide about a stop better dynamic range in normal (non-ML, non-DUAL_ISO) images.

What about using HTP (highlight tone priority) and ISO 200 as the "base"?

OK?

Flawed?

[gotar]

My understanding is that DUAL_ISO works best with the "base ISO" at 100

No - "base" ISO should be base ISO, whatever value this is for specific camera (not necessarily 100). I thought the purpose was to get minimum noise from analog pre-A/D amplifier, however I've read that Canons have high post-ISO noise, so indeed - why lowest ISO?
The purpose of "recovery" ISO is to pull everything possible out of shadows. You won't get any more DR if that was you were referring to, that's simple physics, everything is about lowering noise.

[Audionut]

HTP only effects JPG/H.264 data.  Turning it on for raw simply raises the minimum ISO to 200.

I thought the purpose was to get minimum noise from analog pre-A/D amplifier, however I've read that Canons have high post-ISO noise, so indeed - why lowest ISO?

The purpose is to get extended DR from a single frame.

Why lowest ISO?

The maximum amount of photons (signal) capable of being captured, can only be captured at the cameras base ISO, where no digital manipulation and/or gain is occurring.
The noise from the ADC only affects the shadow detail where it's percentage vs captured light is high (SNR).

Shot noise (the other significant contributor to total noise) is a square root of the number of photons.  Excluding all other sources of noise, if we observe 2 different exposure settings,

f/2.8 - 1/1600s - ISO 100
f/2.8 - 1/100s - ISO 1600

Looking at the 2 captures by a sensel,

1600 electrons
100 electrons

Shot noise being the square root of the number of photons, so the SNR is,

1600/40 = 40:1
100/10 = 10:1

So the ISO 100 shot where the actual exposure (to photons (light)) was 16x longer, has a SNR ratio 4x greater then the ISO 1600 shot. 

The gain (ISO 1600) on the 1/100s exposure hasn't effected the SNR of the shot noise.

100x16/10x16 = 10:1

In summary, less photons (light), lower SNR.

The DR of the camera is limited solely by the cameras electronics.  ie:  The ability to capture light and where the noise from the cameras electronics is greater then the amount of photons captured.  In Canon cameras, this is the noisy ADC.  Here, analog gain before the noisy ADC helps to reduce the read noise.

We can observe the reduced read noise with increased ISO from the data from DxO.  Here for a Canon 5D Mark III,


http://www.sensorgen.info/CanonEOS_5D_MkIII.html

Code: [Select]

ISO Measured ISO Read Noise (e-) Saturation (e-) DR (stops)
100 80         33.1                 67531                 11.0
200 160         18.2                 35189                 10.9
400 323         10.6                 18273                 10.7
800 641         6.1                  9055                 10.5
1600 1280         3.9                  4380                 10.1
3200 2518         3.1                  2179             9.5
6400 5179         2.9                  1079                  8.5
12800 10084         2.4                   526                  7.8
25600 18589         2.9                   308                  6.7
51200 37032         3.1                   166                  5.7
102400 77274         4.4                   123                  4.8
The saturation numbers in the table above (for ISOs above 100) are an estimation of the amount of photons able to be captured, before the gain (ISO) overloads the ADC.

@ ISO 100 there is 11 stops of DR, limited by the the amount of photons able to be captured before overexposure (saturation), and (more importantly) the read noise from the ADC.  @ ISO 200, the DR should be reduced by (all but) 1 stop due to the limited photon capturing ability from gain overloading the ADC.  However, we can see that the DR reduced by only 0.1 stops.  The analog gain (ISO) was useful in reducing the read noise to a point where it offset the limited photon capturing ability.

What is not shown in the data above is the level of shot noise that was described above.  Increasing ISO does not reduce shot noise, it merely applies gain to the already captured exposure.
Where we use ISO to boost the signal level before the ADC, we haven't increased the amount of photons captured, so we do not increase the SNR of the shot noise.  In general, the SNR of the shot noise is lower (more noise) in higher ISO shots, as we use higher ISOs to correctly render lower levels of light.

So we can see how light plays a very important part in photography, not only from a creative standpoint, but in a more accurate rendition of the scene (more light/less noise).


Back on topic.
@ ISO 400 we can see the overall DR has reduced by 0.3 stops as compared to ISO 100.  Here, the reduction in read noise (from the higher ISO) hasn't outweighed the reduced number of photons (able to be) captured as well as @ ISO 200, and we can see that this trend continues with increasing ISO.  @ ISO 1600, the overall DR has reduced by 0.9 stops.

If we observe the data from a base ISO standpoint of ISO 800, we can see that if we use an ISO just 2 stops greater @ ISO 3200, the overall DR is reduced by 1 full stop.  We gain 2 stops from the 2 stop ISO increase, but we lose 1 stop from the reduced DR at that higher ISO.

Base ISO 100 + recovery ISO 800 = net gain 2.5 stops.
Base ISO 800 + recovery ISO 6400 = net gain 1 stop.

And since the base ISO controls the highlight data, we want to use the lowest ISO possible, not only to ensure the maximum amount of light is captured (higher shot noise SNR), but to ensure we get the maximum gain (benefit) from the recovery ISO.

Where we use the recovery ISO in dual ISO, we do not gain or lose SNR of the shot noise in the shadows, since that is controlled directly by exposure (shutter/aperture), which is controlled by the need to capture highlight data.  But we gain reduced read noise in the shadow data from the increased ISO used to sample these areas.

[gotar]

The purpose is to get extended DR from a single frame.

Why lowest ISO?

The maximum amount of photons (signal) capable of being captured, can only be captured at the cameras base ISO, where no digital manipulation and/or gain is occurring.

That happens only when one can do ETTR - but when exposure is somehow limited (camera in hand or dynamic scene limits the shutter speed and long DoF limits aperture) there might be plenty of space on the right side of histogram. That space does not extend DR, there's simply nothing that can use sensor maximum - is it? So the question is: is that space wasted? There are easy available charts of readnoise vs ISO (in constant exposure) - increasing ISO usually gets lower noise, especially for Canons. This technique is called ITTR (ISO to the right).

The noise from the ADC only affects the shadow detail where it's percentage vs captured light is high (SNR).

Exactly. So you should get as bright image as possible without clipping == ETTR. But when ETTR is not enough or can't be used, it's better to boost signal before noise block, and then attenuate it. Any additional noise from pre-ADC amplifier would be attenuated as well then, so resulting S/N should be lower.

Shot noise (the other significant contributor to total noise) is a square root of the number of photons.

Nope, it's the signal to noise ratio - we want this value to be as high as possible. No amplifiers/attenuators improve S/N, they only introduce their own. So we need as much signal as possible at the very beginning - that's the ETTR. What happens if we pass through the sensor and reach ADC? We get additional noise, independent on amplification value (in dB) - no difference between low and high ISO here. It also boosts sensor noise, but this doesn't matter, so is the signal. Next we go into ADC itself and the rest of the electronics - just to get even more noise. So what's the difference? None, unless read noise is worse for low signal, like it happened to be with Canon.

Excluding all other sources of noise, if we observe 2 different exposure settings,

f/2.8 - 1/1600s - ISO 100
f/2.8 - 1/100s - ISO 1600
[...]
Shot noise being the square root of the number of photons, so the SNR is,

1600/40 = 40:1
100/10 = 10:1

So the ISO 100 shot where the actual exposure (to photons (light)) was 16x longer, has a SNR ratio 4x greater then the ISO 1600 shot.

Assuming one can do 16 times longer exposure (i.e. ETTR). Consinder f/8+ aperture and 1/50 s - there's no way to catch more photons, the only thing you can do is to lower the impact of all the noise blocks inside camera. This requiress boosting signal ASAP - and hopefully ISO amplifiers introduce linear noise.

In summary, less photons (light), lower SNR.

The DR of the camera is limited solely by the cameras electronics.  ie:  The ability to capture light and where the noise from the cameras electronics is greater then the amount of photons captured.  In Canon cameras, this is the noisy ADC.  Here, analog gain before the noisy ADC helps to reduce the read noise.

We can observe the reduced read noise with increased ISO from the data from DxO.  Here for a Canon 5D Mark III,
[...]
The saturation numbers in the table above (for ISOs above 100) are an estimation of the amount of photons able to be captured, before the gain (ISO) overloads the ADC.

@ ISO 100 there is 11 stops of DR, limited by the the amount of photons able to be captured before overexposure (saturation), and (more importantly) the read noise from the ADC.  @ ISO 200, the DR should be reduced by (all but) 1 stop due to the limited photon capturing ability from gain overloading the ADC.  However, we can see that the DR reduced by only 0.1 stops.  The analog gain (ISO) was useful in reducing the read noise to a point where it offset the limited photon capturing ability.

That's exactly what I'm talking about.

Increasing ISO does not reduce shot noise, it merely applies gain to the already captured exposure.
Where we use ISO to boost the signal level before the ADC, we haven't increased the amount of photons captured, so we do not increase the SNR of the shot noise.

Increasing ISO reduces read noise (not sensor's of course).

In general, the SNR of the shot noise is lower (more noise) in higher ISO shots, as we use higher ISOs to correctly render lower levels of light.

Not in general, but comparing to lower ISO shot with longer exposure. Assuming anyone can do this is definitely not general.

And since the base ISO controls the highlight data, we want to use the lowest ISO possible, not only to ensure the maximum amount of light is captured

Only as long as maximum amount of light is not limited anyway by other factors. These factors might be exposure itself or low light condition...

OK, having written this all I got myself the answer... When there are no conditions to overexpose picture, one doesn't need dual ISO at all, because some single ISO value covers entire DR available. That's why dual ISO uses base ISO.

...or maybe I'm not. If the base ISO at maximum exposure lefts for example - 2 stops on the right available, this doesn't mean there's nothing in the shadows for ISO 1600 to fetch. Why shouldn't I use 400/1600 then?

[Audionut]

Ok!  So the original question was, "why use the lowest ISO".  I'm pretty sure I explained the reasons why you should always use the lowest ISO in a dual ISO situation, but you now feel the need to pick apart my post!

That happens only when one can do ETTR

No, it's a constant.  Increasing ISO will not allow you to capture more photons.  The maximum amount of photons that can be rendered correctly will always be at the base ISO (no gain) of the camera.

Nope, it's the signal to noise ratio

http://en.wikipedia.org/wiki/Shot_noise#Optics

My own bold for emphasis.

In optics, shot noise describes the fluctuations of the number of photons detected (or simply counted in the abstract) due to their occurrence independent of each other. This is therefore another consequence of discretization, in this case of the energy in the electromagnetic field in terms of photons. In the case of photon detection, the relevant process is the random conversion of photons into photo-electrons for instance, thus leading to a larger effective shot noise level when using a detector with a quantum efficiency below unity. Only in an exotic squeezed coherent state can the number of photons measured per unit time have fluctuations smaller than the square root of the expected number of photons counted in that period of time. Of course there are other mechanisms of noise in optical signals which often dwarf the contribution of shot noise. When these are absent, however, optical detection is said to be "photon noise limited" as only the shot noise (also known as "quantum noise" or "photon noise" in this context) remains.

----------------------

Assuming one can do 16 times longer exposure (i.e. ETTR).

Assuming nothing.

Excluding all other sources of noise

The SNR of f/2.8 - 1/1600s - ISO 100 is 4x greater then f/2.8 - 1/100s - ISO 1600.

While the exposure of the lower ISO is 16x longer then the exposure of the higher ISO (in this case), the rendered brightness is the same.

Increasing ISO reduces read noise

Yes, I did mention something along those lines.

Here, analog gain before the noisy ADC helps to reduce the read noise.

We can observe the reduced read noise with increased ISO from the data from DxO.

And since you want to be picky, increasing ISO does not reduce read noise.  Increasing analog ISO reduces read noise.  Digital gain will not reduce read noise.

Not in general, but comparing to lower ISO shot with longer exposure. Assuming anyone can do this is definitely not general.

Do you generally use a higher ISO to correctly render low light levels in cameras with noisy ADC's?  Yes?
Then you generally have a lower SNR of shot noise with higher ISO.

And while it is true that the SNR of shot noise of a higher ISO is lower when compared to a lower ISO with a longer exposure (which I explained in my OP btw), if you want to pick apart posts in the manner that you are, this is going to become a long winded thread very quickly!

Only as long as maximum amount of light is not limited anyway by other factors.

Yes, but again, in context of your original question, "why use lowest ISO", you left out the important part.

but to ensure we get the maximum gain (benefit) from the recovery ISO.

ETTR, while a well advised practice, has no effect on why you would want to use the lowest ISO as the base ISO.  Your original question!

[gotar]

Ok!  So the original question was, "why use the lowest ISO".  I'm pretty sure I explained the reasons why you should always use the lowest ISO in a dual ISO situation, but you now feel the need to pick apart my post!

Sorry, but you didn't explained why the lowest ISO should be used always. OK, maybe it's me who extended the question meanwhile, I should have asked: "should this be always lowest ISO"?, but now I see you confirm this. I don't. Please, explain me why should I use 100/1600 instead 400/1600 having 2 stops on the right of raw histogram empty? No matter what ISO would be applied, nothing clips (and by 'empty' and 'nothing' I accept the 0.3 stop you've mentioned, from some ‰ of the frame  - or maybe it's all about this?).

No, it's a constant.  Increasing ISO will not allow you to capture more photons.  The maximum amount of photons that can be rendered correctly will always be at the base ISO (no gain) of the camera.

Of course I won't catch more photons in higher ISO, that's obvious. And that base ISO gives maximum photon "spectrum" per ADC quantization in general. But general rules not always are true in specific situations - I can amplify the signal with inpunity as long as it won't clip in next blocks (still remembering the 0.1 stop lost in ISO 200, if that's the case then fine).

Shot noise (the other significant contributor to total noise) is a square root of the number of photons.

Nope, it's the signal to noise ratio

http://en.wikipedia.org/wiki/Shot_noise#Optics

From the same article: SNR=sqrt(N). That is the signal to noise ratio, not the noise. Don't confuse them, noise is the 'N' part of the equation, while sqrt(N) is also stdev of noise shot (but not noise itself).

The SNR of f/2.8 - 1/1600s - ISO 100 is 4x greater then f/2.8 - 1/100s - ISO 1600.

While the exposure of the lower ISO is 16x longer then the exposure of the higher ISO (in this case), the rendered brightness is the same.

And SNR of f/8 1/500 ISO 100 is the same as f/8 1/500 ISO 400. So - when one cannot increase aperture (because he simply needs that DoF) and cannot increase time (dynamic scene, i.e. no ETTR or any other mean to increase exposure except for extra lightning), what's the benefit of using ISO 100 over ISO 400 (except these 0.3)?

And since you want to be picky, increasing ISO does not reduce read noise.  Increasing analog ISO reduces read noise.  Digital gain will not reduce read noise.

First of all I'm not picky - I just want to be as precise as possible, as this forum is being read by both professionals and amateurs and I wouldn't like to write nonsense nor mislead someone who's not going to see the difference between S/N and noise. Second - I was clearly talking about analog amplification, don't know where you get the digital mumbo-jumbo from, but it's also obvious, that one cannot improve signal in postprocessing (which "digital ISO" is).

Do you generally use a higher ISO to correctly render low light levels in cameras with noisy ADC's?  Yes?

Not generally but sometimes - it's not me who states about something (lowest ISO) being the best always.

So, not to entangle indeed this thread to the point of misunderstanding: do you see any specific situations when 400/1600 would not harm shot noise, lowering read noise? Or would you recommend using ISO 100 always? Disregarding the 0.1-0.3 DR loss.

[l_d_allan]

HTP only effects JPG/H.264 data.  Turning it on for raw simply raises the minimum ISO to 200.

That's not my understanding of what HTP does, and not consistent with my observations when using HTP with raw still photos. I'm pretty sure I'm seeing a larger raw dynamic range using HTP-200 than using non-HTP with iso-100 or iso-200.

Unlike settings like WB, picture-styles, etc. that only influence .jpg's, my impression is that HTP actually results in changes in how raw values are recorded from the sensor. It changes the "tone mapping" of the sensor.

[Audionut]

That's not my understanding of what HTP does, and not consistent with my observations when using HTP with raw still photos. I'm pretty sure I'm seeing a larger raw dynamic range using HTP-200 than using non-HTP with iso-100 or iso-200.

Unlike settings like WB, picture-styles, etc. that only influence .jpg's, my impression is that HTP actually results in changes in how raw values are recorded from the sensor. It changes the "tone mapping" of the sensor.

ISO 100


ISO 200 - HTP OFF


ISO 200 - HTP ON


ISO 200 - HTP ON - Metered 1 stop over


note:  except for ISO 200 - HTP ON at the correct meter reading, all exposures are within 1/15 stop before overexposure.


HTP underexposes by 1 stop for the same Canon meter reading.  This highlight data is then mapped out in JPG/H.264 to give the impression of a smoother roll off.  Your raw processor might apply this same roll off when it "sees" the HTP flag.

HTP does not increase DR.  If you rely on the Canon meter when taking the exposure, you effectively reduce the DR by 1 stop.  Not only that, but you move the entire exposure down 1 stop in tonal range.  For instance, your midtone detail is now 1 stop underexposed also.


ISO 200 HTP OFF


ISO 200 HTP ON


A far better solution is to ETTR where possible and map the data to the left in your raw processor.

[Audionut]

Sorry, but you didn't explained why the lowest ISO should be used always.

My mistake.  I did advise in my original OP that we want to use the lowest ISO possible.  Possible, probably isn't the best choice of definition either.
How about, you should use the lowest ISO based on your knowledge of the pros vs cons of ISO itself.  This then (hopefully) leads the reader to seek a better understanding of ISO as it applies to digital photography when making their determinations of the best ISO to use for dual ISO, for their particular circumstance.

But general rules not always are true in specific situations - I can amplify the signal with inpunity as long as it won't clip in next blocks

You are free to do with the data however you see fit.  Did you increase the amount of photos captured by amplifying the signal!  Did you increase the cameras ability to capture photons by amplifying the signal!

From the same article: SNR=sqrt(N). That is the signal to noise ratio, not the noise. Don't confuse them, noise is the 'N' part of the equation, while sqrt(N) is also stdev of noise shot (but not noise itself).

Noise is not the 'N' part of the equation at all.  N is the number of events.

http://en.wikipedia.org/wiki/Shot_noise#Origin
My bold for emphasis.

For large numbers the Poisson distribution approaches a normal distribution, typically making shot noise in actual observations indistinguishable from true Gaussian noise except when the elementary events (photons, electrons, etc.) are so few that they are individually observed. Since the standard deviation of shot noise is equal to the square root of the average number of events N, the signal-to-noise ratio (SNR) is given by:

First of all I'm not picky - I just want to be as precise as possible, as this forum is being read by both professionals and amateurs and I wouldn't like to write nonsense nor mislead someone who's not going to see the difference between S/N and noise. Second - I was clearly talking about analog amplification, don't know where you get the digital mumbo-jumbo from, but it's also obvious, that one cannot improve signal in postprocessing (which "digital ISO" is).

You were clearly talking about ISO, otherwise known (in digital camera systems) as gain.  And there is nothing that is obvious to everyone!
There is analog gain, where the signal is boosted by increasing the voltage of the signal from the sensor before the ADC.  And there is digital gain, where the signal is boosted after the ADC in the digital domain.

http://en.wikipedia.org/wiki/Film_speed#Digital_camera_ISO_speed_and_exposure_index
My bold for emphasis.

For digital photo cameras ("digital still cameras"), an exposure index (EI) rating—commonly called ISO setting—is specified by the manufacturer such that the sRGB image files produced by the camera will have a lightness similar to what would be obtained with film of the same EI rating at the same exposure. The usual design is that the camera's parameters for interpreting the sensor data values into sRGB values are fixed, and a number of different EI choices are accommodated by varying the sensor's signal gain in the analog realm, prior to conversion to digital. Some camera designs provide at least some EI choices by adjusting the sensor's signal gain in the digital realm. A few camera designs also provide EI adjustment through a choice of lightness parameters for the interpretation of sensor data values into sRGB; this variation allows different tradeoffs between the range of highlights that can be captured and the amount of noise introduced into the shadow areas of the photo.

As for your other questions.  I have better things to do then spend my time trying to explain technique, to then have those responses dissected in a negative way with supporting techniques/theories that hold no relevance to the original statements.

If you want to continue to discuss matters with me, I am more then happy for you to do so, provided you support your assertions with accurate data.  Otherwise, I would ask that you discuss matters in a manner that doesn't belie your true understanding of the topics at hand.

[gotar]

Did you increase the amount of photos captured by amplifying the signal!  Did you increase the cameras ability to capture photons by amplifying the signal!

No, I didn't and no, I didn't. What I did however is increase camera ability to process captured signal.

Noise is not the 'N' part of the equation at all.  N is the number of events.

Actually there is N on the left side of the equation (S/N). Apparently this wiki article was written by someone, who knows photo, but is not fluent in signal processing. Note, that only 3 non-en versions did copy this buggy equation (using the same variable for different things). Just waiting for sth like sin(x)/cos(x)=in/co:)

As for your other questions.  I have better things to do then spend my time trying to explain technique, to then have those responses dissected in a negative way with supporting techniques/theories that hold no relevance to the original statements.

I'm sorry if that's what you feel I did - I can assure you, that any of your statements I get out of context and changed it meaning was not intentional (I'm not even sure where this happened, even after reviewing the thread once again). Nonetheless, please accept my apologies, I must put this on trouble understanding/expressing myself in foreign language.

If you want to continue to discuss matters with me, I am more then happy for you to do so, provided you support your assertions with accurate data.  Otherwise, I would ask that you discuss matters in a manner that doesn't belie your true understanding of the topics at hand.

OK, I've found examples I've seen once: http://www.luminous-landscape.com/forum/index.php?topic=56906.0 http://www.luminous-landscape.com/forum/index.php?topic=78677.0 explaining the effect I'm talking about.

[a1ex]

TLDR; HTP is digital, no effect on dual ISO.

That is, when shooting RAW (CR2), ISO 200 with HTP is the same as regular ISO 100.

[Audionut]

What I did however is increase camera ability to process captured signal.

Which is totally unrelated to the original statement.

The maximum amount of photons (signal) capable of being captured, can only be captured at the cameras base ISO, where no digital manipulation and/or gain is occurring.

Somehow this become a discussion on ETTR, again, unrelated to the original statement.

I must put this on trouble understanding/expressing myself in foreign language.

My combined mix of various forms of English is bound to be difficult to translate.  My apologies for any misunderstanding as a result.

explaining the effect I'm talking about.

We have a discussion thread about ETTR here.

[Audionut]

Actually there is N on the left side of the equation (S/N). Apparently this wiki article was written by someone, who knows photo, but is not fluent in signal processing. Note, that only 3 non-en versions did copy this buggy equation (using the same variable for different things). Just waiting for sth like sin(x)/cos(x)=in/co:)

http://www.clarkvision.com/articles/digital.sensor.performance.summary/index.html#intro

In the physics of photon counting, the noise in the signal is equal to the square root of the number of photons counted because photon arrival times are random. The reason for this dependence is Poisson Statistics (Wikipedia has an excellent article on Poisson statistics). For example Table 1 shows the signal-to-noise ratio when detecting different numbers of photons.

[gotar]

Which is totally unrelated to the original statement.

Indeed... the purpose of base ISO in dual ISO is to maximize the DR, so that's the best to start with and remains default. l_d_allan asked about defaulting to ISO 200 (with HTP) and this is not valid in general. Whether that might be the case is the subject of ETTR/ITTR for particular scene, so I wandered too far having this subject began with so simple question.


Everything below is related to noise itself (some physics straightening bad explanations from the web), I was about to put this into separate post to be moved to some more appropriate place, but I don think this thread would be continued anyway, so here it goes.

http://www.clarkvision.com/articles/digital.sensor.performance.summary/index.html#intro

In the physics of photon counting, the noise in the signal is equal to the square root of the number of photons counted because photon arrival times are random

This is just a mental shortcut and should never be given as any kind of definition - it is the "standard deviation of shot noise equal to the square root of the average number of events N" as quoted from 'Shot noise' wiki article you've given (just before the wrong equation there). Proper equations are here: http://en.wikipedia.org/wiki/Signal-to-noise_ratio_(imaging%29 - note the definitions of signal and noise itself, they are much more complex than 'count photons' (hopefully they easy reduce when we need S divided by N). And the standard deviation have it's simple formula thanks to Poisson distribution. To conclude, having simple final formula for signal/noise ratio=n/sqrt(n) doesn't mean one can take apart numerator and denominator and say "signal equals number of photons and noise equals the square root of this number".

Secondly - the noise on the sensor is not related to some random photon arrival times, but time-dependent current fluctuations having 2 main sources: thermal noise and electrical charge quantization (this is the square-root contrubution, essential in optical frequencies - shot noise). There are also sensor area irregularities (semiconductor defects, inter-cell differences) and a bunch of quantum effects involved. That's why you get less noise from cooler sensors (thermal noise reduced) or larger cells (by either enlarging entire sensor or having less Mpix - more photons catched in a cell, less relative shot noise). This has nothing to do with any photons bouncing randomly like balls until they splash on sensor, but rather with different balls (electrons) having strictly specified capacity, that must be fully loaded by the first ones, before they proceed.

[Audionut]

http://www.princeton.edu/~achaney/tmve/wiki100k/docs/Shot_noise.html

It is known to everyone that in a statistical experiment such as tossing a fair coin and counting the occurrences of heads and tails, the numbers of heads and tails after a great many throws will differ by only a tiny percentage, while after only few throws outcomes with a significant excess of heads over tails or vice versa are common; if an experiment with a few throws is repeated over and over, the outcomes will fluctuate a lot. (It can be proved that the relative fluctuations reduce as the square root of the number of throws, a result valid for all statistical fluctuations, including shot noise.)

Shot noise exists because phenomena such as light and electric current consist of the movement of discrete, quantized 'packets'. Consider light—a stream of discrete photons—coming out of a laser pointer and hitting a wall to create a visible spot. The fundamental physical processes that govern light emission are such that these photons are emitted from the laser at random times; but the many billions of photons needed to create a spot are so many that the brightness, the number of photons per unit time, varies only infinitesimally with time. However, if the laser brightness is reduced until only a handful of photons hit the wall every second, the relative fluctuations in number of photons, i.e. brightness, will be significant, just as when tossing a coin a few times. These fluctuations are shot noise.

Noise is the standard deviation.  That's why it has become noise in the first place, because it has that deviation.  If it didn't deviate, we wouldn't be having this discussion.

As a photographer I only need to know that the shot noise is a square root of the photons collected.  And hence, the SNR is also this square root.  And hence, more light captured, more SNR.

I am really not interested in discussing this further when you continue to try and debunk this theory with only your own assertions, that differ from numerous other sources available on the subject.

http://arxiv.org/pdf/cond-mat/9905024.pdf
http://optical-technologies.info/noise-in-photodetectors/

[gotar]

Noise is the standard deviation. That's why it has become noise in the first place, because it has that deviation.  If it didn't deviate, we wouldn't be having this discussion.

I'm sorry to inform you, but you're horribly wrong. None of the 3 sources you've given states this, this only proves you don't know enough physics to understand what you read. Noise is the bogus signal (originating from different sources, I've elaborated the main ones appearing on the sensor), it's deviation is only one of the values that describe it.

As a photographer I only need to know that the shot noise is a square root of the photons collected.  And hence, the SNR is also this square root.  And hence, more light captured, more SNR.

As a photographer you need to know the last one, but not only. Because the previous statements are wrong - as noise of the sensor combines different sources, you should also know that total noise raises with temperature, so you shoud keep your sensor cool.

I am really not interested in discussing this further when you continue to try and debunk this theory with only your own assertions, that differ from numerous other sources available on the subject.

http://arxiv.org/pdf/cond-mat/9905024.pdf
http://optical-technologies.info/noise-in-photodetectors/

None of these differ in any way from what I've said. Apparently you're one of the people that can't see the difference between voltage, current, charge, field and power.

[luxluminis]

Did you ever try to combine HTP with max ETTR using a picture style like Cine Style or UniWB to check blown highlights and while doing so recording a RAW file ? I think this combination produces much more than just an "an uderexposed and afterwards mapped JPG impression". It definetly gives you a lot of headroom for ETTR in the RAW File itself.

[Audionut]

HTP is a one stop digital reduction of raw data.  The simple fact that it's digital means that it delivers no SNR benefit.

Feel free to throw data at me and I'll debunk it.

[luxluminis]

Chuck Westfall, the guy from canon, says the following, and this is also what i have obeserved by using this feature all the time:

"HTP has no effect on the actual dynamic range of the image sensor (You, Audionut say, that it cuts 1 stop). It's just an alternative method of image processing that preserves more highlight detail than Canon's standard processing, without significantly altering midtones or shadows. The effect of HTP is enhanced by Canon's 14-bit A/D converter, which provides finer tonal gradations than the previous 12-bit system. HTP is a Custom Function with a simple on/off setting, and the available range of ISO speed settings is slightly limited when it is on."

"HTP affects RAW data as well as in-camera JPEGs. It is very useful in high-key shooting conditions such as wedding photography and certain kinds of sunsets. "

[DeafEyeJedi]

What exactly does all of this mean, @luxluminis?

Perhaps take a dual ISO shot with HTP on and take another with HTP off and compare them yourself...

Or post them here if you like.

[luxluminis]

I cant provide Dual ISO Shots, ML wont work on my 1DX. I used ML on my 5D, before i sold it last year and never used Dual ISO. I stumbled over this thread because of another forum, where some guys wrote that the ML specialists in this forum thread  are true Gurus when it comes to Canon and ISO but that they would definetly interpret HTP wrong. Thats why i came here. I use HTP always. If it enhaces RAW Files in Single ISO mode, it will probably also do it in Dual ISO. But it is interesting that the opinions about what it can do vary this much. I dont care. I use it practically instead of judging it theoretically...

In the Manual of the 1 DX you can actually read also about an expansion of the Dynamic Range:

"Highlight details are improved. The dynamic range is expanded from the standard 18% gray to bright highlights. The gradation between the grays and highlights becomes smoother."

But what i can do is to provide a comparison of several RAW files shot with the 1 DX. You can see clearly in these files that the Highlight performance is MUCH better with HTP while you have the same amount of details in the shadows. Even at ISO 12.800 which looks pretty clean with my way of ETTRing it. So Audionuts statement about HTP cutting 1 Stop of DR must be wrong, no matter what RAW Digger data might show.

I will open the RAW files in RPP64, because i dont have Raw Digger. RPP also shows the true RAW exposure and not the bullshit conventional Raw Converters show. Fact is, that i can produce a totally usable ISO 12.800 file which looks almost as fine as ISO 1.250. Without applying any Noise Reduction. Compared to the file that the normal exposure of the 1 DX would deliver at ISO 12.800 my file looks stellar clean.

[dmilligan]

So Audionuts statement about HTP cutting 1 Stop of DR must be wrong

I see no where that Audionut said that HTP reduces dynamic range by one stop, you are completely misreading this. What HTP does is to reduce ISO by one stop, thus preserving more highlight detail. DR is unaffected.

they would definetly interpret HTP wrong. Thats why i came here.

[luxluminis]

If you cant see it, here is a little reminder... Audionut said:

"HTP does not increase DR.  If you rely on the Canon meter when taking the exposure, you effectively reduce the DR by 1 stop.  Not only that, but you move the entire exposure down 1 stop in tonal range.  For instance, your midtone detail is now 1 stop underexposed also."

Before perfoming the joker, better read all the comments in detail.

Also, you should try to understand my point and what i said: I recommended a combination of HTP and maximum ETTR. With HTP activated ETTR works much more effective. I have the files to show what i mean. The way i do it, there is no "reduction of the DR by one stop." as Audionut says. This is simply not true.

[dmilligan]

Uh, I still don't see anywhere Audionut says that HTP reduces dynamic range, even in your quote. Saying that something does not increase DR is not the same as saying it decreases it. Saying that thing B effectively reduces DR is not the same as saying thing A reduces DR. But I'll let him speak for himself.

Also, you should try to understand my point and what i said

If you care to use maths, data, and solid reasoning to make a point, I will try to understand it. Otherwise, I could care less about your opinion, and your quest to correct all of us mistaken ML users.

[Audionut]

A sensor, the bit that actually captures light, has a fixed DR .  Fixed, as in doesn't change.  ISO, HTP, ALO, or any other fancy acronym used by makers doesn't change this.
The highlight end of DR is determined solely by the amount of photons the sensor can capture, which is determined by the size of the pixel wells on the sensor.  This is a mechanical thing, so no amount of code or electronic tricky changes this fact.

The shadow end of the DR measurement is determined by the amount of noise that the camera (as a whole) produces.  The sensor could capture thirteen gazillion stops of DR, but it makes no difference if a piece of electronics in the camera produces noise which limits this shadow end of the DR.  This happens in all cameras that I am aware of, but in Canon cameras, their downstream (from the sensor) electronics typically limits DR to around eleven stops.

But anyway, lets get back to the highlight end.  The entire premise of ETTR is to ensure that we are capturing the maximum amount of photons that the sensor can capture.  Because light itself contains noise..................Secondary to that, SNR theory states, in a nutshell, that the more signal you push through a device, the more faithful the output.  So not only do we want to ensure the maximum amount of photons hitting the sensor, but we also want to ensure that even if we can't capture the maximum amount of photons, whatever we did capture, lets boost that signal before sending it through the rest of the electronic chain.  This is what ISO control on a digital camera does.  Boosts the signal from the sensor before sending it (the signal) through the other electronic components.

But the main reason is because the light itself contains noise.  The noise in light is equal to the square root of the signal captured.  So if we capture four photons, the noise value equals two.  If we capture ten thousand photons, the noise value equals one hundred.  So we can see that the stronger the light source, the more noise it actually has, but luckily for us, the amount of (usable) signal increases greater then the amount of noise.  So even though ten thousand photons has a noise value of one hundred, compared to four photons which only has a noise value of two, we can see that when we capture four photons, half of that light is actually noise.  Whereas when we can capture ten thousand photons, only one percent of that light is noise, the other ninety nine percent being usable signal.

But I still digress.

The main point as far as HTP is concerned is that the sensor has a fixed amount of light that it can capture.  Remember, this is a mechanical limitation.  HTP will not, can not, and does not change this fact (period!).

So, you're probably asking yourself, how the hell do I seem to gather more highlight detail when I have HTP activated.  Simple, your camera meter is lying to you.

HTP does not increase DR.  If you rely on the Canon meter when taking the exposure, you effectively reduce the DR by 1 stop.  Not only that, but you move the entire exposure down 1 stop in tonal range.  For instance, your midtone detail is now 1 stop underexposed also.

Why does this happen?  Because your camera meter is lying to you.  When you enable HTP, the camera makes the meter read one stop higher then it otherwise would.  So with HTP enabled, 0EV on the camera meter is actually -1EV.  +3EV is actually +2EV, -2EV is actually -3EV.

So the reason why it appears you can capture more highlight detail is not because you are magically increasing the amount of light that the sensor can capture, but because your camera meter is lying to you, and fooling you into adjusting your exposure settings to compensate.



So here are the kickers.

Did you really need that extra highlight detail?  If you did, then just expose for it.  You don't need HTP to do that, HTP just makes it harder to determine the actual exposure. 
If you didn't actually need that extra highlight detail, and just thought you were getting it for free, you've effectively pushed all of the wanted detail down one stop, and hence, increased the noise level of the entire signal.  Everything!  Highlights, midtones and shadows. 

It hasn't reduced the DR of the camera per se, but the noise level of the signal is the same as if the camera had one stop less DR.  Why?  Because light contains noise, and you've exposed it one stop lower.

Hope this helps.