Digital BW, The Print

----- Original Message -----

From: "Austin Franklin" <darkroom@...>
To: <DigitalBlackandWhiteThePrint@yahoogroups.com>
Sent: Saturday, October 12, 2002 6:36 AM
Subject: RE: [Digital BW] Re: 'combed' histograms in 16 bit ?


(snip earlier)

> > So as an example let's say that a raw scan of a particular negative from
a
> > 12-bit scanner produces 2500 levels with the highest falling on
> > say 3500 and
> > the lowest on 1000. I assume that in 12-bit space these 2500
> > levels are all
> > adjacent without any levels that have no pixels (assuming it is a
> > continuous
> > tone negative.)
>
> Exactly correct again, except that there is no such thing as 12 bit space,
> 12 bit space IS 16 bit space.

Austin,

I see, it is not coming from a 12-bit space but rather is a stream of data
with values that will fall between 0 and 2^12-1.
>
> > Now when this is mapped to 16-bit space let if
> > each level is
> > multiplied by 16 then the highest value would be on 56,000 and
> > the lowest on
> > 16,000 giving a spread of 40,000 in 16-bit space.
>
> Nope...it is simply the exact same values, except there are four more high
> bits, the data is exactly the same.
>
> > Or is the highest value mapped to 56,000 and the lowest to 56,000 - 2500
=
> > 53,500?
>
> No...again, the data values are EXACTLY the same, 1000-3500, all
contiguous.
>
> > This second case seems unlikely from the raw scans I have
> > seen from
> > 12-bit scanners. 2500 adjacent levels in 16-bit space would be extremely
> > narrow when you looked at a histogram.
>
> Yes, it is very narrow.

Okay understood. I guess my misunderstand comes from the fact that the
histograms I see of raw scabs, while narrow, are often considerably larger
the 6% of the 16-bit range I would have expected. However since the
histograms are 8-bit I guess we cannot take them literally.

So we have 2500 adjacent levels in 16-bit space. Obivously then if you do
any gamma, curves or similar adjustment without first adjusting the end
points outward you will lose or damage the data. pixels will be moved from
one level to another level and added to the pixels already at that level
resulting in spikes and gaps in the data reducing the number of total
levels.

If you spread or remap the end points on your data set first, you create
room to move the pixels from one level to another without adding them to a
level that already contains pixels or at least reduce the certainty to a
possibility. With our example of 2500 adjecent level from 1000 to 3500 in
16-bit space let's say we move the 3500 value to 65,000 and the 1000 value
to 10. We still only have pixels assigned to 2500 discrete levels but those
levels are now scattered across a range of 64,000 possible levels and while
moving them around during adjustments the chances of their interferring with
eachother is small. Am I tracking this correctly so far?
>
> > So, if the first case is correct, in 16-bit space are there then 15
levels
> > with not pixels at those values between each of the mapped
> > values? In other
> > words is the highest level in my example now 56,000 and the next lowest
> > value 55,984, then 55,968, etc.?
>
(snip)
> >
> > I suspect it may have been a victim of a phenomenon in Silverfast
> > where you
> > can specify a gamma setting to be applied to the raw scan on the fly
which
> > then of course it is not really a raw scan.
>
> You should not apply ANY tonal curve adjustment to raw data until you
> setpoint it, for obvious reasons!

Believe it or not Silverfast is designed to do just that if you check a
certain option box.
>
> > When you open this
> > "raw", gamma
> > adjusted file directly in PS it initially has a good histogram but after
> > applying Levels the histogram looks mildly combed. No full gaps
> > but it gets
> > rather spiky. Doing a raw scan in Silverfast and not applying the gamma
> > setting seems to avoid this issue.
>
> That is exactly what I would suspect would happen if you apply tonal
changes
> to un-setpointed data.

Well this makes a lot more sense. If you know what is going on the problem
is easy to avoid.
>
(snip)
>
> > So
> > in my example the highest value was 56,000 or 01101101011000000 and in
> > converting it to 8-bit PS drops the last 8 digits of the binary number
> > giving 011011010 or 218.
>
> Sounds right, but I won't check your arithmetic ;-)

Sorry about that. No experience in working with binary. Forgot 16-bit space
is not 2^16 but 0 to (2^16)-1.
>
(snip)
>
> > How difficult would this be for a programmer to create as a plug-in for
> > Photoshop?
>
> Not very, there are thousands of them available, and a plug-in SDK
(Software
> Development Kit) is available free from Adobe.

Well if there are any programers lurking out there who would like to take
this on I will buy a copy!

Thanks for helping me sort this out.

Martin

Hi again, Martin!

> > > So as an example let's say that a raw scan of a particular
> negative from
> a
> > > 12-bit scanner produces 2500 levels with the highest falling on
> > > say 3500 and
> > > the lowest on 1000. I assume that in 12-bit space these 2500
> > > levels are all
> > > adjacent without any levels that have no pixels (assuming it is a
> > > continuous
> > > tone negative.)
> >
> > Exactly correct again, except that there is no such thing as 12
> bit space,
> > 12 bit space IS 16 bit space.
>
> Austin,
>
> I see, it is not coming from a 12-bit space but rather is a stream of data
> with values that will fall between 0 and 2^12-1.

Correct.

> > > Now when this is mapped to 16-bit space let if
> > > each level is
> > > multiplied by 16 then the highest value would be on 56,000 and
> > > the lowest on
> > > 16,000 giving a spread of 40,000 in 16-bit space.
> >
> > Nope...it is simply the exact same values, except there are
> four more high
> > bits, the data is exactly the same.
> >
> > > Or is the highest value mapped to 56,000 and the lowest to
> 56,000 - 2500
> =
> > > 53,500?
> >
> > No...again, the data values are EXACTLY the same, 1000-3500, all
> contiguous.
> >
> > > This second case seems unlikely from the raw scans I have
> > > seen from
> > > 12-bit scanners. 2500 adjacent levels in 16-bit space would
> be extremely
> > > narrow when you looked at a histogram.
> >
> > Yes, it is very narrow.
>
> Okay understood. I guess my misunderstand comes from the fact that the
> histograms I see of raw scans, while narrow, are often considerably larger
> then 6% of the 16-bit range I would have expected. However since the
> histograms are 8-bit I guess we cannot take them literally.

Yeah, that is why I want to understand what on earth is going on with the
raw data...and how it's justified.

A digression:

Let's say a typical film scanner has a dMax of say, 3.6...and it's got a 12
bit A/D...  12 bits can only represent 4096 distinct values...  Now, typical
film scanners we use do not have adjustable front ends, they do have
adjustable exposure time, but that doesn't change the data relativeness that
the sensor sees...they record the data with linear differences in density.

Now, this is a tough one, and I know it doesn't make practical sense, but
bear with me.  B&W film has a density range of say, 2...well, according to
how our scanner works, that's only 100 discernable tones if the scanner
records linearly throughout the entire density range, where each discernable
density step is the same, and that's how most all of them work as far as I
know.

The scanner, when recording, doesn't know what the density range of the film
is, that it is scanning, it has to record the entire range of 3.6 for EVERY
film that it scans.  That assumes that it doesn't have an adjustable front
end, and none of the typical film scanners do...  The density range of the
film comes into place when you set the setpoints, you are telling the image
file what the density limits of the scanned film is.

Now back to our regularly scheduled program ;-)

> So we have 2500 adjacent levels in 16-bit space. Obivously then if you do
> any gamma, curves or similar adjustment without first adjusting the end
> points outward you will lose or damage the data.

Exactly correct, and why you need to set setpoints first, then expand the
setpointed image data over the larger range, to give you gaps between the
data to allow for tonal corrections.

> pixels will be moved from
> one level to another level and added to the pixels already at that level
> resulting in spikes and gaps in the data reducing the number of total
> levels.

Bingo.

> If you spread or remap the end points on your data set first, you create
> room to move the pixels from one level to another without adding them to a
> level that already contains pixels or at least reduce the certainty to a
> possibility.

Correct.

> With our example of 2500 adjecent level from 1000 to 3500 in
> 16-bit space let's say we move the 3500 value to 65,000 and the 1000 value
> to 10. We still only have pixels assigned to 2500 discrete levels
> but those
> levels are now scattered across a range of 64,000 possible levels
> and while
> moving them around during adjustments the chances of their
> interferring with
> eachother is small. Am I tracking this correctly so far?

Absolutely.  That's exactly what happens.

> > > So, if the first case is correct, in 16-bit space are there then 15
> levels
> > > with not pixels at those values between each of the mapped
> > > values? In other
> > > words is the highest level in my example now 56,000 and the
> next lowest
> > > value 55,984, then 55,968, etc.?
> >
> (snip)
> > >
> > > I suspect it may have been a victim of a phenomenon in Silverfast
> > > where you
> > > can specify a gamma setting to be applied to the raw scan on the fly
> which
> > > then of course it is not really a raw scan.
> >
> > You should not apply ANY tonal curve adjustment to raw data until you
> > setpoint it, for obvious reasons!
>
> Believe it or not Silverfast is designed to do just that if you check a
> certain option box.

Well, they still sell matches ;-)

> > > When you open this
> > > "raw", gamma
> > > adjusted file directly in PS it initially has a good
> histogram but after
> > > applying Levels the histogram looks mildly combed. No full gaps
> > > but it gets
> > > rather spiky. Doing a raw scan in Silverfast and not applying
> the gamma
> > > setting seems to avoid this issue.
> >
> > That is exactly what I would suspect would happen if you apply tonal
> changes
> > to un-setpointed data.
>
> Well this makes a lot more sense. If you know what is going on the problem
> is easy to avoid.

Hence why I like having these discussions for not only my self, but others
to understand the theory behind this stuff.  I DO believe it helps allow you
to get better images.

> Thanks for helping me sort this out.

I believe you have the justification and understanding of it's implications
spot on, except for the "right" and "left" issue, which I believe I clarify
above.

Regards,

Austin

So what you are saying is that film scanners have no amplifaction on the 
sensor prior to the A/D? This means that tonal range for higher bit 
scans is accomplished by interpolation (call in what ever you want, 
filtering, smoothing, etc.)?

Truman

From: "Austin Franklin" <darkroom@...>
To: <DigitalBlackandWhiteThePrint@yahoogroups.com>

Sent: Sunday, October 13, 2002 9:15 AM
Subject: RE: [Digital BW] Data Mapping During Scanning was: 'combed'
histograms in 16 bit ?


(snip earlier)
> >
> > Okay understood. I guess my misunderstand comes from the fact that the
> > histograms I see of raw scans, while narrow, are often considerably
larger
> > then 6% of the 16-bit range I would have expected. However since the
> > histograms are 8-bit I guess we cannot take them literally.
>
> Yeah, that is why I want to understand what on earth is going on with the
> raw data...and how it's justified.
>
> A digression:
>
> Let's say a typical film scanner has a dMax of say, 3.6...and it's got a
12
> bit A/D...  12 bits can only represent 4096 distinct values...  Now,
typical
> film scanners we use do not have adjustable front ends, they do have
> adjustable exposure time, but that doesn't change the data relativeness
that
> the sensor sees...they record the data with linear differences in density.
>
> Now, this is a tough one, and I know it doesn't make practical sense, but
> bear with me.  B&W film has a density range of say, 2...well, according to
> how our scanner works, that's only 100 discernable tones if the scanner
> records linearly throughout the entire density range, where each
discernable
> density step is the same, and that's how most all of them work as far as I
> know.

Austin,

Do we need to sort out the difference between optical and electronic dmax?
The optical dmax of most scanners is far below 3.6. I believe only the older
Howteks such as the 4500 come close at 3.5. Other scanners even the Imacon's
are more in the 2.5 to 2.8 range. So does the highest optical density a
scanner can read then get represented by the highest bit value? Does your
12-bit scanner then place its 2.7 optical dmax at 3.6?
>
> The scanner, when recording, doesn't know what the density range of the
film
> is, that it is scanning, it has to record the entire range of 3.6 for
EVERY
> film that it scans.  That assumes that it doesn't have an adjustable front
> end, and none of the typical film scanners do...  The density range of the
> film comes into place when you set the set points, you are telling the
image
> file what the density limits of the scanned film is.

That makes sense in that the optical density range of the film will most
likely be a subset within the range of the scanner's optical density
capabilities. You certainly would not want the film to have a larger range.
>
> Now back to our regularly scheduled program ;-)
>
> > So we have 2500 adjacent levels in 16-bit space. Obviously then if you
do
> > any gamma, curves or similar adjustment without first adjusting the end
> > points outward you will lose or damage the data.
>
> Exactly correct, and why you need to set setpoints first, then expand the
> setpointed image data over the larger range, to give you gaps between the
> data to allow for tonal corrections.

This is also the situation you are in when you drop from 16-bit mode to
8-bit mode in PS. All the adjacent values are filled and any adjustment
start to drop out tonal values.
>
> > pixels will be moved from
> > one level to another level and added to the pixels already at that level
> > resulting in spikes and gaps in the data reducing the number of total
> > levels.
>
> Bingo.
>
> > If you spread or remap the end points on your data set first, you create
> > room to move the pixels from one level to another without adding them to
a
> > level that already contains pixels or at least reduce the certainty to a
> > possibility.
>
> Correct.
>
> > With our example of 2500 adjecent level from 1000 to 3500 in
> > 16-bit space let's say we move the 3500 value to 65,000 and the 1000
value
> > to 10. We still only have pixels assigned to 2500 discrete levels
> > but those
> > levels are now scattered across a range of 64,000 possible levels
> > and while
> > moving them around during adjustments the chances of their
> > interferring with
> > eachother is small. Am I tracking this correctly so far?
>
> Absolutely.  That's exactly what happens.
>
(snip earlier)
>
> Hence why I like having these discussions for not only my self, but others
> to understand the theory behind this stuff.  I DO believe it helps allow
you
> to get better images.

I am sure a lot of people are groaning over our discussion at this point but
I really feel that you cannot avoid this kind of technical discussion if you
really want to grasp how these new tools work. Scanning is incredible
important to the end quality of a digital B&W print and we need to know how
the data is manipulated.

All of this makes my desire for a good file analysis tool stronger. There is
a Mac utility on the ScanHi list that count the number of discrete levels in
a 16 to 8-bit file. I wish that there was something similar for the PC.
Still want that 16-bit histogram. Even if it is as long as a bus you could
have zoom capability and be able to scroll. Even a utility that produced a
comma delimited text file of how many pixels are assigned to each level in
16-bit space would let you go in an see exactly how the data is changed from
step to step.

Thanks,
Martin

(snip)

Hi Truman,

> So what you are saying is that film scanners have no amplifaction on the
> sensor prior to the A/D?

It's really a voltage matching circuit.  The typical A/D has an input
voltage range of say +3V to -3V (which is 6V peak to peak), and the CCD
output, let's say, has a voltage swing of 2.8V to saturation.  You have to
adjust the output voltage of the CCD to the input voltage of the A/D in
order that you take advantage of the full swing of the A/D.

Let's use the number we used above of the CCD having a 2.8V swing, and a 6V
swing for out A/D.  That would be a gain of 2.143 provided by the analog
section between the CCD and the A/D.  If the CCD were putting out 1V, the
A/D would see 2.143V...  There can be some curve adjusting in this analog
section, some scanners do that.

> This means that tonal range for higher bit
> scans is accomplished by interpolation (call in what ever you want,
> filtering, smoothing, etc.)?

No, not at all.  How did you arrive at that conclusion?

The CCD will output a voltage, and that voltage will pretty linearly
correspond to the number of electrons the CCD sensor "sees".  The lower the
number of electrons it sees, the lower the output voltage, and the higher
the density that it saw...  The converse is true as well, the higher the
number of electrons it sees, the higher the output voltage, and the lower
the density that it saw.

Austin

Actually the sensor see's photons and electrons or a potnetial 
difference (voltage) or a current flow (depending on the particulars of 
the sensor and the A/D ) results which is what is recorded. In most 
imaging systems I have worked on, there is a circuit that adjust the 
output the voltage of the image sensor to maximize the dynamic range of 
the sensor into the A/D. That can result in shot noise coming out of the 
senor in some cases but it maximizes the dynamic range of the imaging 
system. So my question is do the scanners on the market have such a 
circuit or is the photon range coming through the negative mapped 
linearly (or almost linearly) onto input range of the A/D?

That is pretty critical to the question if a 16 bit A/D really a 16 bit A/D.

Truman

Austin Franklin wrote:

> Hi Truman,
>
> >
> No, not at all.  How did you arrive at that conclusion?
>
> The CCD will output a voltage, and that voltage will pretty linearly
> correspond to the number of electrons the CCD sensor "sees".  The 
> lower the
> number of electrons it sees, the lower the output voltage, and the higher
> the density that it saw...  The converse is true as well, the higher the
> number of electrons it sees, the higher the output voltage, and the lower
> the density that it saw.
>
> Austin
>

Truman,

> Actually the sensor see's photons and electrons or a potnetial
> difference (voltage) or a current flow (depending on the particulars of
> the sensor and the A/D ) results which is what is recorded.

I don't quite understand what you are saying here...  Of course the sensor
"sees" photons, as in light, it does not "see" electrons, but during the
integration period, an image is obtained by gathering electrons generated by
photons incident upon the photodiodes.  The output is a voltage based on the
number of electrons generated.

> In most
> imaging systems I have worked on, there is a circuit that adjust the
> output the voltage of the image sensor to maximize the dynamic range of
> the sensor into the A/D.

That's exactly what I just described in my other post.  It is typically not
an adjustable circuit, it's simply a voltage gain (or attenuation) depending
on the output voltage of the CCD and the input voltage requirements of the
A/D that simply matches them to, as you say, maximize the "range" (I'm not
quite clear that it actually maximizes the "dynamic range", I'd have to
think about that, but it certainly matches the two ranges) of the sensor
into the range of the A/D.

> So my question is do the scanners on the market have such a
> circuit or is the photon range coming through the negative mapped
> linearly (or almost linearly) onto input range of the A/D?

The ALL have to have a circuit to do what I described, unless the voltage
from the CCD is the exact as the input requirement of the A/D.  That means
that "the photon range coming through the negative" is mapped linearly (yes,
almost linearly ;-) into the input range of the A/D.  I believe that is what
I said in my last post, and provided an example.

There are some scanners that do not do it that way, as I believe the
Leafscan actually adds more gain in the higher density regions, though I'm
not convinced that does much in reality, as you are still limited by noise.

> That is pretty critical to the question if a 16 bit A/D really a
> 16 bit A/D.

No, there are no scanners of the typical type we use that use actual 16 bit
A/Ds, and if they do, they are reading noise in the low bits if they do use
a 16 bit A/D.  Typically, scanners today are 12 or 14 bits at best, as
that's all they need at this point in time with current CCDs.

I think the issue you may not be getting here, is the minimum increment of
discernability is noise in the CCD, and that's what dictates what happens
downstream  If you know the noise level of the sensor, say, it's got a noise
floor of 35 electrons, and it has a saturation of 230,000 electrons.  That
basically gives you  230,000/35 or 6571 discernable steps.  You only need a
13 bit A/D to be able to represent those 6571 different values, plus one bit
for good measure ;-) (and they don't make 13 bit A/Ds that I've seen), or a
14 bit A/D.  That will give you a TRUE density range of 3.8 (not one based
on the number of bits in the A/D), which is actually pretty good.

Regards,

Austin

> Truman
>
> Austin Franklin wrote:
>
> > Hi Truman,
> >
> > >
> > No, not at all.  How did you arrive at that conclusion?
> >
> > The CCD will output a voltage, and that voltage will pretty linearly
> > correspond to the number of electrons the CCD sensor "sees".  The
> > lower the
> > number of electrons it sees, the lower the output voltage, and
> the higher
> > the density that it saw...  The converse is true as well, the higher the
> > number of electrons it sees, the higher the output voltage, and
> the lower
> > the density that it saw.
> >
> > Austin
> >