Processing

Remark : Outdated , principles correct but needs thorough update to reflect the now flexible pipe

This page tries to describe in reasonable detail what processing is done, how it is done, and what are the assumptions. It should enable advanced users and developers to review correctness and understand the underlying code. Any and all feedback or discussion on this description or the underlying code is welcomed on the developers list dlraw-devel@lists.sourceforge.net ( Overview and options )

Simplified representation of the graphical pipe - workflow

Following image shows the different operations that can be done on an image. Also it is shown in which colorspace they are done and it is indicated to which 'tab' in the application it is linked.

Remark : the red shapes are as well reference points where a preview can be done as places where the image data are cached for performance.

Detailed description of the graphical pipe

Phase in 'RunPipe' (dlMain.cxx) Description Influential parameters
Phase=1, SubPhase=1 Identify Image, Camera, Raw sizes and depending on the type of raw also some other settings.
All coming from dcraw.		 TheDcRaw->
m_UserSetting_InputFileName
Phase=1, SubPhase=1
Decode Image, Remove bad pixels, Subtract darkframe.
All coming from dcraw.
When TheDcRaw->m_UserSetting_HalfSize = 1 then 2X2 pixels of the Bayer array are mapped onto one pixel of the Image. This way no interpolation is needed anymore and an important speedgain is made. This feature is coded into the BAYER(row,col) macro.

Further also m_MatrixSRGBToCamRGB is calculated here from m_MatrixCamRGBToSRGB. The latter in turn is a not yet (fully) understood function from TheDcRaw->m_UserSetting_CameraMatrix, TheDcRaw->m_UserSetting_CameraWb which sometimes throws in another matrix (m_cmatrix). 		(TheDcRaw->)

m_UserSetting_HalfSize
m_UserSetting_BadPixelsFileName
m_UserSetting_DarkFrameFileName
m_UserSetting_CameraMatrix)
m_UserSetting_CameraWb)
Phase=1, SubPhase=2
Calculate the multipliers for a correct whitebalance.
Largely taken from dcraw.
The multipliers can be obtained from user input (TheDcRaw->m_UserSetting_Multiplier) or from a 'greybox' (TheDcRaw->m_UserSetting_GreyBox) that can ultimately extend to the whole image (TheDcRaw->m_UserSetting_AutoWB). Also the whitebalance can be taken from the camera (TheDcRaw->m_UserSetting_CameraWb). Whitebalance from camera can be obtained because the camera provides the multipliers or because the camera provides a sample grey area.

The result of this operation is TheDcRaw->m_Multipliers[].
At this stage, and as done by dcraw, also a wavelet denoising algorithm on R-G/G-B is done. This algorithm might rescale the image , resulting in a different m_WhiteLevel and m_BlackLevel.
Based on this Multipliers a whitebalanced image is calculated, simply by multiplying the different channels with their respective multiplier. The multipliers are normalized such that the whitelevel of the sensor (the maximum value for that specific camera) is mapped onto the value 0xFFFF for that channel with the largest multiplier. This ensures all other values for all channels are below 0xFFFF.

Interpolation (Demosaicing)
Largely taken from dcraw.
First the m_BlackLevel is subtracted from the image.
Then a 'preinterpolation' is done, but in fact this boils down to equalizing G1 to G2 or postpone that if TheDcRaw->m_MixGreen = 1 (which is the case if m_UserSetting_FourColorRGB=1 or m_UserSetting_HalfSize=1). m_Filters is adapted accordingly or set to 0 if m_UserSetting_HalfSize=1 (because then no interpolation is needed) Also dimensions are adapted in case of m_UserSettingHalfSize=1.
Then , if there is still a Bayer (m_Filters) , interpolation is done according to one of the 4 algorithms determined by TheDcRaw->m_UserSetting_Quality : linear,vng,ppg or ahd.
Finally, if needed, also 'green mixing' is done, as well a median filtering, with a user defined number of passes.. 		(TheDcRaw->)

m_UserSetting_Multiplier
m_UserSetting_CameraWb
m_UserSetting_AutoWb
m_UserSetting_GreyBox
m_UserSetting_BlackPoint
m_UserSetting_Saturation
m_UserSetting_HalfSize
m_UserSetting_FourColorRGB
m_UserSetting_Quality
m_UserSetting_RawDenoiseThreshold
m_UserSetting_MedianPasses
Phase=1, SubPhase=3
Handling of the highlights
This part of the pipe handles the highlights, i.e. those parts of the image that are on the maximum level of the sensor and thus inherently clipped. Different methods are used to handle those highlights.
Remember from previous phase that the image is now scaled such that the maximum sensor value of the most multiplied channel arrives at 0xFFFF. This is referred to as the unclipped pixels. Here we recalculate the image with renormalized multipliers such that the maximum sensor value of the least multiplied channel arrives at 0xFFFF. We refer to it as the 'Clipped Pixel' and it has thus the characteristic that all channels are at 0XFFFF as soon as one of the channels obtained its maximum sensor value.
  • ClipMode = dlClipMode_Clip :
    In this case the ClippedPixel is used for the highlights. It ensures no color artifacts are created due to the clipping. However color channels that were not blown before might be now and the linearity is lost.
  • ClipMode = dlClipMode_NoClip :
    In this case the unclipped pixel as it existed after the colorscaling is used in the final image. No unnecessary blowing is done and the linearity is preserved. However due to one of the color channels being clipped in a pixel an unbalance leading to (pink) color artifacts is possible.
    Above is true for ClipParameter 0. By increasing the ClipParameter more and more of the ClippedPixel is mixed in until ClipParameter becomes 100 and again only the ClippedPixel is used.
  • ClipMode = dlClipMode_Lab
    In this case the pixel (as well the unclipped as the clipped) is converted to CIE Lab. The L of the final image is choosen to be a combination of the 'unclipped' L (ClipParameter 0) to the 'clipped' L (ClipParameter 100).
    Final ab components are both taken from the 'clipped' pixel.
    This is equivalent to the 'LCH' restore of ufraw. Preservation of ab should preserve colors to the maximum extent. Unclear why the clipped value rather than the unclipped is taken though. Someone ?
  • ClipMode = dlClipMode_HSV
    In this case the pixel (as well the unclipped as the clipped) are converted to HSV values. The V (Luminance) of the final image is choosen to be a combination of the 'unclipped' V (ClipParameter 0) to the 'clipped' V (ClipParameter 100).
    Saturation is taken from the 'clipped' pixel and Hue from the 'unclipped'.
    This is identical to the 'HSV' restore of ufraw and it tends to retain a bit more details (but is harder than LCH).
  • ClipMode = dlClipMode_Blend
    This is the 'highlight blending' mode of dcraw. As the manual describes : Blend clipped and unclipped values together for a gradual fade to white. I'm not yet fully sure of what the code exactly does. (Dave Coffin from dcraw is one of those guys that prefers writing if A/B > 1 in his code rather than if A > B ...) But it looks like the dlClipMode_Lab where the final L value is function from the saturation of the other ab channels. Someone more clear on what is done exactly here ?
  • ClipMode = dlClipMode_Rebuild
    This is the 'highlight rebuilding' mode of dcraw. These modes will take the tone from the non blown areas in the neighbourhood to fill the blown areas. The higher the ClipParameter value is set (7 is about a practical maximum) , the more effort will be done in emulating the surrounding colors and less guarantee to obtain neutral (grey) in the burnt pixels.
    Warning : this is slow especially for large ClipParameter !
(TheDcRaw->)

m_UserSetting_dlRaw_ClipMode
m_UserSetting_dlRaw_ClipParameter
Phase=1, SubPhase=4
Lensfun lens corrections

See the lensfun page.
Basically this step applies lens related corrections on photos taken with a lens, provided that the lens is characterized in the lensfun database. The corrections are :
  • Vignetting Correction
  • Chromatic Aberration Correction
  • Distortion Correction
  • Geometry Correction

At this moment no effort is made to guess the lense used as this is very unstandardized information. So you have to choose your lenses manually out of a set which is compatible (based on 'mount' information) with the camera. Remark that lensfun functionality is depending on lensparameters being available for the lensfun library, which is a potential weakness. If anyone wants to contribute lenses, please do so to the lensfun project !

Working space conversion.
The Image in TheDcRaw is in the color space of the camera, which is some variant on an RGB color space.
In previous phases also TheDcRaw->m_MatrixCamRGBToSRGB was obtained, be it via matrices delivered by the camera itself, be it by (Cam RGB -> XYZ) matrices delivered by Adobe (TheDcRaw::adobe_coeff).
This matrix (or a profile) will be used to convert to the working RGB color space.
  • When no Icc profile is defined :
    Following chain of matrices is used : TheDcRaw->m_MatrixCamRGBToSRGB => MatrixRGBToXYZ[dlSpace_sRGB_D65] => MatrixXYZToRGB[TargetWorkingSpace].
  • When an Icc profile is defined :
    The profile is used in a cmsCreateTransform to transform into the TargetWorkingSpace.
    Note that also in this case no gamma is added in this transform, i.e. the result is in the target space but linear encoded !
    On the net there are hanging around a number of profiles that are clearly not profiled starting from the camera rgb space, but from some intermediate rgb space. Those cannot be used. Some programs use the assumption that intermediate space is sRGB, but that's arbitrarely. I don't support it. The only relevant profile would be one that is profiled from the camera space directly, such as could be done with lprof
(GuiSettings->)

m_WorkColor
m_CameraProfileName
m_EnableLensfun
m_LensfunCameraMake;
m_LensfunCameraModel;
m_LensfunCameraIndex;
m_LensfunLensIndex;
m_LensfunFocalLength;
m_LensfunF;
m_LensfunDistance;
m_LensfunTCAEnable;
m_LensfunHaveTCAModel;
m_LensfunTCAModel;
m_LensfunVignettingEnable;
m_LensfunHaveVignettingModel;
m_LensfunVignettingModel;
m_LensfunDistortionEnable;
m_LensfunHaveDistortionModel;
m_LensfunDistortionModel;
m_LensfunGeometryEnable;
m_LensfunGeometry;
m_LensfunScale;
Phase=2
  • Rotate
    A Rotation by 3 shears algorithm is used for rotation over an arbitrary angle.
  • Crop
    Part of the image can be cropped from the preview. Some support is in place for cropping in a predefined ratio H/W. Cropped sizes on the preview are calculated back to the relevant sizes on the full size image.
  • Resize
    A bilinear interpolation resizing algorithm is used.
  • Channel Mixer
    In the Channel Mixer the RGB channels are remixed. For each target R,G,B one can define how much of the original R,G,B is transferred to the target RGB. There are a few applications for that, the most known being to make a B/W picture. In that case each of the targets R,G,B is getting an equal amount of the original R,G,B. For instance Target R = Target G = Target B = 30%R + 59%G + 11%B. A number of interesting presets is defined.
  • Exposure
    It has been choosen explicitly to do the exposure at this point, as I see it as an RGB operation very much like a curve. In fact exposure can (and in some case is) be done by a curve.
    Also I believe the handling of 'clipping' is different then it is in the color scaling (where some other programs put this step). At color scaling time 'clipping' and 'highlight' is all about guessing and restoring channels that are fundamentally lost (once a sensor reaches its maximum, the game is over). Here we are starting from an image that is not clipped or of which the clipped information is restored in earlier phase. When we multiply now the values we might clip again (go over 0XFFFF) but with that important distinction we do have the original information to do something sensible with it.
    Steps involved in exposure
    • Calculate a correction (exposure normalization) in case of EOS camera. This idea is coming from ufraw and it seems indeed needed to have a decent default exposure.
    • Then the normal exposure normalization is calculated on the basis of 1/MinimalPreMultiplier.
    • Alternatively that exposure can be determined by the usersetting m_Exposure.
    • Alternatively, when m_AutoExposure=1, an exposure is calculated that brings a fraction m_WhiteFraction (default 0.1) of the pixels above the 90% level.
    • Finally, the exposure correction is applied.
    Handling of the clipping
    Depending on the parameter m_ExposureClipMode.
    • dlExposureClipMode_None : No specific measurements are taken at 0XFFFF except for clipping. Channels might get unbalanced and color artifacts.
    • dlExposureClipMode_Ratio : The channel that clips the most is put at 0XFFFF, but the other channels are put in the same ratio as they were before. This gives a certain amount of colour preservation.
    • dlExposureClipMode_Curve : All of the image is exposed, not by simply multiplying, but by feeding it through a film like curve with a shoulder that prevents clipping. The function is (1-exp(-Alpha*x)/(1-exp(-Alpha)) with Alpha such that f(0)=0, f(1)=1 and f'(0) = Exposure.
  • Gamma Tool
    This is a weird ad-hoc part on which I really would like to receive thoughts and explanations. It is a curve transformation that adds some gamma like compression to the image, in such a way that , in an sRGB working space, the resulting gamma after application of the sRGB gamma becomes as described best by the function itself :

    double GammaTool(double r, double Gamma, double Linearity) {
    const double g = Gamma * (1 - Linearity) / (1-Gamma*Linearity);
    const double a = 1/(1 + Linearity*(g-1));
    const double b = Linearity * (g-1)*a;
    const double c = pow((a*Linearity+b),g) / Linearity;
    return r<Linearity ? c*r : pow (a*r+b,g); }

    It is originating from the observation that when applying this gamma to the image it simply looks better than applying the standard sRGB which looks 'foggy'. This was at occasions discussed on the ufraw lists, but I don't believe it was ever explained adequately.
  • RGB,R,G,B Curves
    A curve is applied onto the image. With this all kind of tonal effects, like mimicking of filmcurves etc., can be obtained. An RGB curve is applied equally on all three the RGB channels. The R,G,B curves are applied on their respective channel only.
    There are internally two types of curves : 'Anchor' or 'Full'. The 'Anchor' curve is obtained by cubic spline interpolation on a number of anchors. The curve is editable by (re)moving or adding anchors. The 'Full' curve is obtained by applying a mathematical function that fully describes the curve. Those ones are not editeable. (The GammaTool curve as described above would be such one).
(GuiSettings->)

m_Rotate
m_Crop
m_CropX
m_CropY
m_CropW
m_CropH
m_Resize
m_ResizeW
m_ResizeH
m_EnableChannelMixer
m_ChannelMixer
Exposure
AutoExposure
WhiteFraction
ExposureClipMode
m_EnableGamma
m_Gamma
m_Linearity
m_HaveCurve[]
Phase=3
Conversion to Lab colorspace
At least if one of the following operations is effectively requested.
  • Enhance Contrast and Saturation
    Here contrast is increased by applying a sigmoidal curve onto the L channel. Also saturation can be increased by applying a sigmoidal curve onto the ab channel.
    The parameter 'Amount' determines the slope of the curve at its steepest point.
    The parameter 'Threshold' determines where the steepest point is located.
  • Apply Lab Curves
    This curve transformation is applying a curve transformation but only on the L (=luminance) channel or ab channel of Lab. Lab has a fairly good separation of Luminance and color, which mean colors can be enhanced without affecting luminance or the other way around.
  • Luminance denoising
    A wavelet denoising with the given Threshold is applied onto the L channel.
  • USM sharpening
    A pretty standard USM sharpening is applied.
    The parameters :
    • Radius: The radius of the Gaussian kernel in pixels.
    • Sigma: The standard deviation of the Gaussian. In pixels.
    • Amount: The percentage of the difference between the original and the blur image that is added back into the original.
    • Threshold: The threshold in pixels needed to apply the diffence amount.
  • Refocus sharpening
    Refocus sharpening is applied.
    • Matrix Radius: This parameter determines the size of the transformation matrix. (Size = 1+2*Radius) Increasing the Matrix Radius may give better results, especially when you have chosen large values for Radius or Gauss. High values slow down a lot. Typical values would be in the range 3-10.
    • Radius: The radius of the circular convolution in pixels. 1 gives good results. Increase if very blurred.
    • Gauss: The radius of the Gaussian convolution. Use this parameter when your blurring is Gaussian but keep it preferably at 0 as it causes artifacts. When you use non-zero values, you will probably have to increase the Correlation and Noise paramaters as well.
    • Correlation: Increasing it may help reducing artifacts. The value is 0 to 1, with 0.5 as default. Values close to 1 (0.95,0.99) will reduce the sharpening effect.
    • Noise: Increasing it may help reducing artifacts. The value is 0 to 1. A useful value is 0.01. Lower impact image quality badly. Higher values reduce the sharpening effect.
  • Chroma denoising
    A wavelet denoising with the given Threshold is applied onto the ab channels.
(GuiSettings->)

m_HaveCurve[dlCurveChannel_L]
m_HaveCurve[dlCurveChannel_a]
m_HaveCurve[dlCurveChannel_b]
m_EnableContrast
m_ContrastAmount
m_ContrastThreshold
m_EnableSaturation
m_SaturationAmount
m_SaturationThreshold
m_LuminanceDenoiseThreshold
m_USM
m_USMRadius
m_USMSigma
m_USMAmount
m_USMThreshold
m_Refocus
m_RefocusMatrixRadius
m_RefocusRadius
m_RefocusGauss
m_RefocusCorrelation
m_RefocusNoise
m_ChromaDenoiseThreshold
Phase=4
GREYCstoration noise reduction

See the GREYCstoration page.
The different parameters you can find here.

Be aware that this is a very slow algorithm and that sometimes nearly as good results can be obtained by one of the other noise reduction features in a much faster way. 		(GuiSettings->)

m_GREYC
m_GREYCAmplitude
m_GREYCSharpness
m_GREYCAnisotropy
m_GREYCAlpha
m_GREYCSigma
m_GREYCdl
m_GREYCda
m_GREYCGaussPrecision
m_GREYCInterpolation
m_GREYCFast

Gamma

Here, I want to elaborate specifically on gamma on which I believe a lot of confusion is existing.

When an image is gamma encoded, a mathematical relation is applied such that the encoded values are not linear proportional to the intensity of the captured light. Rather the encoding is such that in the most simple case EncodedValue = pow(LinearValue,1/2.2). (the precise function is dependent on system and environment but let's skip for the moment that detail).

In fact there are two reasons for doing so. The first one is nowadays probably the most important. Let's assume LinearValue is ranging from 0 to 100%. Then we get following encoded values :

LinearEncoded
0%0%
25%54%
50%73%
75%88%
100%100%

What you see is that the first 50% of linear values do use 73% of the available encoded values (f.i. 186 steps of 256 in a 8 bit system). This way much more detail (smaller steps) is kept in the low luminance areas then in the high luminance areas. Like much of our senses, the eye is heavily non-linear and is by far not as sensible in the high luminance areas as it is in the low luminance areas. This way we have an encoding that makes optimum use of the available encoding steps while our eye does not see the 'deceit'.

A second reason for this encoding (and becoming less important) is coming from CRT and TV technology. It happens that a CRT is not responding linear neither. The output light is about proportional to pow(AppliedVoltage,2.2). So if our image is now encoded with a gamma of 2.2 the two transformations cancel out :
LightOfCRT = pow(EncodedImage,2.2) = pow(pow(LinearLight,1/2.2),2.2) = LinearLight !

Linear processing in 16 bit

Nearly all image processing algorithms are linear and are assuming that the underlying representations are linear. When an image would be gamma-encoded as above, the algorithm gives wrong results.

However, when linear encoding is done, more bits are needed to keep the same 'accuracy' (technically spoken : Signal to Noise Ratio, SNR). Suppose for instance that you have a gamma encoded 8 bit system. Then the encoded value 1/256 corresponds to pow(1/256,2.2) = 5.034e-6. So that's the first fine step representable. When linear encoded, all steps would be equal to 1/256 = 3906e-6, so it is about 1000 times less precise. To obtain the same precision, a number of encoded values of 1/5.034e-6 = 198668 is needed. This would need 18 bits. Let's see however in some more detail to the first encoding steps of a gamma encoded 8 bit system :

Encoded valueLinear ValueStep Needed ValuesNeeded bits
00
1/2565.034e-65.034e-6 19864918
2/25623.13e-618.10e-6 5524816
3/25656.43e-633.30e-6 3003015

One can learn from the table that except for the very first quantization step the same precision (SNR) is reached with 16 bits or less as in a gamma encoded 8 bit representation. It is generally accepted that operations in a 16 bit linear encoded representation give sufficient precision.

dlRaw does all of its operations on a 16 bit linear representation !

Here follow some links that might clarify the points made above :

Encoding of values

Encoding of values is done as suggested by ICC. This means - and to a certain extent this was validated - that matrix operations in dlRaw and profile operations are mutual compatible.

CIE XYZ :
Value Encoding
0.0 0x0000
1.0 0x8000
1.0+(32767/32768) 0xFFFF

LAB L* encoding :
Value Encoding
0.0 0x0000
100 0xFF00
100+(25500/65280) 0xFFFF

LAB a*,b* encoding :
Value Encoding
-128 0x0000
0 0x8000
127 0xFF00
127+255/256 0xFFFF