[PanoTools-devel] Fwd: Histogram Matching Panoramic Application

[JD S. <jd...@as...>]

Hi All,

While thinking about the nuts and bolts of PTBlender, I recently  
corresponded with the author of this article:

  "Color Histogram Specification by Histogram Warping", http:// 
tinyurl.com/yzbbjc

He has a real interest in some "test case" panoramic image sets for  
histogram matching, and seems interested in working with these for  
future research in the area.  Does anyone have such a test suite, or  
are there existing ones Helmut used early on to develop the histogram  
matching capabilities of PanoTools?  Ideally such a pano would be  
"worst case", auto-white balance shifting, exposure shifting, etc., etc.

He also provided some Matlab code (sent separately to some of you by  
email; let me know if you want to have a look) which implements the  
1D histogram "warping" method referenced in the SPIE article.  I  
don't read Matlab, but it looks like it could be a useful start.

On top of that, he pointed out some papers of their group on multi- 
resolution spline, which I'm sure Andrew has seen, but could be  
useful nonetheless.

Seems like this could be useful to follow up.

JD

On Nov 4, 2006, at 9:33 PM, JD Smith wrote:

> Dear Dr. Grundland,
>
> I read with interest your SPIE proceeding entitled "Color Histogram  
> Specification by Histogram Warping".  I'm involved in the panoramic  
> imaging free software community, which had its roots in the  
> Panorama Tools software written by German mathematics professor  
> Helmut Dersch.  More information at http://wiki.panotools.org/ 
> Panorama_tools.
>
> A standard issue in building large panoramic scenes from multiple  
> overlapping images (from as few as 2 to many hundreds) is matching  
> color and intensity in the overlapping frames.  These can vary due  
> to improper camera settings, changes in scene illumination during  
> the shoot (e.g. clouds pass), etc.  Dr. Dersch wrote a simple tool  
> to match histograms among a set of overlapping images, but it  
> suffers from color distortion, banding,  and other issues, and  
> (along with many other tools) is being updated and re-written (see  
> panotools.sf.net).
>
> There are two novel features for histogram matching in the field of  
> panoramic imaging, which to my knowledge have not been addressed in  
> depth:
>
> 1. The pixels in the area of overlap between two images should, in  
> theory, have precisely the same histogram.
> 2. A consistent network of mapping functions must be developed  
> which bring all images forming the scene into agreement.
>
> Number 1 is not strictly true, since movement in the scene (people  
> moving, trees blowing in the wind, etc.) can invalidate it, but  
> methods have been developed to reject such changing elements,  
> either automatically or by hand processing of masks.  My feeling is  
> that this condition would impose a much tighter set of constraints  
> on the color matching, compared to the general case of matching  
> colors in one image to an unrelated target template image (or  
> specified histogram shape).  It should also imply (I would guess)  
> less sensitivity to the exact color space used to match the  
> histograms (RGB and HSV have been employed, without great success),  
> but I don't have a good feeling for this.
>
> Within the community, we have a number of mature methods for  
> blending over the areas of overlap using multi-resolution spline  
> methods (see http://enblend.sourceforge.net/).  What is lacking is  
> a more general histogram-matching toolset, which can match images  
> taken with different camera settings or scene variables (changing  
> light source, etc.), yet doesn't introduce unwanted color shifts.   
> It would seem your research would be very relevant for the  
> development of such a tool.  I wondered if you have any code  
> implementing any of the methods you document (ideally GPL, as the  
> PanoTools are)?  Any additional insights or advice you could offer  
> on the best method to proceed would also be very much appreciated.   
> Involvement in the project at any level is also always very welcomed.
>
> Best regards,
>
> JD Smith

And received the response:

> From: Mark Grundland
> Date: November 22, 2006 8:59:09 PM MST
> To: JD Smith
> Subject: Re: Histogram Matching Panoramic Application
>
> Dear Dr. J.D. Smith,
>
> I have been away on vacation so it has taken me a while to respond  
> to your letter.
>
> I am happy to hear that you find my work on histogram warping  
> potentially useful for the color correction of panoramic images. If  
> you have a chance to apply my algorithms, please let me know how  
> they performed.
>
> There has been significant research on histogram matching  
> specifically designed for use in panoramic imaging:
> http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?tp=&arnumber=911119
> http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?tp=&arnumber=1028817
> http://rivit.cs.byu.edu/a3dg/publications/histogramPainting_final.pdf
>
>> 1. The pixels in the area of overlap between two images should, in
>> theory, have precisely the same histogram.
>> 2. A consistent network of mapping functions must be developed which
>> bring all images forming the scene into agreement.
>
> I am really not sure what is best for your application. If you  
> believe that you truly require exact color matching then you should  
> use a 3D histogram specification algorithm (see references 29, 33,  
> and 36 in my paper for possible starting points). If you believe  
> that you may benefit more from approximate or interpolated 1D  
> histogram specification then my histogram warping algorithm may  
> prove effective (see reference 17 in my paper for additional  
> theory). In such a case, you still have the choice of color space:  
> standard RGB avoids mapping colors out of gamut, decorrelated  
> R'G'B' is simple to implement using principal component analysis of  
> the color axes, and my suggestion of La"b" relies on independent  
> component analysis of the chromatic axes. In my experiments on  
> color transfer between unrelated photographic images, I observed  
> that La"b" was generally more reliable than R'G'B' which was in  
> turn generally more reliable than RGB. However, these observations  
> may not carry over to panoramic images.
>
>> Within the community, we have a number of mature methods for blending
>> over the areas of overlap using multi-resolution spline methods
>
> Perhaps our recent papers on improving the quality of image  
> blending with multi-resolution splines could prove useful:
> http://www.eyemaginary.com/Portfolio/CompositeContrastEnhance.html
> http://www.eyemaginary.com/Compositing/
>
>> Involvement in the project at any level is also always very welcomed.
>
> Do you have a standard set of test images available? If so, I would  
> greatly appreciate the opportunity to use them in the future  
> development of our image processing research. Could I have your  
> permission to reproduce them in our online and print publications  
> and presentations? Whenever we reproduce your images, we will  
> naturally include a reference to your work and send you a copy of  
> our papers when they are published.
>
>> I wondered if you have any code implementing any of the
>> methods you document (ideally GPL, as the PanoTools are)?
>
> Unfortunately, the source code for our technique is not public  
> domain and it has not been publicly released. All that I can  
> presently give you is the following fragment of my raw MatLab code  
> that implements just the bare bones of the 1D histogram warping  
> technique. Please be aware that this research code is presently  
> unsupported and sparsely commented.
>
> Best regards,
>
> Mark Grundland
> Rainbow Research Group
> Computer Laboratory
> University of Cambridge