Thread: [Lcms-user] local convergence problems?
An ICC-based CMM for color management
Brought to you by:
mm2
From: Cory P. <pap...@ju...> - 2006-01-25 20:47:59
|
Hey all. I've been playing with the new dev snapshot (with 16-bit support) and it's fixed one part of my problem, but not the other. I've been looking at the generated profiles with iccexamin, and I think I've finally figured out what's causing my issues. When I generate a profile with the local convergence enabled, there seems to be a relatively abrubtly changing areas. It's difficult to describe, and I haven't completely convinced myself that it's not just a series of bad IT8.7 test target shots. It's so weird that I haven't ruled out bugs either. Don't know if I can be anymore help, but I could at least provide the test target shot and see if anyone else could explain it. Thanks, -Cory -- ************************************************************************* * Cory Papenfuss * * Electrical Engineering candidate Ph.D. graduate student * * Virginia Polytechnic Institute and State University * ************************************************************************* |
From: Hal V. E. <hv...@as...> - 2006-01-25 21:59:39
|
On Wednesday 25 January 2006 12:47 pm, Cory Papenfuss wrote: > Hey all. I've been playing with the new dev snapshot (with 16-bit > support) and it's fixed one part of my problem, but not the other. I've > been looking at the generated profiles with iccexamin, and I think I've > finally figured out what's causing my issues. When I generate a profile > with the local convergence enabled, there seems to be a relatively > abrubtly changing areas. It's difficult to describe, and I haven't > completely convinced myself that it's not just a series of bad IT8.7 test > target shots. It's so weird that I haven't ruled out bugs either. > > Don't know if I can be anymore help, but I could at least provide > the test target shot and see if anyone else could explain it. > > Thanks, > -Cory There is a problem with local convergence extrapolation that results in local reversals (abrupt localized changes in the curves) . There is a section in the help files that describes the problem. There is also a bug report that has been opened on this issue. Please see http://sourceforge.net/tracker/index.php?func=detail&aid=1296658&group_id=146038&atid=764370 The problem was worse before 16 bit target image support was added but that did not completely fix the problem. My testing indicates that it only happens in the darkest parts of images and is worse in areas that have highly saturated very dark colors (dark reds and yellows seem to be the worst). It does not show up at all in images with lower contrast levels as far as I can tell since these images will not have areas dark enough to show the problem. I have not had a chance to dig into the code to figure out what is happening but I think that this is related to the fact that digital cameras, particularly those with larger sensor sites like DSLRs, have a gamut that is considerably larger than what is represented on an IT8.7 chart. In other words there are significant areas of the cameras gamut where there is little or no data from the IT8.7 image and local convergence extrapolation in affect tries too hard to make these areas "right" and the curves end up with discontinuities. So far most of my work on LPROF has been to consolidate the user interface and to remove redundancies from the code. Little has been done to the underlaying profile generation algorithms at his point. The current recommendation is to not use local convergence extrapolation with wide gamut devices as this will result in smoother curves that do not exhibit the problem. I suspect that the problem is for the most part limited to those who are working with raw images since raw processing is cabable of pulling way more gamut/dynamic range out of images then would be typical for images that are processed in camera. Cory - When I look at the curves in the LPROF profile checker I can't really see the discontinuities in the curves even though I know that these exist. I have ICCExamin 0.38 and I don't see anyplace to inspect the curves as such. Could you fill me in on how you are doing this? Hal |
From: Kai-Uwe B. <ku...@gm...> - 2006-01-26 06:52:41
|
Am 25.01.06, 13:59 -0800 schrieb Hal V. Engel: > Cory - When I look at the curves in the LPROF profile checker I can't really > see the discontinuities in the curves even though I know that these exist. I > have ICCExamin 0.38 and I don't see anyplace to inspect the curves as such. > Could you fill me in on how you are doing this? The mtf1/2 curves are visible in the A2B and B2A tables. Select one table from the tag list (window below the menubar). Below the tag list window a mtf1/2 selector appears. The first entry shows an overview of the whole mtf table. The second the matrix, followed by the input curves. After this the 3D tables (3D Lut's). There you have a right button menu to select the table channel, navigate and tune appearance like background colour or show the spectral curve. And as last entry in the mtf selector comes the output curve of that table. regards Kai-Uwe Behrmann + development for color management + imaging / panoramas + email: ku...@gm... + http://www.behrmann.name |
From: Cory P. <pap...@ju...> - 2006-01-26 13:51:44
|
> There is a problem with local convergence extrapolation that results in local > reversals (abrupt localized changes in the curves) . There is a section in > the help files that describes the problem. There is also a bug report that > has been opened on this issue. Please see > > http://sourceforge.net/tracker/index.php?func=detail&aid=1296658&group_id=146038&atid=764370 > > The problem was worse before 16 bit target image support was added but that > did not completely fix the problem. My testing indicates that it only > happens in the darkest parts of images and is worse in areas that have highly > saturated very dark colors (dark reds and yellows seem to be the worst). It > does not show up at all in images with lower contrast levels as far as I can > tell since these images will not have areas dark enough to show the problem. > > I have not had a chance to dig into the code to figure out what is happening > but I think that this is related to the fact that digital cameras, > particularly those with larger sensor sites like DSLRs, have a gamut that is > considerably larger than what is represented on an IT8.7 chart. In other > words there are significant areas of the cameras gamut where there is little > or no data from the IT8.7 image and local convergence extrapolation in affect > tries too hard to make these areas "right" and the curves end up with > discontinuities. > > So far most of my work on LPROF has been to consolidate the user interface and > to remove redundancies from the code. Little has been done to the > underlaying profile generation algorithms at his point. The current > recommendation is to not use local convergence extrapolation with wide gamut > devices as this will result in smoother curves that do not exhibit the > problem. I suspect that the problem is for the most part limited to those > who are working with raw images since raw processing is cabable of pulling > way more gamut/dynamic range out of images then would be typical for images > that are processed in camera. > > Cory - When I look at the curves in the LPROF profile checker I can't really > see the discontinuities in the curves even though I know that these exist. I > have ICCExamin 0.38 and I don't see anyplace to inspect the curves as such. > Could you fill me in on how you are doing this? > > Hal > Good to know that what I'm describing isn't just due to my ignorance of how the whole system works. Like I said, it's only been recently that I've been able to analyze the profiles enough to narrow it down to the localized convergence parameter. The built-in LPROF profile checker doesn't show any of the badness really... dE always looked good. Just for the record, the problem is worse as you say with RAW files and linear data... probably due to the reduced numerical contrast of linear data. As far as what I used to notice the problem... first it was the little graphical curve in UFRAW's input histogram plot. What nominally looks like a gamma curve ("fuzzy" due to the matrix) had large jumps in the curve in the brightest area of the histogram. The resulting RAW conversion had strange artifacts in the brightest areas of the image. Yesterday, I figured out how to load up multiple profiles in iccexamin (0.38) and compare the gamuts. You can see the discontinuities there, but only if you turn on the Spektral line and show the 3D surface in color (not gray). If you do that, there is a projection onto the Lab plane that shows discontinuities. Behrmann's answer to your question was another feature of iccexamin I was unaware of. I didn't know there was a way to look into the matrix before. Now that I've done that a bit, you can definately see some funny stuff going on with the local-convergence-enabled profile. Without it, all the slices are pretty much grescale cubes. With it, there are some awfully strange discontinuous "blocky" shapes in all of the different slices. All of the curves (TRC, shaper, etc) have always looked good... nominally linear or gamma as appropriate. Just FYI, I am using a Pentax *ist-DS DSLR, so the sensor likely has the larger gamut than the IT8.7 can adequately profile. I've even found the spectral response curves for the Sony sensor inside... :) -Cory -- ************************************************************************* * Cory Papenfuss * * Electrical Engineering candidate Ph.D. graduate student * * Virginia Polytechnic Institute and State University * ************************************************************************* |
From: Cory P. <pap...@ju...> - 2006-01-26 15:35:44
|
> The problem was worse before 16 bit target image support was added but that > did not completely fix the problem. My testing indicates that it only > happens in the darkest parts of images and is worse in areas that have highly > saturated very dark colors (dark reds and yellows seem to be the worst). It > does not show up at all in images with lower contrast levels as far as I can > tell since these images will not have areas dark enough to show the problem. > I would also like to add (having played with a bunch of different profiles in the past few hours) that the low-luminance measurements end up with significantly more error than the high luminance measurements when using linear data. It almost implies that the curve-fitting optimization implies log-spacing on the data in its error minimization. Possible? BTW, iccexamin is becoming cooler all the time. I was unaware of the matrix visualization and the measurement display capability. Much easier to see in 3-space which swatches don't match well. -Cory -- ************************************************************************* * Cory Papenfuss * * Electrical Engineering candidate Ph.D. graduate student * * Virginia Polytechnic Institute and State University * ************************************************************************* |
From: Hal V. E. <hv...@as...> - 2006-01-26 19:59:34
|
On Thursday 26 January 2006 05:51 am, Cory Papenfuss wrote: snip > > Good to know that what I'm describing isn't just due to my > ignorance of how the whole system works. Like I said, it's only been > recently that I've been able to analyze the profiles enough to narrow it > down to the localized convergence parameter. The built-in LPROF profile > checker doesn't show any of the badness really... dE always looked good. > Just for the record, the problem is worse as you say with RAW files and > linear data... probably due to the reduced numerical contrast of linear > data. > > As far as what I used to notice the problem... first it was the > little graphical curve in UFRAW's input histogram plot. What nominally > looks like a gamma curve ("fuzzy" due to the matrix) had large jumps in > the curve in the brightest area of the histogram. The resulting RAW > conversion had strange artifacts in the brightest areas of the image. I have stayed away from using the matrix in UFRAW as I don't think this gives good results in a color managed workflow. This was not based on any objective data but more on how I subjectively felt about the results when I tested with the matrix turned on. I found this to be true even when I used another profiler (ProfilePrism). So this in not LPROF specific. This may also explain why you are seeing artifacts in the bright areas and I am not with local convergence turned on. I should document this in the LPROF help file UFRAW section. I should add that profiles generated by LPROF (without local convergence) and ProfilePrism give results that are only very slightly different. Subjectively the PP profiles are very slightly more saturated and that is only difference that I can detect. Overall tonality, white balance, shadow detail, highlight detail ... are almost identical. Of course it would be a simple matter for a UFRAW user to make adjustments in how much saturation (or contrast, shadow detail, brightness) the profile will add to the final result when profiling with LPROF by adjusting the saturation level in UFRAW in the opposite direction when converting the IT8 target. So with a slight adjustment I could make the LPROF and PP profiles give subjectively identical results. > > Yesterday, I figured out how to load up multiple profiles in > iccexamin (0.38) and compare the gamuts. You can see the discontinuities > there, but only if you turn on the Spektral line and show the 3D surface > in color (not gray). If you do that, there is a projection onto the Lab > plane that shows discontinuities. This helped and I now know what I am looking at. The differences between a non-local convergence profile and a local convergence profile are very clear when viewed this way. snip > All of the curves (TRC, shaper, etc) have always looked good... > nominally linear or gamma as appropriate. Just FYI, I am using a Pentax > *ist-DS DSLR, so the sensor likely has the larger gamut than the IT8.7 can > adequately profile. I've even found the spectral response curves for the > Sony sensor inside... :) The curves have always looked good and that is one of the reasons that I have not looked into this in more detail. That is I did not have a good way to visualize what was happening so it would have been difficult to tell if changes to the code had the desired affect. I am going to dig into the code to see if I might be able to get this fixed as part of the 1.11 series. This might take a while since I have not looked that this part of the code in detail and I suspect that the changes will not be trivial. My D70 uses the same sensor as do most, if not all, of the 6 mega pixel DSLRs with the exception of the Canon DSLRs. > > -Cory |
From: Cory P. <pap...@ju...> - 2006-01-26 20:08:39
|
> I have stayed away from using the matrix in UFRAW as I don't think this gives > good results in a color managed workflow. This was not based on any > objective data but more on how I subjectively felt about the results when I > tested with the matrix turned on. I found this to be true even when I used > another profiler (ProfilePrism). So this in not LPROF specific. This may > also explain why you are seeing artifacts in the bright areas and I am not > with local convergence turned on. I should document this in the LPROF help > file UFRAW section. > Ooops. I guess I wasn't clear. The matrix I was referring to is the LUT in the profile, not the "use color matrix" checkbox from within UFRAW. I think I finally (and recently) figured out what that's supposed to do. It enables the RGB->RGB transform matrix from dcraw (who got it from Adobe) *before* it does any of the color management stuff. In other words, it's only really useful if you're going to do a default sRGB non-color-managed conversion. For any real profiles (like the ones I've been playing with generated by LPROF), the checkbox should be off. > I should add that profiles generated by LPROF (without local convergence) and > ProfilePrism give results that are only very slightly different. > Subjectively the PP profiles are very slightly more saturated and that is > only difference that I can detect. Overall tonality, white balance, shadow > detail, highlight detail ... are almost identical. Of course it would be a > simple matter for a UFRAW user to make adjustments in how much saturation (or > contrast, shadow detail, brightness) the profile will add to the final result > when profiling with LPROF by adjusting the saturation level in UFRAW in the > opposite direction when converting the IT8 target. So with a slight > adjustment I could make the LPROF and PP profiles give subjectively identical > results. > I don't trust myself to do much subjective adjustments. :) The profile I got with LPROF without local convergence yields a very similar result to that which I ripped out of a BibblePro install. That's what I'd been using so far. >> >> Yesterday, I figured out how to load up multiple profiles in >> iccexamin (0.38) and compare the gamuts. You can see the discontinuities >> there, but only if you turn on the Spektral line and show the 3D surface >> in color (not gray). If you do that, there is a projection onto the Lab >> plane that shows discontinuities. > > This helped and I now know what I am looking at. The differences between a > non-local convergence profile and a local convergence profile are very clear > when viewed this way. > ... in particular the much larger gamut (at least in my case). Without local convergence, the 3D space is more or less contained within the Lab space. With local convergence on, the gamut is significantly larger (and more broken). > snip > >> All of the curves (TRC, shaper, etc) have always looked good... >> nominally linear or gamma as appropriate. Just FYI, I am using a Pentax >> *ist-DS DSLR, so the sensor likely has the larger gamut than the IT8.7 can >> adequately profile. I've even found the spectral response curves for the >> Sony sensor inside... :) > > The curves have always looked good and that is one of the reasons that I have > not looked into this in more detail. That is I did not have a good way to > visualize what was happening so it would have been difficult to tell if > changes to the code had the desired affect. I am going to dig into the code > to see if I might be able to get this fixed as part of the 1.11 series. This > might take a while since I have not looked that this part of the code in > detail and I suspect that the changes will not be trivial. > > My D70 uses the same sensor as do most, if not all, of the 6 mega pixel DSLRs > with the exception of the Canon DSLRs. > The matrix view (NOT gamut view) is the most telling. Pretty funky on the broken profiles, but very smooth on the non-local convergence ones. -Cory -- ************************************************************************* * Cory Papenfuss * * Electrical Engineering candidate Ph.D. graduate student * * Virginia Polytechnic Institute and State University * ************************************************************************* |
From: Hal V. E. <hv...@as...> - 2006-01-26 20:26:48
|
On Thursday 26 January 2006 12:07 pm, Cory Papenfuss wrote: snip > Ooops. I guess I wasn't clear. The matrix I was referring to is > the LUT in the profile, not the "use color matrix" checkbox from within > UFRAW. I think I finally (and recently) figured out what that's supposed > to do. It enables the RGB->RGB transform matrix from dcraw (who got it > from Adobe) *before* it does any of the color management stuff. In other > words, it's only really useful if you're going to do a default sRGB > non-color-managed conversion. For any real profiles (like the ones I've > been playing with generated by LPROF), the checkbox should be off. Thanks for clearing this up. > > > I should add that profiles generated by LPROF (without local convergence) > > and ProfilePrism give results that are only very slightly different. > > Subjectively the PP profiles are very slightly more saturated and that is > > only difference that I can detect. Overall tonality, white balance, > > shadow detail, highlight detail ... are almost identical. Of course it > > would be a simple matter for a UFRAW user to make adjustments in how much > > saturation (or contrast, shadow detail, brightness) the profile will add > > to the final result when profiling with LPROF by adjusting the saturation > > level in UFRAW in the opposite direction when converting the IT8 target. > > So with a slight adjustment I could make the LPROF and PP profiles give > > subjectively identical results. > > I don't trust myself to do much subjective adjustments. :) The > profile I got with LPROF without local convergence yields a very similar > result to that which I ripped out of a BibblePro install. That's what > I'd been using so far. One of the features of ProfilePrism is that it allows you to tweak your profiles to change contrast, saturation, gamma (labled brightness in PP) and color balance. This is useful in PP for doing printer profiles since it uses a scanner as it's "measurement tool" and needs this to compensate for the scanner. These same settings can also be used when profiling a camera or scanner. But when I use PP I set these to not make any changes (default settings). I was mainly pointing out that the differences between the results from LPROF and PP (with default settings) are very small and that LPROF/UFRAW users can influence the profile much like a PP user can but that this is done in a slightly different way. I think that the results I am getting without doing this are actually sightly better than PP as I think the default setting in PP give a very slightly over saturated result. snip > > The matrix view (NOT gamut view) is the most telling. Pretty > funky on the broken profiles, but very smooth on the non-local convergence > ones. Exactly, this will be very helpful when working on fixing this as I will be able to tell if a code change has made this worse or better or not changed anything. > > -Cory |