Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

Scanner Cal and Gamma

Quantum
2006-03-13
2013-05-01
  • Quantum
    Quantum
    2006-03-13

    Hi, I just calibrated my Microtek Scanmaker5 with lprof, but am puzzled.

    Without cal, scans are dark and muddy (Debian Sarge, Gimp, Xsane), and Xsane defaults to a gamma of 1.  I carefully followed the lprof instructions (although some aspects seem to be out of date, in particular loading the Kodak reference file section). 

    The instructions say that Xsane should be set to gamma of 2.2. or 2.4, and as predicted 2.2 looks better on the flatbed.  However in comparison with the paper target, the image looks washed-out.  The exposure is about the same as the target, but it does not have the contrast.  This is disturbing.  Was I supposed to adjust levels of the scan as well?

    This scanner has a second scan function below, which eliminates the intervening glass, for negs, slides, and transparencies.  Using the Kodak film target here yields even worse results.  In Xsane I have a pulldown to select color: Full Color, Slides, or Negatives.  When I scan the film target in Full Color it is too dark, even as I increase gamma from 2.2 through 3.0, although the colors seem pretty accurate below 2.8.  At 2.8 and above the picture is washed out even as it is too dark.  When I set color to Slides (since the target is positive film), a gamma of 2.2 is about right, but numerous of the lighter colors on the target are white, implying clipping of the color space.  I suspect the Negative color choice is clipped as well.  Any advice?

    And finally, I am reading here and there that even after the scanner is calibrated, this does not mean that a scan will give a photo in ICC.  It is recommended that you convert the picture to ICC using the 'convert' command of ImageMagick.  If so, what's the point?  I thought calibrating the scanner would cause this conversion automatically?  The image file should be in ICC.  It seems this is what Xsane is supposed to be doing, but maybe not?

    Thanks

     
    • Hal Engel
      Hal Engel
      2006-03-13

      It is a common misconception that profiling software is used to "calibrate" devices.  This is not correct.  Profiling software is used to characterize a device.  Actually it characterizes the device and all of the software in the reproduction chain for that device including the settings used in that software at the time the profile is created.   This is a concept that is a little difficult for those new to color management to understand.  But the distinction is a critical one that color management users need to understand.

      Again profiling does not calibrate your scanner.  You must do any "calibration' before you profile the scanner.  By "calibration" I mean adjusting the scanners setting to get scans that don't have significant problems like blown out highlights.   In fact you should try to find default settings that get you the best overall results before you attempt to profile the scanner.   The best setting may vary with the type of scan.  So your settings for a negative scan might be significantly different from a slide scan or a reflective scan and you may (very likely) need to create different profiles for each of these.

      It says in the help file "These setting are supposed be your working scanning mode, so set all of them to reasonable defaults."  Reasonable defaults means images that look OK.  Not blown out, shadows not blocked up, reasonable gamma, contrast and so on.   If you think the levels need adjusting then adjust them.    Just remember that whatever settings you used to capture the profiling target are going to be your working default settings for that type of image.

      To utilize the profile(s) you need to be using color management aware software or you need to handle some of this manually.  XSane is not CM aware at this time so when using a scanner you will have to do some things manually.    Some scanning software is CM aware and has a place for setting things like the scanner profile and the working space output profile.  The only scanning software that runs on Linux that has this capability that I am aware of is VueScan with is not open source and will cost $.  I would really like to see the XSane folks step up and implement CM awareness and I have been encouraging them to do this.  Perhaps other users need to also do this so that this is higher on their priority list.

      Right now there are a limited number of CM aware applications on Linux but this number is increasing and lots of work is underway in this area.   One of the best at this time is CinePaint which I use myself.   To see the affect of color management create a "rough monitor profile" with LPROF for your machine.  In the CinePaint Color Management preferences set the Display profile to the one you created for your monitor.  Then open your scanned image (the IT8.7 scan you used to create the profile will do nicely) and assign the scanner profile you created to the image.  You will see that the image is now color corrected and now has good tonality and contrast.   The reason this happens is because CinePaint uses the characterization information from both the scanner profile and the monitor profile to correct the image so that it has the correct tonality and colors when displayed on your monitor.   This is the whole point of color management - it does the needed transformations to make colors correct as you move the image to different devices such as displays and printers.   If you save the image at this point CinePaint will embed the profile in the image file and from that point on you will have that information whenever you open that image in a CM aware program.  This is like using ImageMagick convert.  You can also use CinePint to transform that image to another color space before saving it.  Many users like to convert all of their images to a standard working color space.  It is common for users to use AdobeRGB, ProPhotoRGB and BetaRGB as this standardized working color space.  Some users also use sRGB but I discourage this because it's limited gamut it is really only suited to work intended for display on web pages.

      There is a significant learning curve to Color Management and at first the concepts are hard to understand.  Your confusion over calibration vs. profiling is a typical example.  After you are a little farther up the learning curve you will find that this is a very powerful tool.   But expect to struggle with it at first since this a normal thing that everyone goes through.  After all most tools that are powerful will also have a learning curve.  I recommend that anyone starting down this path spend some time reading the materials on the Norman Koren web site http://www.normankoren.com/color_management_2.htm .  The information on this site is Windows centric so Linux users will sometimes need to read between the lines but it does give a very good overview of the subject matter, is very accessable and has a wealth of information on the subject.  

      Please feel free to ask questions as I am sure that many potential users of LPROF will have these same questions and I am both willing and able to answer most of these.  If fact I suspect that I have only left you with more questions on the subject.

      So what do you need to do to get a good scanner profile.  It sounds like you have both a reflective and a slide IT8.7 target.  Select one to work with and play around with the scanner controls until you get a good scan of the target.  The most improtant thing is to make sure that you do not have clipping at either end of the tonality scale but this may not be possible at the dark end of the scale.   In the XSane preview make sure that none the patches are blown out (this is critical).    DMIN (the patch in the lower left) should have output RGB values in the range of 235 to 245.  But also check the patches at A13 through A19 to make sure none of these is blown out.   In the histogram make sure the high and low levels triangles are located at the extreme end points of the scale.   You want to capture the full dynamic range of the scanner.

      Many flat bed scanners will not have enough dynamic range to capture the full range of tones in the target.   To check this after making sure the DMIN is correctly set (see last paragraph) you should check the RGB values of the DMAX patch (lower right) and those next to it.  If the values of these patches do not change until you get to say patch 21 or 20 or 19 then the scanner may not be able to reproduce the range of tones in the target.  This is OK as far as profiling is concerned.  You can try increasing the gamma to see if this changes.  If it does not then reduce gamma until you reach 2.2 or the output RGB values of the DMAX patch are < 10.  On my scanner this resulted in a gamma of 2.2 and an exposure of - 4.1 and contrast set to 0.0.   Note that most scanning software tends to clip the highlights and it is normal to reduce the exposure significantly to prevent this.  The output RGB values for the DMAX patch were 3.   My scanner does not have enough dynamic range so DMAX and patches 20, 21 and 22 all have the same output RGB values.

      Now capture the IT8.7 image.  You may also want to write down your scanner settings for later reference.  Now it is time to use LPROF.  In LPROF load the IT8.7 image.  Also make sure that you have selected the correct reference file for this target and that the reference file is correctly installed with the right pick template.  You need to use the reference file that is specific to your target (this is critical).  This is normally shipped by the vendor with the target but in some cases you may need to get this from the vendors web site.

      Now set the profile parameters and tweaks and profile information then mark the corners of the target and make sure that the hot zones are in the right locations.  Select a profile file name and location (location defaults to $HOME/.icc/color which is the new standard location for user profiles on OSS platforms so you may want to create this directory if it does not exist) and create the profile.

      Hope this helps.

      PS: Perhaps I need to write an XSane tutorial for the LPROF help system.

       
      • Hal Engel
        Hal Engel
        2006-03-13

        I played around with XSane some more and found some additional setting that can be tweaked in the Standard options dialog.  This is located under Windows Show standard options.  On my scanner it allows me to tweak some of the Analog setting and as a result I was able to pull slightly more dynamic range out of the scanner.  This is also where you would set the scan bit depth to either 8 or 16.  I would always use a depth of 16.

        In addition you can save all of these setting using Prefernces --> Save device settings.  This will alow you to save these with a name so that you can save and load these for differenct setups.  This would be a good way to save setting configurations for different image types and profiles.

         
    • Quantum
      Quantum
      2006-03-16

      Thank you Hal, this was very helpful.

      The problem most people are up against, is we feel that roughing in the scanner settings is too subjective, and we don't necessarily trust our judgment at this stage, but I have discovered an excellent way to do this objectively: 
      - Preview the Q-60 target in the scanning software.
      - Open all the color adjustments (gamma, brightness, contrast).
      - Use the white eyedropper and *middle*-click on the target's white.  Use the black eyedropper and middle-click on the target's black.  Use the gray eyedropper and middle-click on the target's gray 10.

      This will correct 95% of color and gamma problems in all scanners, for both reflective and film targets, because it takes advantage of the fact that gray's RGB numbers are always equal, and the gray 10's gamma is close.  From this, one can profile.  Only thing is, on the film target I was not able to get good quality on the girl.  She was washed out (although color patches were brilliant and slides scan fine), but Scarse is getting almost exactly what I'm getting:
      http://www.scarse.org/images/sample/scarse/q60.tif

      My scanner cost $2,200 7 years ago (now selling for $0.99 on Ebay...), but it is a fine 1000x2000dpi optical with dynamic range of "up to" 3.6 .  I can see a difference in all patches, and none in the top row are blown out.  I've saved my flatbed and slide settings derived above, as media definitions.  Still need to calibrate negs, and this is a little daunting as no IT8 target, and I know I need to correct out the sepia manually.  Maybe if I can find a neg with known black, gray, and white...

      FYI, for the target reference files, I found every target Kodak ever made on their FTP:
      ftp://ftp.kodak.com/GASTDS/Q60DATA/

      R1 for reflective, and E1 for film.  Named by year and month.

      As to Xsane, I sent several emails to Oliver, but he only answered the first one substantively, telling me that Xsane does not support ICC!  Nowhere is this said publically.  His subsequent email said that unless I had $thousands to give him, he does not have to respond, so... whatever, I chose to *not* tell him about my calibration method.  That's a dead-end.  Don't ever expect ICC in Xsane.

      Would you say that CinePaint is superior to Gimp for photos?  It's good to know that it embeds the CM profile, but it appears that this is not the same as the colorspace?  Man... I guess Adobe is the safest bet for the future.  These will be archival family images, 1850-present.

      I'd tried the rough monitor calibration on my analog monitor, but could not fully grasp the instructions. ("equal gray all across"?)  My main machine is a notebook, so totally digital and thus no CM should be needed. 

      Aside from that, lprof worked flawlessly, and was quite easy to use.  Thank you for this Hal (and prior authors).

       
    • Quantum
      Quantum
      2006-03-16

      Hm, Cinepaint is a revelation.  It has everything The Gimp lacks.

      But Gimp plugins and scripts do not work in Cinepaint, and I can't find any Cinepaint-specific plugins, particularly for scanning.  Docs are sparse, so I can't even figure out how to change the tools from a horizontal layout to a vertical, nor how to display tool options all the time. (doy)

      The Glasgow version seems to be overdue.  Maybe I'll join the mailinglist.

       
    • Hal Engel
      Hal Engel
      2006-03-17

      Thank you for the tip on how to get a good base calibration for XSane.  If you are going to profile the scanner and use a good CM workflow then the "calibration" only really needs to be in the ball park and the profile will take care of the rest.  So if the gamma and color are 95% and the highlights are not blown out you are definitely good to go.

      It's sad to hear that the XSane maintainer will not implement color management.  But that seems strange to me and maybe he was just having a bad day.  For example, on the XSane web site one of his private wish list items is a book about color management.   And on the XSane future plans page under internal: is the following "add color management - may be via xcmsdb or scarse" which is highlighted in the "planned to do this soon/higher priority" color.  I would be willing to guess that if some CM knowledgeable programmer sent him a patch that had a good part of this done that it would not be too much longer before it would happen.  I suspect that the major hang up is that he does not know enough about CM to feel confident enough to move ahead with it.  I can relate to this since this would be a non-trivial enhancement and he would likely be on a fairly steep learning curve.

      The current help files have revised the wording of the section that you found confusing.  But you may need to get CVS for this. 

      I think it is debatable that "totally digital" (I suspect that you mean) display hardware does not need CM.  At some point things become analog again.  The real issue is how do you do calibration when the interface between the video "card" and monitor is digital and only allows 256 fixed levels/channel.  I don't have the answer.  Some believe that you need to have a way to set the CLUT values that are in the displays D to A hardware.  But no interface for this exists today.  With a CRT you still have the digital to analog CLUTS on the video card that can be used for calibration.

      The current version of CinePaint is in some ways very limited.  But it does handle 16 and 32 bit per channel images and the CM stuff is well done.  I am also hoping that Glasgow is a big step forward.  The first Beta release was supposed to have been March 1st.  But it looks like there were a lot of updates to Glasgow in CVS about 2 months ago then no real activity in CVS since that time.  I think I might grab CVS and try to build it.  But unreleased Beta software can be a real challenge to get it to build or run correctly.

      GIMP will also have CM in version 2.4.  Perhaps not as advanced as CinePaint.  If the GEGL project were moving along at a little better pace then I would feel confident that GIMP will get high bit depth image support in a time frame that is reasonable.  But it is moving at a snails pace right now which is faster than the pace it was moving 6 months ago.

      One other option for supporting high bit depths and CM in an image editor is Krita which is part of the KOffice suite.  They have been doing some good work on this lately.  Worth having a look.

       
    • quantum
      quantum
      2006-03-17

      >> I suspect that the major hang up is that he does not know enough about CM to feel confident enough to move ahead with it.

      I took away the impression that he's trying to make a living, and time is way too limited as a result, so has no room for n00bs like me.  But the cynical side of me says maybe he has brought the Xsane project to the next level, but has plateaued.  In either case, I wouldn't expect CM unless someone forks. (You maybe, I hope?) A patch would be too involved.

      >> I think it is debatable that "totally digital" (I suspect that you mean) display hardware does not need CM. At some point things become analog again.

      Only at the LCD elements.  Virtually all laptops use digital DVI all the way through the display, and there is no A/D anywhere in the hardware, so no error.  The image file is digital and the pixels have known qualities, and so negligible error.  The big error is in A/D for *VGA (D-15) monitors.  I am no expert in CM, but I know hardware pretty well.

      I'll check the new help file and Krita, thanks.

       
    • quantum
      quantum
      2006-03-17

      I should have noted also that not all laptops succeed in doing the display correctly.  Some are too blue, and maybe in that case CM is needed.

      My absolute statement meant a *quality* laptop, and an *accurate* display, like IBM, Macpro, Alien, or Tadpole.

       
    • Hal Engel
      Hal Engel
      2006-03-17

      I think it is worth pointing out that even if a device is perfectly calibrated that you still need a profile for that device to do color management correctly.  Lets assume that your display does have output that is very close to (or even exactly like) a perfectly calibrated device.  You still need a profile that characterizes that device so that programs like Photoshop or CinePaint will know how to do the color space transformations needed to correctly display an image which presumably is coming from a device that has been profiled and is in a known color space. 

      Again don't mix up calibration and characterization.  These are totally different and have different purposes.  Even a perfectly calibrated device such as a display will have different color and tonality characteristics than some other device such as a scanner or camera.  Even if those other devices also have perfect calibration.  For example DSLR cameras have gamuts that are significantly larger than the gamut of any current display device.  How do you correctly display these images without having both the camera and display profiled?