Reading uncompressed TIFF images

2009-07-14
2013-03-27
  • Matt Walkenhorst

    Thanks for an awesome utility. It really is amazing. I've got a question that I've been looking for an answer to everywhere, and I can't find anything - I'm not sure whether I'm using the right search terms or not.

    I am using 1.3.5-Q8 on Windows. I have noticed interesting behaviour when reading uncompressed RGB TIFF files. When using gm convert with very large source TIFF source files I was seeing process times of more than 5 minutes. When I used the same image as a TIFF with 100% JPEG compression as the source, processing time was reduced to a matter of seconds. The uncompressed tiff was 120Mb, the compressed tiff about 15 Mb

    I ran sysinternals Process Monitor, filtering for gm.exe. I noticed that gm spent a very long time reading the uncompressed image file over and over. When using the jpeg compressed TIFF as input, gm read the file once from start to finish.

    I thought that the re-reading might be related to the size of the uncompressed input TIFF. I took a crop of a small section of the uncompressed tiff and ran gm convert on that. The same repeat reads were observed.

    I've included a snippet of the process monitor output for the ~8Mb crop file below:

    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 23,054, Length: 2,195,456
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 2,218,510, Length: 16,384
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 2,225,617, Length: 2,195,456
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 4,421,073, Length: 16,384
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 4,428,180, Length: 2,195,456
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 6,623,636, Length: 16,384
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 6,630,743, Length: 2,195,456
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 8,826,199, Length: 7,373
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 23,054, Length: 2,195,456
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 2,218,510, Length: 16,384
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 2,225,617, Length: 2,195,456
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 4,421,073, Length: 16,384
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 4,428,180, Length: 2,195,456
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 6,623,636, Length: 16,384
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 6,630,743, Length: 2,195,456
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 8,826,199, Length: 7,373
    gm.exe    ReadFile    C:\gm\crop.tif    SUCCESS    Offset: 23,054, Length: 2,195,456

    You can see that it just keeps reading - once it gets to the end of the file it starts again. Eventually it stops, and starts writing the output file.

    My question, and I apologise if it's something that's been covered before: Is this the way that gm is supposed to read uncompressed TIFF files? It seems the larger the file, the more times it is re-read. Is there something I can do to reduce the impact of this?

    example command that exhibits this behaviour:

    gm convert   "uncompressed_tiff.tif"[0] -units PixelsPerInch -density 72  "target.tif"

    output of gm convert -list resource

    Resource Limits (Q8, 32 bits/pixel, 32bit address)
    ----------------------------------------------------
        Disk:  Unlimited (MAGICK_LIMIT_DISK)
        File:       1.5K (MAGICK_LIMIT_FILE)
         Map:      1.6GB (MAGICK_LIMIT_MAP)
      Memory:      1.6GB (MAGICK_LIMIT_MEMORY)
      Pixels:    819.2MP (MAGICK_LIMIT_PIXELS)
     
    Looking forward to your response - please let me know if there is any info that is missing that could help.

    Matt

     
    • Bob Friesenhahn

      Bob Friesenhahn - 2009-07-14

      This is very interesting and I wonder if it is due to a bug in the libtiff which is being used?
      Using a MinGW/GCC build based on recent development libtiff I see a file read time of 1.8 seconds but using the MSVC build (which uses the older libtiff) it is taking 99 seconds.  The fast/slow behavior seems to be rather file-specific.

      Note that you can see the operation of the GM TIFF reader by adding '-debug coder' to the options before the input file name:

      gm convert -debug coder "uncompressed_tiff.tif"[0] -units PixelsPerInch -density 72 "target.tif"

      If GraphicsMagick code was reading the file multiple times you should see it in this debug output.  Since it is not, it seems like something that libtiff would have to be doing.

      Bob

       
    • Bob Friesenhahn

      Bob Friesenhahn - 2009-07-14

      Ok, I did some more testing.  After the system stabilized, the performance with older libtiff and development libtiff was similar.

      Can you obtain 'tiffinfo' and 'tiffdump' from libtiff and see how the problem TIFF is organized?  Perhaps the organization of the TIFF is causing it to be read several times.

       
      • Matt Walkenhorst

        Hi Bob,

        thanks for the prompt replies. I've obtained the output from tiffinfo and tiffdump as well as the convert -debug coder output. With the latter you'll see that there is a big pause after the line written at 09:26:43 - this is where Process Monitor shows re-reading of the 120Mb uncompressed tiff file > 6,000 times over 12 minutes, for a total of 800Gb of IO!

        As you suggest it seems it's something to do with libtiff. I don't currently know enough about tiff file organisation to make anything of libtiff tool output.

        The -debug code output:

        09:26:43 0:01 0.000u 3712 constitute.c/ReadImage/8469/Coder:
          Invoking "TIFF" decoder (Tagged Image File Format)
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1312/Coder:
          Geometry: 4888x6510
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1314/Coder:
          PlanarConfiguration: separate
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1318/Coder:
          Samples per pixel: 4
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1320/Coder:
          Sample format: Unsigned integer
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1329/Coder:
          Bits per sample: 8
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1331/Coder:
          Min sample value: 0
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1333/Coder:
          Max sample value: 255
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1348/Coder:
          Photometric: RGB
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1350/Coder:
          Compression: not compressed
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1352/Coder:
          Byte swapped: true
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1355/Coder:
          Bit fill order: MSB2LSB
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1360/Coder:
          Rows per strip: 6510
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1412/Coder:
          ICC ICM embedded profile with length 560 bytes
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1425/Coder:
          Photoshop embedded profile with length 11652 bytes
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1448/Coder:
          XMP embedded profile with length 16507 bytes
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1498/Coder:
          Extra sample 1 contains UNSPECIFIED alpha
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1554/Coder:
          Image has a matte channel of type: Unspecified
        09:26:43 0:01 0.000u 3712 tiff.c/ReadTIFFImage/1739/Coder:
          Using scanline RGB read method with 8 bits per sample
        09:34:19 7:36 479.500u 3712 constitute.c/ReadImage/8477/Coder:
          Returned from "TIFF" decoder: monochrome=False grayscale=False class=DirectClass colorspace=RGB
        09:34:19 7:36 479.500u 3712 constitute.c/WriteImage/9037/Coder:
          Invoking "TIF" encoder (Tagged Image File Format): monochrome=False grayscale=False class=DirectClass colorspace=RGB
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3099/Coder:
          Using native endian byte order
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3115/Coder:
          Opening TIFF file "target.tif" using open flags "w".
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3256/Coder:
          Image characteristics: cmyk=n, gray=n, mono=n, opaque=n, palette=n
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3640/Coder:
          Using RGB photometric, 4 samples per pixel, 8 bits per sample, format Unsigned
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3664/Coder:
          Using MSB2LSB bit fill order
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3834/Coder:
          Using not compressed compression
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3838/Coder:
          Image depth 8 bits
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3842/Coder:
          Rows per strip: 1 (19552 bytes/strip)
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3864/Coder:
          Resolution 72x72 pixels per inch
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3911/Coder:
          XMP embedded profile with length 16507 bytes
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3925/Coder:
          ICC ICM embedded profile with length 560 bytes
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/3943/Coder:
          Photoshop embedded profile with length 11652 bytes
        09:34:19 7:36 479.500u 3712 tiff.c/WriteNewsProfile/2983/Coder:
          TIFFSetField(tiff=0x009ED9D0,tag=34377,length=11652,data=0x009E3E70)
        09:34:19 7:36 479.500u 3712 tiff.c/WriteTIFFImage/4128/Coder:
          Using scanline RGB write method with 8 bits per sample (19552 bytes/scanline)
        09:34:23 7:40 480.125u 3712 constitute.c/WriteImage/9047/Coder:
          Returned from "TIF" encoder

        The output from tiffinfo (I have removed XMP field padding):

        TIFF Directory at offset 0x8 (8)
          Subfile Type: (0 = 0x0)
          Image Width: 4888 Image Length: 6510
          Resolution: 300, 300 pixels/inch
          Bits/Sample: 8
          Compression Scheme: None
          Photometric Interpretation: RGB color
          Extra Samples: 1<unspecified>
          Orientation: row 0 top, col 0 lhs
          Samples/Pixel: 4
          Rows/Strip: 6510
          Planar Configuration: separate image planes
          Make: Phase One
          Model: P 30+
          Software: Adobe Photoshop CS3 Macintosh
          DateTime: 2009:03:27 14:21:50
          XMLPacket (XMP Metadata):
        <?xpacket begin="?" id="W5M0MpCehiHzreSzNTczkc9d"?>
        <x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 4.1-c036 46.276720, Mon Feb 19 2007 22:13:43        ">
           <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
              <rdf:Description rdf:about=""
                    xmlns:tiff="http://ns.adobe.com/tiff/1.0/">
                 <tiff:ImageWidth>4888</tiff:ImageWidth>
                 <tiff:ImageLength>6510</tiff:ImageLength>
                 <tiff:BitsPerSample>
                    <rdf:Seq>
                       <rdf:li>8</rdf:li>
                       <rdf:li>8</rdf:li>
                       <rdf:li>8</rdf:li>
                       <rdf:li>8</rdf:li>
                    </rdf:Seq>
                 </tiff:BitsPerSample>
                 <tiff:Compression>1</tiff:Compression>
                 <tiff:PhotometricInterpretation>2</tiff:PhotometricInterpretation>
                 <tiff:SamplesPerPixel>4</tiff:SamplesPerPixel>
                 <tiff:PlanarConfiguration>2</tiff:PlanarConfiguration>
                 <tiff:XResolution>3000000/10000</tiff:XResolution>
                 <tiff:YResolution>3000000/10000</tiff:YResolution>
                 <tiff:Make>Phase One</tiff:Make>
                 <tiff:Model>P 30+</tiff:Model>
                 <tiff:Orientation>1</tiff:Orientation>
                 <tiff:ResolutionUnit>2</tiff:ResolutionUnit>
                 <tiff:NativeDigest>256,257,258,259,262,274,277,284,530,531,282,283,296,301,318,319,529,532,306,270,271,272,305,315,33432;CB51B9D8083A51E78DD10F7EBB0CA9AD</tiff:NativeDigest>
              </rdf:Description>
              <rdf:Description rdf:about=""
                    xmlns:xap="http://ns.adobe.com/xap/1.0/">
                 <xap:ModifyDate>2009-03-27T14:21:50+11:00</xap:ModifyDate>
                 <xap:CreatorTool>Adobe Photoshop CS3 Macintosh</xap:CreatorTool>
                 <xap:CreateDate>2009-03-27T14:21:50+11:00</xap:CreateDate>
                 <xap:MetadataDate>2009-03-27T14:21:50+11:00</xap:MetadataDate>
              </rdf:Description>
              <rdf:Description rdf:about=""
                    xmlns:exif="http://ns.adobe.com/exif/1.0/">
                 <exif:ExifVersion>0220</exif:ExifVersion>
                 <exif:PixelXDimension>4888</exif:PixelXDimension>
                 <exif:PixelYDimension>6510</exif:PixelYDimension>
                 <exif:DateTimeOriginal>2009-02-27T12:03+11:00</exif:DateTimeOriginal>
                 <exif:DateTimeDigitized>2009-02-27T12:03+11:00</exif:DateTimeDigitized>
                 <exif:ExposureTime>39456/9865475</exif:ExposureTime>
                 <exif:FNumber>38473/3206</exif:FNumber>
                 <exif:ExposureProgram>0</exif:ExposureProgram>
                 <exif:ISOSpeedRatings>
                    <rdf:Seq>
                       <rdf:li>100</rdf:li>
                    </rdf:Seq>
                 </exif:ISOSpeedRatings>
                 <exif:ShutterSpeedValue>7966/1000</exif:ShutterSpeedValue>
                 <exif:ApertureValue>7170/1000</exif:ApertureValue>
                 <exif:ExposureBiasValue>0/1000000000</exif:ExposureBiasValue>
                 <exif:FocalLength>120000/1000</exif:FocalLength>
                 <exif:ColorSpace>-1</exif:ColorSpace>
                 <exif:NativeDigest>36864,40960,40961,37121,37122,40962,40963,37510,40964,36867,36868,33434,33437,34850,34852,34855,34856,37377,37378,37379,37380,37381,37382,37383,37384,37385,37386,37396,41483,41484,41486,41487,41488,41492,41493,41495,41728,41729,41730,41985,41986,41987,41988,41989,41990,41991,41992,41993,41994,41995,41996,42016,0,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,22,23,24,25,26,27,28,30;6DC6B80B933A79ABA86C7D320C88D086</exif:NativeDigest>
              </rdf:Description>
              <rdf:Description rdf:about=""
                    xmlns:dc="http://purl.org/dc/elements/1.1/">
                 <dc:format>image/tiff</dc:format>
              </rdf:Description>
              <rdf:Description rdf:about=""
                    xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">
                 <photoshop:ColorMode>3</photoshop:ColorMode>
                 <photoshop:ICCProfile>Adobe RGB (1998)</photoshop:ICCProfile>
                 <photoshop:History/>
              </rdf:Description>
              <rdf:Description rdf:about=""
                    xmlns:xapMM="http://ns.adobe.com/xap/1.0/mm/"
                    xmlns:stRef="http://ns.adobe.com/xap/1.0/sType/ResourceRef#">
                 <xapMM:InstanceID>uuid:A73551F16C1CDE1180E4A92CBE4B2478</xapMM:InstanceID>
                 <xapMM:DerivedFrom rdf:parseType="Resource">
                    <stRef:instanceID>uuid:4CBF4D2E6C1CDE1180E4A92CBE4B2478</stRef:instanceID>
                    <stRef:documentID>uuid:A56C343EEF1ADE119ED2CC709D5F739B</stRef:documentID>
                 </xapMM:DerivedFrom>
                 <xapMM:DocumentID>uuid:A63551F16C1CDE1180E4A92CBE4B2478</xapMM:DocumentID>
              </rdf:Description>
           </rdf:RDF>
        </x:xmpmeta>
        <?xpacket end="w"?>
          RichTIFFIPTC Data: <present>, 8 bytes
          Photoshop Data: <present>, 11652 bytes
          EXIFIFDOffset: 127312684
          ICC Profile: <present>, 560 bytes
        TIFF Directory at offset 0x796a32c (127312684)
          ExposureTime: 0.003999
          FNumber: 12.000312
          ExposureProgram: 0
          ISOSpeedRatings: 100
          ExifVersion: 0x30,0x32,0x32,0x30
          DateTimeOriginal: 2009:02:27 12:03:00
          DateTimeDigitized: 2009:02:27 12:03:00
          ShutterSpeedValue: 7.966000
          ApertureValue: 7.170000
          ExposureBiasValue: 0.000000
          FocalLength: 120.000000
          Tag 40961: 65535
          PixelXDimension: 4888
          PixelYDimension: 6510
        TIFFReadCustomDirectory: Warning, uncompressed_tiff.tif: unknown field with tag 40961 (0xa001) encountered.

        The output from tiffdump:

        uncompressed_tiff.tif:
        Magic: 0x4d4d <big-endian> Version: 0x2a
        Directory 0: offset 8 (0x8) next 0 (0)
        SubFileType (254) LONG (4) 1<0>
        ImageWidth (256) SHORT (3) 1<4888>
        ImageLength (257) SHORT (3) 1<6510>
        BitsPerSample (258) SHORT (3) 4<8 8 8 8>
        Compression (259) SHORT (3) 1<1>
        Photometric (262) SHORT (3) 1<2>
        Make (271) ASCII (2) 10<Phase One\0>
        Model (272) ASCII (2) 6<P 30+\0>
        StripOffsets (273) LONG (4) 4<29164 31850044 63670924 95491804>
        Orientation (274) SHORT (3) 1<1>
        SamplesPerPixel (277) SHORT (3) 1<4>
        RowsPerStrip (278) SHORT (3) 1<6510>
        StripByteCounts (279) LONG (4) 4<31820880 31820880 31820880 31820880>
        XResolution (282) RATIONAL (5) 1<300>
        YResolution (283) RATIONAL (5) 1<300>
        PlanarConfig (284) SHORT (3) 1<2>
        ResolutionUnit (296) SHORT (3) 1<2>
        Software (305) ASCII (2) 30<Adobe Photoshop CS3 Maci ...>
        DateTime (306) ASCII (2) 20<2009:03:27 14:21:50\0>
        ExtraSamples (338) SHORT (3) 1<0>
        700 (0x2bc) BYTE (1) 16507<0x3c 0x3f 0x78 0x70 0x61 0x63 0x6b 0x65 0x74 0x20 0x62 0x65 0x67 0x69 0x6e 0x3d 0x22 0xef 0xbb 0xbf 0x22 0x20 0x69 0x64 ...>
        33723 (0x83bb) LONG (4) 2<469893120 33554432>
        34377 (0x8649) BYTE (1) 11652<0x38 0x42 0x49 0x4d 0x4 0x4 00 00 00 00 00 0x7 0x1c 0x2 00 00 0x2 00 00 00 0x38 0x42 0x49 0x4d ...>
        34665 (0x8769) LONG (4) 1<127312684>
        ICC Profile (34675) UNDEFINED (7) 560<00 00 0x2 0x30 0x41 0x44 0x42 0x45 0x2 0x10 00 00 0x6d 0x6e 0x74 0x72 0x52 0x47 0x42 0x20 0x58 0x59 0x5a 0x20 ...>

         
        • Bob Friesenhahn

          Bob Friesenhahn - 2009-07-15

          Ok, I see the problem.  It is related to this:

          "Planar Configuration: separate image planes"

          Which is fairly unusual since it is RRRGGGBBBAAA rather than RGBARGBARGBA. GraphicsMagick is apparently accessing the plane data in a different order than PhotoShop wrote it.  I will see how easy this issue is to fix.

          Bob

           
          • Matt Walkenhorst

            Hi Bob,

            I can confirm that using the same uncompressed image that has been saved using interleaved pixel order (Planar Configuration: single image plane) results in it being read once in under a second and processed almost instantaneously.

            thanks very much for looking into the fix - I hope it is something that can be easily implemented.

            Matt

             
        • Bob Friesenhahn

          Bob Friesenhahn - 2009-07-15

          I used tiffcp to create a planar TIFF of 541MB.  Unfortunately, GM reads this file almost as fast as the contiguous version.  It is possible to output the TIFF with scanlines of red, green, and blue in quick succession, or as huge planes so that all of the red in the image is written before writing any green, or blue.  I am suspecting that tiffcp uses the former, and your file uses the latter. I do have Photoshop CS2 and I will see if I can use it to write a planar TIFF.

           
          • Bob Friesenhahn

            Bob Friesenhahn - 2009-07-17

            This nasty performance problem has been fixed in CVS HEAD and the latest snapshot package (1.4.020090717).  Windows builds are available.  Please test.

            Bob

             
            • Matt Walkenhorst

              Hi Bob,

              Sorry for the delay in getting back to you. My tests on the Q8 windows build show that per-channel tiff reads now take a similar amount of time to read as the interleaved ones.

              Thanks a lot for fixing this,

              Matt

               
              • Bob Friesenhahn

                Bob Friesenhahn - 2009-07-21

                This is good to hear.  My own tests show that separated TIFF reads a bit slower (20-30%).  It has to read slower since GraphicsMagick does not use separated storage for its internal representation so it now has to access its pixels three times.

                It turns out that fixing this also fixed a number of mystery error cases regarding compression and so GraphicsMagick is much more capable now.

                 
                • Matt Walkenhorst

                  Hi Bob,

                  Yes - I noticed a slightly longer read time, but it's a fantastic improvement nonetheless - thank you. With regard to the compression error cases, these must be the "Compression algorithm does not support random access" errors that it used to throw for LZW and ZIP compressed tiffs. I see that these errors no longer happen for LZW, which is awesome.

                  However, I now find ZIP compressed, Per-channel tiff crashes gm Q8 on windows when it gets to the following point:

                  11:25:29 0:02 1.391u 4584 tiff.c/WriteTIFFImage/4201/Coder:
                    Using scanline RGB write method with 8 bits per sample (4888 bytes/scanline)

                  I was using the following:

                  "C:\Program Files\GraphicsMagick-1.4-Q8\gm.exe" convert -debug coder "perchannel_ZIP.tif"[0] -units PixelsPerInch -density 72  "target.tif"

                  The interleaved tiff, ZIP compressed converted fine.

                  Matt

                   

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks