#464 allocation failure in ReadTIFFImage

[bestshow]

On GraphicsMagick 1.4
An allocation failure vulnerability was found in function ReadTIFFImage （tiff.c:2282）,which allow attackers to cause a denial of service via a crafted file.

#./gm convert $FILE /dev/null
==70921==ERROR: failed to allocate 0x2424245000 (155225182208) bytes of LargeMmapAllocator (error code: 12)
==70921==Process memory map follows:
    0x000000400000-0x0000012e8000    /home/test/Downloads/GMhg-afl-build/bin/gm
    0x0000014e7000-0x0000014ea000    /home/test/Downloads/GMhg-afl-build/bin/gm
    0x0000014ea000-0x00000160e000    /home/test/Downloads/GMhg-afl-build/bin/gm
    0x00000160e000-0x000002295000   
    ......
    0x7f7559ed7000-0x7f755a0ac000   
    0x7f755a0ac000-0x7f755a0ad000    /usr/lib64/ld-2.17.so
    0x7f755a0ad000-0x7f755a0ae000    /usr/lib64/ld-2.17.so
    0x7f755a0ae000-0x7f755a0af000   
    0x7ffc307c3000-0x7ffc307e4000    [stack]
    0x7ffc307f0000-0x7ffc307f2000    [vdso]
    0xffffffffff600000-0xffffffffff601000    [vsyscall]
==70921==End of process memory map.
==70921== CHECK failed: /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:120 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
    #0 0x4f3dbf in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:69
    #1 0x50b6e5 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_termination.cc:79
    #2 0x4fc380 in __sanitizer::ReportMmapFailureAndDie(unsigned long, char const*, char const*, int, bool) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:120
    #3 0x504b5e in __sanitizer::MmapOrDie(unsigned long, char const*, bool) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix.cc:132
    #4 0x42fe0f in __sanitizer::LargeMmapAllocator<__asan::AsanMapUnmapCallback>::Allocate(__sanitizer::AllocatorStats*, unsigned long, unsigned long) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_allocator_secondary.h:41
    #5 0x42fe0f in __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<__asan::AP64>, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<__asan::AP64> >, __sanitizer::LargeMmapAllocator<__asan::AsanMapUnmapCallback> >::Allocate(__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<__asan::AP64> >*, unsigned long, unsigned long, bool, bool) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_allocator_combined.h:70
    #6 0x42fe0f in __asan::Allocator::Allocate(unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType, bool) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:407
    #7 0x4e9789 in __interceptor_malloc /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:67
    #8 0xe87553 in ReadTIFFImage /home/test/Downloads/GM/coders/tiff.c:2822:26
    #9 0x641cf5 in ReadImage /home/test/Downloads/GM/magick/constitute.c:1607:13
    #10 0x567f27 in ConvertImageCommand /home/test/Downloads/GM/magick/command.c:4348:22
    #11 0x5aff8a in MagickCommand /home/test/Downloads/GM/magick/command.c:8869:17
    #12 0x5f5d9e in GMCommandSingle /home/test/Downloads/GM/magick/command.c:17396:10
    #13 0x5f47be in GMCommand /home/test/Downloads/GM/magick/command.c:17449:16
    #14 0x7f75567d5b34 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
    #15 0x4247fb in _start (/home/test/Downloads/GMhg-afl-build/bin/gm+0x4247fb)

The poc file is in the attachment.

Credit：ADLab of Venustech

[Bob Friesenhahn]

This allocation failure appears to be due to a weakness in libtiff itself rather than GM.

[Bob Friesenhahn]

Fixed by Mercurial changeset 15171:752c0b41fa32. Thanks for the report.