On GraphicsMagick 1.4
An allocation failure vulnerability was found in function ReadTIFFImage (tiff.c:2433),which allow attackers to cause a denial of service via a crafted file.
#./gm convert $FILE /dev/null
==41413==ERROR: failed to allocate 0x366e7ab000 (233781768192) bytes of LargeMmapAllocator (error code: 12)
==41413==Process memory map follows:
0x000000400000-0x0000012e8000 /home/test/Downloads/GMhg-afl-build/bin/gm
0x0000014e7000-0x0000014ea000 /home/test/Downloads/GMhg-afl-build/bin/gm
0x0000014ea000-0x00000160e000 /home/test/Downloads/GMhg-afl-build/bin/gm
0x00000160e000-0x000002295000
......
0x7fba2f191000-0x7fba2f366000
0x7fba2f366000-0x7fba2f367000 /usr/lib64/ld-2.17.so
0x7fba2f367000-0x7fba2f368000 /usr/lib64/ld-2.17.so
0x7fba2f368000-0x7fba2f369000
0x7ffe84f65000-0x7ffe84f86000 [stack]
0x7ffe84fb2000-0x7ffe84fb4000 [vdso]
0xffffffffff600000-0xffffffffff601000 [vsyscall]
==41413==End of process memory map.
==41413== CHECK failed: /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:120 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
#0 0x4f3dbf in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:69
#1 0x50b6e5 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_termination.cc:79
#2 0x4fc380 in __sanitizer::ReportMmapFailureAndDie(unsigned long, char const*, char const*, int, bool) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:120
#3 0x504b5e in __sanitizer::MmapOrDie(unsigned long, char const*, bool) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix.cc:132
#4 0x42fe0f in __sanitizer::LargeMmapAllocator<__asan::AsanMapUnmapCallback>::Allocate(__sanitizer::AllocatorStats*, unsigned long, unsigned long) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_allocator_secondary.h:41
#5 0x42fe0f in __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<__asan::AP64>, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<__asan::AP64> >, __sanitizer::LargeMmapAllocator<__asan::AsanMapUnmapCallback> >::Allocate(__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<__asan::AP64> >*, unsigned long, unsigned long, bool, bool) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_allocator_combined.h:70
#6 0x42fe0f in __asan::Allocator::Allocate(unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType, bool) /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/asan_allocator.cc:407
#7 0x4e9789 in __interceptor_malloc /home/test/Downloads/llvm-clang/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:67
#8 0xe8a637 in ReadTIFFImage /home/test/Downloads/GM/coders/tiff.c:2433:19
#9 0x641cf5 in ReadImage /home/test/Downloads/GM/magick/constitute.c:1607:13
#10 0x567f27 in ConvertImageCommand /home/test/Downloads/GM/magick/command.c:4348:22
#11 0x5aff8a in MagickCommand /home/test/Downloads/GM/magick/command.c:8869:17
#12 0x5f5d9e in GMCommandSingle /home/test/Downloads/GM/magick/command.c:17396:10
#13 0x5f47be in GMCommand /home/test/Downloads/GM/magick/command.c:17449:16
#14 0x7fba2ba8fb34 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
#15 0x4247fb in _start (/home/test/Downloads/GMhg-afl-build/bin/gm+0x4247fb)
The poc file is in the attachment.
Credit:ADLab of Venustech
Fixed by Mercurial changeset 15171:752c0b41fa32. Thanks for the report.