#202 Heap buffer overflow in GifQuantizeBuffer via integer overflow

[sidhartha kumar]

giflib GifQuantizeBuffer: Width*Height integer overflow → heap OOB write

Location

quantize.c:62-200, function GifQuantizeBuffer(unsigned int Width, unsigned int Height, int *ColorMapSize, const GifByteType *RedInput, const GifByteType *GreenInput, const GifByteType *BlueInput, GifByteType *OutputBuffer, GifColorType *OutputColorMap).

Tested against giflib 6.1.3 (latest release, shipped 2026-04-12).

Public-API reach

GifQuantizeBuffer is part of the public C API of giflib:

• Prototype is exported via getarg.h:40 and was deliberately restored to the library at 5.0 because client applications depend on it (quantize.c:8-10 comment: "This doesn't really belong in the core library, was undocumented, and was removed in 4.2. Then it turned out some client apps were actually using it, so it was restored in 5.0.").
• The only in-tree caller is gif2rgb.c (which uses it in the rgb2gif direction with attacker-controllable input dimensions when used as a service).
• Out-of-tree callers across the ecosystem accept user-supplied image dimensions.
• OSS-Fuzz egif_fuzz_common.cc calls it but with Width = 100 hardcoded — the bug condition was never explored by the existing harness.

Defect — signed integer overflow on Width * Height (CWE-190 → CWE-787)

/* quantize.c:90 */
for (i = 0; i < (int)(Width * Height); i++) {

Width and Height are unsigned int. The product is computed in unsigned int arithmetic and then cast to int for the loop comparison.

For Width * Height > INT_MAX (smallest square trigger: 46341 * 46341 = 2,147,488,281 > 2,147,483,647 = INT_MAX), the cast is implementation-defined per C99 §6.3.1.3 and on every mainstream platform produces a negative value (-2,147,479,015 for the smallest-square trigger). The loop body never executes; every ColorArrayEntries[i].Count stays at zero.

Downstream chain:

/* quantize.c:111-115 */
for (i = 0; i < COLOR_ARRAY_SIZE; i++) {
    if (ColorArrayEntries[i].Count > 0) {
        break;
    }
}

/* quantize.c:117 */
QuantizedColor = &ColorArrayEntries[i];

/* quantize.c:119-127 */
NewColorSubdiv[0].QuantizedColors = QuantizedColor;
NumOfEntries = 1;
while (++i < COLOR_ARRAY_SIZE) {  /* skipped, i is already COLOR_ARRAY_SIZE */
    if (ColorArrayEntries[i].Count > 0) {
        QuantizedColor->Pnext = &ColorArrayEntries[i];
        QuantizedColor = &ColorArrayEntries[i];
        NumOfEntries++;
    }
}
QuantizedColor->Pnext = NULL;   /* <-- heap OOB write of 8 bytes */

The find-non-empty loop exits with i == COLOR_ARRAY_SIZE == 32768, so &ColorArrayEntries[32768] is one-past-end. The C standard permits forming it but not dereferencing it; line 126 writes 8 bytes (a QuantizedColorType *) past the end of the 32768-entry, 24-byte-per-entry allocation made at quantize.c:75-76.

Secondary OOB writes/reads exist at quantize.c:151-156 in SubdivColorMap if execution continues past the first write (the same one-past-end pointer is walked through Pnext).

Verification — ASan crash log

Built giflib 6.1.3 unmodified with -fsanitize=address, then linked the poc.c reproducer (in this directory) against libgif.a:

$ ASAN_OPTIONS=detect_leaks=0 ./poc
[poc] Width=46341 Height=46341  uint product=2147488281  int cast=-2147479015
=================================================================
==136242==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f64059ef810 at pc 0x562d06a9c033 bp 0x7fff65db6850 sp 0x7fff65db6840
WRITE of size 8 at 0x7f64059ef810 thread T0
    #0 0x562d06a9c032 in GifQuantizeBuffer /tmp/giflib-audit/giflib/quantize.c:126
    #1 0x562d06a9b5f5 in main /tmp/giflib-audit/poc/poc.c:45
    ...
0x7f64059ef810 is located 16 bytes to the right of 786432-byte region [0x7f640592f800,0x7f64059ef800)
allocated by thread T0 here:
    #0 0x7f6408a5c887 in __interceptor_malloc
    #1 0x562d06a9b8b1 in GifQuantizeBuffer /tmp/giflib-audit/giflib/quantize.c:75

Region size 786,432 = 32768 * sizeof(QuantizedColorType) = 32768 * 24. The 16-byte offset past the end corresponds to the Pnext field of the (one-past-end) QuantizedColorType struct on x86-64 (16 bytes of {RGB[3] + NewColorIndex + long Count} precede Pnext).

Trigger surface

Any Width * Height whose 32-bit unsigned product exceeds INT_MAX:

Width	Height	uint product	(int) cast	trigger?
46340	46340	0x7FFF7C30	2147119152	no (just under INT_MAX)
46341	46341	0x80001259	-2147479015	yes (smallest square)
50000	50000	0x95028200	-1795524096	yes
65535	65535	0xFFFE0001	-131071	yes (max 16-bit GIF dims)
65536	65536	0x00000000	0	yes (wraps to zero)
0	any	0	0	yes (zero-dimension trips same path)

The function does not validate Width or Height at entry.

Reachability

GifQuantizeBuffer is reachable from any program that converts RGB pixel data to a paletted/indexed format via the giflib API and accepts attacker-controllable dimensions. The bug condition does not require valid pixel data — Red/Green/BlueInput are never read once the sampling loop is skipped.

Examples of exposed surfaces (illustrative, not enumerated audits):

• Server-side image conversion services that accept a width/height from request metadata before reading pixel payload.
• Imaging/animation libraries that wrap giflib and quantize user-supplied PNG/BMP/raw RGB.
• The in-tree gif2rgb -r (rgb2gif) path with -s W H on attacker-supplied input.

Patch summary

See patch.diff. The change adds 10 lines (1 include, 9 statement lines) to quantize.c:

1. #include <limits.h> for INT_MAX.
2. An early-return guard at function entry that rejects Width == 0, Height == 0, and any (Width, Height) whose product would exceed INT_MAX in unsigned int arithmetic.
3. Returns GIF_ERROR (giflib's existing error convention; matches the OOM path at line 78).

The check uses Width > (unsigned int)INT_MAX / Height — the standard overflow-safe form. No new system dependencies, no behavior change for any valid Width * Height <= INT_MAX, no abort().

[sidhartha kumar]

added the patch.diff attachment couldn't more than one attachment at a time