#175 CVE-2024-28579 / CVE-2024-28582: reject overflowing HDR RLE scanline counts

[Daniel Rosser]

Applies to FreeImage SVN trunk r1911 (also 3.19.0 [r1909]).

CVE-2024-28579 / CVE-2024-28582 — heap buffer overflow in PluginHDR.cpp rgbe_ReadPixels_RLE() / rgbe_RGBEToFloat().

The two "not RLE, read flat" fallbacks passed (scanline_width * num_scanlines) into rgbe_ReadPixels() as an unsigned pixel count. scanline_width is a signed int: a negative width becomes a huge unsigned value via the usual arithmetic conversions, and even a positive product was never checked for overflow. rgbe_ReadPixels() then loops that count and writes FIRGBF pixels (and FreeImage_Unload later walks the smashed DIB).

This patch introduces rgbe_SafeScanlinePixelCount() and rejects a negative width or a product that does not fit in 32-bit unsigned.

NVD:
https://nvd.nist.gov/vuln/detail/CVE-2024-28579
https://nvd.nist.gov/vuln/detail/CVE-2024-28582

GitHub: https://github.com/danoli3/FreeImage/commit/d866b72 (PR https://github.com/danoli3/FreeImage/pull/55)

Apply from the FreeImage tree root:

patch -p0 < CVE-2024-28579.patch