#28 Tighten array check for realloc/malloc

[loganaden Velvindron]

Problem:

Instead of only checking if req <= PNG_SIZE_MAX/element_size, go further and check that req and element_size do not overflow individually.

The change is based on OpenBSD's reallocarray() API:

/
* This is sqrt(SIZE_MAX+1), as s1s2 <= SIZE_MAX
* if both s1 < MUL_NO_OVERFLOW and s2 < MUL_NO_OVERFLOW
*/

define MUL_NO_OVERFLOW ((size_t)1 << (sizeof(size_t) * 4))

void
reallocarray(void optr, size_t nmemb, size_t size)
{
if ((nmemb >= MUL_NO_OVERFLOW || size >= MUL_NO_OVERFLOW) &&
nmemb > 0 && SIZE_MAX / nmemb < size) {
errno = ENOMEM;
return NULL;
}
return realloc(optr, size * nmemb);
}

The patch passes the regression test suite.

[Glenn Randers-Pehrson]

No, thanks. The png_malloc_array_checked() is only used by libpng to process the sPLT chunk and multiple chunks such as tEXt, and the patch would cause valid sPLT chunks, which current libpng can process safely, to be rejected, and would also reject large numbers of small text and other chunks, which are also valid. For either of these situations, the check "if (req <= PNG_SIZE_MAX/element_size)" is sufficient to prevent overflow.