#62 Potential Overflow at `pnm2y4m.c`

[Changgong Lee]

Dear authors,
There exists a potential integer overflow at the function read_bpm_raw in lavtools/pnm2y4m.c

603    int row_bytes = (width + 7) >> 3; 
604    int total_bytes = row_bytes * height; // potential integer overflow
605    uint8_t *pbm  = buffer + total_bytes - 1;
606    uint8_t *luma = buffer + (width * height) - 1; // potential integer overflow

it could potentially cause null pointer dereferences or other undefined behaviors by *(pbm--) and *(luma--) at 617 and 618 line
when row_bytes, or width * height overflow by arbitrary values assigned to width and height.

617    uint8_t bits = *(pbm--) >> shift;
618      while (shift < 8) {
619        *(luma--) = (bits & 0x1) ? Y_BLACK : Y_WHITE;
620        bits >>= 1;
621        shift++;
622        x--;
623      }

A possible fix suggestion would be adding additional if statements.
For example,

if ( width > INT_MAX / height || row_bytes > INT_MAX / height ) {
    mjpeg_error_exit1("invalid pnm width and height, %dx%d", width, height);
  } 

[Bernhard Praschinger]

That could be a problem. On a typically system the Int_Max it 4 byte, so width * height mus be larger than 4 bytes (2147483647) means that the resolution needs to be larger than 46340 (width) * 46340 (height) pixels. It's quite unlikely that this happens and I think that a lot of other tools in the encoding pipeline can handle images at that size.
Can you give a practical example how that overflow can be triggered ?

[Changgong Lee]

This error was discovered by a static analyzer, and trace was manually reviewed.
I apologize that I can't provide a triggering testcase for this.