[xine-cvs] HG: xine-lib-1.2: demux_yuv4mpeg2: fix stack overflow

[Petri H. <phi...@us...>]

# HG changeset patch
# User Petri Hintukainen <phi...@us...>
# Date 1511433731 -7200
# Node ID fa5fa03e2239354d32943226da43abf9daccef18
# Branch  default
# Parent  b7fada9c66432838fbd589413561f9a12014c515
demux_yuv4mpeg2: fix stack overflow

Optimized memmem() overreads the buffer if limit is not correctly set:
AddressSanitizer: stack-buffer-overflow on address 0x7ffeda7069e5 at pc 0x7f562a4552b0 bp 0x7ffeda706870 sp 0x7ffeda706018
READ of size 100 at 0x7ffeda7069e5 thread T0
    #0 0x7f562a4552af  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x702af)
    #1 0x7f55e00ffb91 in open_yuv4mpeg2_file /usr/src/devel/xine-lib-1.2-OUT/src/demuxers/demux_yuv4mpeg2.c:217

Also corrupt input could trigger this.

diff --git a/src/demuxers/demux_yuv4mpeg2.c b/src/demuxers/demux_yuv4mpeg2.c
--- a/src/demuxers/demux_yuv4mpeg2.c
+++ b/src/demuxers/demux_yuv4mpeg2.c
@@ -214,11 +214,13 @@
   this->frame_pts_inc = (90000 * this->fps_d) / this->fps_n;
 
   /* finally, look for the first frame */
-  char *data_start_ptr = memmem(header_ptr, Y4M_HEADER_BYTES, "FRAME", 5);
+  size_t left = (size_t)Y4M_HEADER_BYTES - (size_t)(header_ptr - header);
+  char *data_start_ptr = memmem(header_ptr, left, "FRAME", 5);
 
   /* make sure the first frame was found */
-  if ( !data_start_ptr )
+  if ( !data_start_ptr ) {
     return 0;
+  }
 
   this->data_start = data_start_ptr - header;