#425 glPixelMapusv with huge map kills X

closed-fixed
Brian Paul
5
2002-04-24
2002-04-24
Ian Romanick
No

Calling glPixelMapusv with a huge map size (e.g.,
65552) and a mapping of GL_PIXEL_MAP_S_TO_S (and
possibly others) will kill the X server. This behavior
was seen on the Radeon TCL branch as of 4/23/2002. On
the trunk from 4/22/2002 on G400 this call will only
segfault the application.

The program that I used to re-create this bug just
opens a GL window using GLUT and calls each of the
glPixelMap*v functions in turn with the
GL_PIXEL_MAP_S_TO_S mapping, a size of 65552, and a
pointer to the same buffer. It starts with the us
variant, and never gets past it.

Unfortunately, gdb is not terribly helpful here.

(gdb) r
Starting program: /home/idr/map_test
[New Thread 1024 (LWP 1233)]

Program received signal SIGFPE, Arithmetic exception.
[Switching to Thread 1024 (LWP 1233)]
0x405e23f7 in _mesa_test_os_sse_exception_support ()
from /usr/X11R6-DRI/lib/modules/dri/mga_dri.so
(gdb) c
Continuing.
Calling glPixelMap{us,ui,f}v with too large of a map
size...
glPixelMapusv( 3185, 65552, 0x08052ba0 )

Program received signal SIGSEGV, Segmentation fault.
0x40501b99 in _mesa_PixelMapusv (map=0, mapsize=0,
values=0x0) at pixel.c:379
379
fvalues[i] = (GLfloat) values[i];
(gdb) bt
#0 0x40501b99 in _mesa_PixelMapusv (map=0, mapsize=0,
values=0x0) at pixel.c:379

When run on the SAME G400 based system with indirect
rendering forced, it does kill the X server. Running
the X server from GDB and running the application shows
XFree86 segfaulting in ramdacModuleData. This may be a
bogus crash location.

$ LIBGL_ALWAYS_INDIRECT=y ./map_test
Calling glPixelMap{us,ui,f}v with too large of a map
size...
glPixelMapusv( 3185, 65552, 0x08052ba0 )
glPixelMapuiv( 3185, 65552, 0x08052ba0 )
XIO: fatal IO error 104 (Connection reset by peer) on
X server ":0.0"
after 33 requests (28 known processed) with 0
events remaining.

Discussion

  • Brian Paul
    Brian Paul
    2002-04-24

    Logged In: YES
    user_id=983

    Ian, can you get the value of 'i' when it segfaults in
    _mesa_PixelMapusv()? Also, is it the values[] or fvalues[]
    array that's causing the segfault?
    I'm inclined to guess that this is a GLX problem - either on
    the client or server side. Mesa does the proper array
    bounds checking.

     
  • Ian Romanick
    Ian Romanick
    2002-04-24

    Logged In: YES
    user_id=423974

    Using direct rendering on the G400 system, when it segfaults
    i is 600. The access to fvalues causes the crash.

    (gdb) print i
    $1 = 600
    (gdb) print fvalues[i]
    Cannot access memory at address 0xc0000000
    (gdb) print values[i]
    $2 = 0

     
  • Brian Paul
    Brian Paul
    2002-04-24

    Logged In: YES
    user_id=983

    I was looking at the Mesa 4.1 code. It does bounds checking
    but the 4.0.x code doesn't. I'll back-port this change.
    That should do the job.

     
  • Brian Paul
    Brian Paul
    2002-04-24

    • assigned_to: nobody --> brianp
    • status: open --> closed-fixed
     
  • Brian Paul
    Brian Paul
    2002-04-24

    Logged In: YES
    user_id=983

    Ian confirms that the fix works. Closing this bug now.