#425 glPixelMapusv with huge map kills X
Calling glPixelMapusv with a huge map size (e.g.,
65552) and a mapping of GL_PIXEL_MAP_S_TO_S (and
possibly others) will kill the X server. This behavior
was seen on the Radeon TCL branch as of 4/23/2002. On
the trunk from 4/22/2002 on G400 this call will only
segfault the application.
The program that I used to re-create this bug just
opens a GL window using GLUT and calls each of the
glPixelMap*v functions in turn with the
GL_PIXEL_MAP_S_TO_S mapping, a size of 65552, and a
pointer to the same buffer. It starts with the us
variant, and never gets past it.
Unfortunately, gdb is not terribly helpful here.
(gdb) r
Starting program: /home/idr/map_test
[New Thread 1024 (LWP 1233)]
Program received signal SIGFPE, Arithmetic exception.
[Switching to Thread 1024 (LWP 1233)]
0x405e23f7 in _mesa_test_os_sse_exception_support ()
from /usr/X11R6-DRI/lib/modules/dri/mga_dri.so
(gdb) c
Continuing.
Calling glPixelMap{us,ui,f}v with too large of a map
size...
glPixelMapusv( 3185, 65552, 0x08052ba0 )
Program received signal SIGSEGV, Segmentation fault.
0x40501b99 in _mesa_PixelMapusv (map=0, mapsize=0,
values=0x0) at pixel.c:379
379
fvalues[i] = (GLfloat) values[i];
(gdb) bt
#0 0x40501b99 in _mesa_PixelMapusv (map=0, mapsize=0,
values=0x0) at pixel.c:379
When run on the SAME G400 based system with indirect
rendering forced, it does kill the X server. Running
the X server from GDB and running the application shows
XFree86 segfaulting in ramdacModuleData. This may be a
bogus crash location.
$ LIBGL_ALWAYS_INDIRECT=y ./map_test
Calling glPixelMap{us,ui,f}v with too large of a map
size...
glPixelMapusv( 3185, 65552, 0x08052ba0 )
glPixelMapuiv( 3185, 65552, 0x08052ba0 )
XIO: fatal IO error 104 (Connection reset by peer) on
X server ":0.0"
after 33 requests (28 known processed) with 0
events remaining.
Logged In: YES
user_id=983
Ian, can you get the value of 'i' when it segfaults in
_mesa_PixelMapusv()? Also, is it the values[] or fvalues[]
array that's causing the segfault?
I'm inclined to guess that this is a GLX problem - either on
the client or server side. Mesa does the proper array
bounds checking.
Logged In: YES
user_id=423974
Using direct rendering on the G400 system, when it segfaults
i is 600. The access to fvalues causes the crash.
(gdb) print i
$1 = 600
(gdb) print fvalues[i]
Cannot access memory at address 0xc0000000
(gdb) print values[i]
$2 = 0
Logged In: YES
user_id=983
I was looking at the Mesa 4.1 code. It does bounds checking
but the 4.0.x code doesn't. I'll back-port this change.
That should do the job.
Logged In: YES
user_id=983
Ian confirms that the fix works. Closing this bug now.