#806 X Server crash on drawing thick polygon with dash line style

[Kit]

This is a description of a rare fatal of X server and the problem workaround. The crash happens under very rare circumstances on some X servers and some platforms. When X server dies it also causes the client application to crash with message signaling about IO error. The typical message is "XIO: fatal IO error 104 (Connection reset by peer)" or "Fatal IO error: client killed". Although the server dies because of a buggy code, the thing which triggers the crash is inside a client application and that makes it possible to workaround it.

The "must have" conditions for the crash are:
- The poly lines should be drawn with XDrawLines and should represent a
closed polygon/shape.
- The line style should be set to any dash type and line caps should be
either flat or squares.
- The line width should be greater then one pixel
- The first segment of shape should be vertical and it should go up.
(dx ==0 && dy < 0)

There is one more condition which cannot be easily met, but I will try describing it. The last dash of last segment should be in certain position to the first point (dash) of first segment, so that it will require special rendering of that joint. That can explains why the same shape did not crash the X server every time it was drawn. Basically last condition could only be met with particular scale factor and that factor is different for different shape.

What happens inside X server when all conditions are met? The buggy code tries to allocate the memory for some drawing needs based on the height of first segment. The allocation routine gets called with negative number, which translates into very large unsigned number. This is especially bad for 64 platforms where the numbers are huge. What happens next is that X server crashes with segmentation fault accessing some other part of memory. That happens mostly on amd64 bit platforms with TightVNC (X Server) installed. Sounds ridicules? Please look yourselves at file "mi/miwideline.c" of X server source code tree. The calling stack sequence is as this:
miWideDash ->
miLineProjectingCap ->
miLineProjectingCap ->
miFillPolyHelper ->
malloc.

Currently there are many different implementations of X server on the market, but it seems they all diverged from the same ancestor and may have a problem described above. I have looked through some open source implementations and can tell that this problem is known, at least for someone. More over, some X servers have been already fixed. For example, you can see the description of bug # 2690 at http://bugs.freedesktop.org ! But some other X servers (TightVNC is one of them) still have this bug in their code. That means we cannot rely on someone fixing all existing X servers out there and we need to put some workaround in our code. The simplest workaround is to reverse the polygon points if first segment is "negative-vertical".

[Kit]

Demo application to crash TightVNC running on 64 bit platform. It requires building with Qt 4.2 and up.

[sjanki]

Logged In: YES
user_id=1943174
Originator: NO

I've been experiencing this crash too and after too many crashes the last few days, playing videos and opening PDFs I've reverted to a 32-bit x86 Tightvnc package since I use it on my production system.

I can't try it on other architectures yet (my MIPS machine's firmware can only boot 32-bit kernels at the moment and TightVNC has endianness issues on SPARC) but I'll see if I can reproduce this crash there too.

I was wondering if it isn't possible to base future TightVNC releases off Xorg, so these fixes (and possibly bugs too) could be inherited from that code base. A nice side-effect would be that it would work on big-endian architectures.