#1628 TightVNCServer(Xtightvnc) 1.3.10 crashed when trying to draw a big image

[Ming Williams]

TightVNCServer(Xtightvnc) 1.3.10 crashed when trying to draw a big image

if ((widthBytesLine != widthBytesLineProto) ||
        ((unsigned long)shmdesc->addr + stuff->offset & (sizeof(long)-1))) 
    {
        /* temp stuff for 64 bit alignment stuff */
        register char * bufPtr, * protoPtr;
        register int i;

        if(!(tmpImage = (char *) ALLOCATE_LOCAL(length))) 
          return (BadAlloc);
        tmpAlloced = 1;

        (*pDraw->pScreen->GetImage)(pDraw, stuff->x, stuff->y,
                    stuff->width, stuff->height,
                    stuff->format, stuff->planeMask,
                    tmpImage);

I was able to find out the reason of this bug. In function ProcShmGetImage, when allocating memory for future GetImage call, the function called ALLOCATE_LOCAL and returned the result to tmpImage. However, the parameter (length) is not restricted, and ALLOCATE_LOCAL actually happens on stack, which means with carefully designed length, we can manipulate the stack pointer($rsp) to another memory region. Then we can build our own stack in that memory region, hence in theory take control of the process. The parameter(length) is calculated by the program when trying to determine how much space to draw a single frame, which can be controlled by user. When I tested this bug, I was able to draw a frame about 3500x1700 pixels with 24bit depth, making the stack pointer go out of the stack space. I think this bug will be easier to be exploited on 32bit OS as allocating memory to a specific address is harder on x64 platforms.
This bug is actually not affected by compiler optimization, as I recompiled the code with -O0 the bug can still be triggered.
Btw, the coredump file is too big(~681MB) for me to upload.

[Ming Williams]

My bad, the raw coredump file is way too big, so I compressed it.
Post it now.