#75 Bug in CoRRE encoder


From: andy@gloomy-place.com
There is a bug in the current version 1.3.8 of the
CoRRE encoder. The following patch fixes this. The
problem is caused by the buffer allocated by buffer.c
not being large enough for some complex screens.
Basically the code in RequiredBuffSize makes calls to
NumCodedRecs to determine how much space would be
needed to encode and entire screen. However
NumCodedRecs makes use of the current values of
m_maxwidth and m_maxheight, but these my change later
in a session. In particular if the values of
m_maxheight or m_maxwidth does down during the session
then the space estimate is now an underestimate of the
space required and if a sufficiently complex large
image (like a full screen photograph) is encoded then
the encoder will perform a memory exception as it
writes past the end of the buffer. The fix for this is
to save the current values of m_maxwidth and
m_maxheight and to set them to the minimum value while
calculating the buffer size. The following code shows
the modified version of RequiredBuffSize:

UINT vncEncodeCoRRE::RequiredBuffSize(UINT width, UINT
RECT fullscreen;
UINT codedrects;
int savedmaxheight = m_maxheight;
int savedmaxwidth = m_maxwidth;

// Need to set worse case size in case we ever low the
size to it.
m_maxheight = 8;
m_maxwidth = 8;

// Work out how many rectangles the entire screen would
// be re-encoded to...
fullscreen.left = 0;
fullscreen.top = 0;
fullscreen.right = width;
fullscreen.bottom = height;
codedrects = NumCodedRects(fullscreen);

// restore max width/height
m_maxwidth = savedmaxwidth;
m_maxheight = savedmaxheight;
// The buffer size required is the size of raw data
for the whole
// screen plus enough space for the required number of
// headers.
// This is inherently always greater than the RAW
encoded size of
// the whole screen!
return (codedrects * sz_rfbFramebufferUpdateRectHeader) +
(width * height * m_remoteformat.bitsPerPixel)/8;

Feel free to contact me if you need more details. I
suspect that few people see this as CoRRE is not a
frequently used encoder.

Andy Shaw