pixel format is out of sync, after refreshing and suddenly changing pixel format

[Anon S. <ano...@gm...>]

Hi,

I'm not sure to call this a bug in vncviewer, or a bug in VNC protocol.

Anyhow, this bug apply to realvnc 4.0 to 4.1.1.  The bug may
demonstrate the flaw in VNC protocol, which don't have any mechanism
to ensure the synchronization of the pixel format between client and
server.

Let's see how to reproduce this bug first, and then see how pixel
format is out of sync.
(To view this bug in a clear formatted text, visit
http://opensource.wikia.com/wiki/Bugs:realvnc4/pixel_format_is_unpredictable_after_refreshing_and_suddenly_changing_pixel_format
)

== How to reproduce ==
Run the following command in some terminal, "terminal1", in this example,

   terminal1 $ Xvnc :1 -nolisten tcp -auth /dev/null -localhost
-SecurityTypes=None -geometry 800x600 -depth 24 &
   terminal1 $ DISPLAY=:1 xterm -e perl -e '$|=1; for(;;){print".";
select(undef,undef,undef,0.001); }' &

Then open another terminal, "terminal2", run vncviewer,

   terminal2 $ vncviewer :1 -shared -AutoSelect=0 -ViewOnly=1 -FullColor

   VNC Viewer Free Edition 4.1.1 for X - built Jan  6 2007 00:46:26
   Copyright (C) 2002-2005 RealVNC Ltd.
   See http://www.realvnc.com for information on VNC.

   Fri Dec  7 20:11:20 2007
    CConn:       connected to host localhost port 5901

   Fri Dec  7 20:11:21 2007
    CConnection: Server supports RFB protocol version 3.8
    CConnection: Using RFB protocol version 3.8
    TXImage:     Using default colormap and visual, TrueColor, depth 24.
    CConn:       Using pixel format depth 24 (32bpp) little-endian rgb888
    CConn:       Using ZRLE encoding

Switch back to "terminal1", suspend the Xvnc,

   terminal1 $ killall -TSTP Xvnc

Then switch to vncviewer x11 window, refresh screen, and change colour level,

   vncviewer x11: press F8 --> select "Refresh screen"
   vncviewer x11: press F8 --> select "Options..." --> select "Very
low (8 colours)" in "Encoding and Colour Level:" --> click "Ok"

Switch to "terminal1", resume Xvnc,

   terminal1 $ killall -CONT Xvnc

vncviewer will then fail like this,

   terminal2 $ vncviewer :1 -shared -AutoSelect=0 -ViewOnly=1 -FullColor
   <--cut-->
   ...
   <--cut-->
    CConn:       Using ZRLE encoding

   Fri Dec  7 20:11:45 2007
    CConn:       Using pixel format depth 3 (8bpp) rgb111
   vncviewer: ../rfb/zrleDecode.h:196: void rfb::zrleDecode8(const
rfb::Rect&, rdr::InStream*, rdr::ZlibInStream*, rdr::U8*,
rfb::CMsgHandler*): Assertion `len <= end - ptr' failed.
   Aborted



== Cause ==
This is because the client and server fail to synchronize the pixel
format, which is depicted as follow,

                             server send FramebufferUpdate0
                             server is suspended
   client receive FramebufferUpdate0
   client send FramebufferUpdateRequest1:incremental=true

   user request refresh screen
   client send FramebufferUpdateRequest2:incremental=false
   user change colour level

                             server is resumed
                             server receive
FramebufferUpdateRequest1:incremental=true
                             server send FramebufferUpdate1
   client receive FramebufferUpdate1
   client change its internal pixel format to new pixel format
   client send SetPixelFormat
   client send FramebufferUpdateRequest3:incremental=false
              (non-incremental since pixel format changed)
   client is now expecting new pixel format from server
                             server receive
FramebufferUpdateRequest2:incremental=false
                             server send FramebufferUpdate2 with old
pixel format
   client receive FramebufferUpdate2 with old pixel format
   client interprete FramebufferUpdate2 as new pixel format
   client mis-interprete pixel format, and then fail