Bug description:
When trying to connect with a custom-made client that advertises only the RRE and RAW encodings (with that order of preference) and the big endian flag is set to 1 in SetPixelFormat client's message, the resulting data sent by Tight VNC server results in a yellow-ish screen (i.e. the blue channel is completely missing).
How to re-produce:
This can be reproduced only by a programmer. In the Tight VNC client souce code perform the following changes in your favourite IDE:
1) To advertise only the RRE and RAW encodings: Edit the file tvnjviewer-2.8.3\src\main\java\com\glavsoft\rfb\protocol\Protocol.java edit the method sendSupportedEncodingsMessage(), find the line:
SetEncodingsMessage encodingsMessage = new SetEncodingsMessage(encodings);
and insert before it:
// *** new lines start
encodings.clear();
encodings.add(EncodingType.RRE);
encodings.add(EncodingType.RAW_ENCODING);
// *** new lines end
SetEncodingsMessage encodingsMessage = new SetEncodingsMessage(encodings);
This way the server will use the RRE encoding in the framebuffer updates that it sends.
2) To use big endian byte order: Edit the file tvnjviewer-2.8.3\src\main\java\com\glavsoft\rfb\protocol\auth\AuthHandler.java at the end of the file, in the last method completeContextData(ServerInitMessage serverInitMessage, Protocol protocol) and replace the line:
protocol.setServerPixelFormat(serverInitMessage.getPixelFormat());
with the following:
// Use the server's pixel format but use big endian byte order
PixelFormat pf = serverInitMessage.getPixelFormat();
pf.bigEndianFlag = 1;
protocol.setServerPixelFormat(pf);
Run the modified TightVNC viewer from your IDE and connect to a Tight VNC server. The result is a yellow-ish screen, That means that the blue channel is missing and that the Tight VNC server encodes the frame's bytes incorrectly. If you revert only the second change (i.e. use little endian, same as the server) the servers works as expected.
Addtional information:
My c++ skills are a litlle rusty, however by taking a look at the file tightvnc-2.8.8-gpl\rfb-sconn\RreEncoder.cpp there is a suspicious code block in RreEncoder::rreEncode() method that creates a mask and then all framebuffer's pixels are masked with that value. If the framebuffer's pixels have already been converted to big endian, this masking could easily be the reason of why the blue channel is missing. However this is just a best guess, I haven't taken a thorough look at the server's source code.
