#1473 Black image sent when connecting with a big-endian client at lower color pixel formats

[Vladislav Sorkin]

Bug:
When connecting to TightVNC with a big-endian client (destination), BLACK image is sent by the TightVNC server (source) when host is running in 16-bit color, when client (destination) requested pixel format is less than 32 bits per pixel (16 or 8bpp).

How to reproduce:

• On the host (source) running TightVNC server set screen color depth to 16-bit, where host is natively little-endian
• Connect using a big-endian VNC client (destination) with requested pixel format either 16bpp or 8bpp
• Observe black image sent from TightVNC server

I have built 2.8.11 from source and debugged the code, and found the (likely very old) bug in PixelConverter. The bug in the code is in fillHexBitsTable function, where at the end the endianness conversion flips byte order around for UINT32, same for all destination pixel formats. This is wrong. Doing this for 16bpp and 8bpp causes lowermost bits containing the shifted color bits to become uppermost, and then later in convert function all of the color bits get clipped off during casting to UINT16 or UINT8, producing nothing but zeroes (black color), resulting in all-black image to be sent to the client.

This appears to have been a very old bug as it has propagated to other projects forked from TightVNC (such as TigerVNC), and symptoms have been discussed on forums but the cause was not found before.

I have fixed the bug and attached the updated source file below. Please test and release a new installer build with this fix.

You are welcome, and thank you!

[Vladislav Sorkin]

Attaching diff file.

[Anton]

Hello.
Can you test this version?

[Vladislav Sorkin]

Tested. Works properly with a big-endian client where server host color depth is 16-bit, and client color depths are 32, 16, and 8-bit.

Thank you.