#11 Support Mac OS X vnc client with no password
Currently in order for the Mac OS X built-in VNC client to connect to a LibVNCServer running with no password, the server must be started so that it advertises an RFB version of 3.6 or less. If LibVNCServer advertises a server version of 3.7 or greater and has no password, Apple's built-in VNC client will hang while connecting.
The attached patch supports connections from the Mac OS X built-in VNC client to LibVNCServers running with no password and advertising a server version of 3.7 or greater.
Some software (VBoxHeadless for example) does not allow passing the server version in to LibVNCServer. With this patch the built-in Mac OS X VNC client can successfully connect to VBoxHeadless instances running with no password.
Note: See http://secrets.blacktree.com/edit?id=46761 for information on how to allow the OS X built-in VNC client to connect to the same machine it's running on.
DETAILS:
When the Mac OS X built-in client connects to a VNC server advertising version 3.7 or later with no password, it correctly selects the security type none, but then expects no SecurityResult response and furthermore, in this case it skips sending the ClientInit message (while expecting an implicit value of true for shared-flag) and immediately expects to receive the ServerInit message. It only skips sending the ClientInit in this one case (server is 3.7 or later and security type is none). Without this patch, the Mac OS X VNC client hangs waiting for the ServerInit message while the LibVNCServer is waiting for the ClientInit message before sending the ServerInit message. Since the Mac OS X VNC client sends a unique RFB minor version (889) it's easy to support this as a special case without impacting any other clients.
Hi Kyle!
Looks okay, but why do you introduce a new client state called RFB_INITIALISATION_SHARED? Wouldn't it be enough to just act depending cl->protocolMinorVersion?
And could you spice up the patch with some explanatory comments for the afterworld?
Thanks!
I assume you mean have rfbProcessClientInitMessage check protocolMinorVersion and skip reading the ClientInit message in that case.
That will not work since rfbClientPtr does not have a selected security type field and the skip must only occur if the protocolMinorVersion is 889 AND the server sent a security type list AND the server received a choice of none back. The new client state reflects that combination.
Is there some way to check that combination in rfbProcessClientInitMessage other than by introducing a new field/state/flag?
A new client state representing a state part of the way into initialization (we've already received a ClientInit flag of SHARED, but haven't finished initialization yet [by sending the server info]) seemed cleanest to me.
The current version of the patch was created for MacPorts where no one sees it (or the lack of comments).
I'll work on an updated patch with more comments.
Right, seems there'sno other way to reflect that combination. So if you add some explanatory in-code comments, I think this can be commited and pushed.
Thanks!
Version 2 of patch against LibVNCServer-0.9.8.2
I've attached an updated version 2 of the patch with plenty of comments.
It also has a minor change over the old patch to guarantee ABI compatibility with previous versions (moved the new enum state down so that RFB_NORMAL does not change its value and now also prevents any external software from ever seeing the new RFB_INITIALISATION_SHARED state).
I've tested it against the built-in OS X client and other clients running both with and without a password any everything works like it should.
A shared-library version of the LibVNCServer built with this patch should be a drop-in replacement for any version without the patch thus providing the new support while retaining 100% ABI backwards compatibility.
Version 2 of patch as git format-patch against master
I've also attached a git format-patch copy of the version 2 patch against master.
Thanks, that's a LOT of comments ;-) - committed and pushed to master.