The current WinVNC does not work in service mode on Windows 7. As DRC says on http://article.gmane.org/gmane.network.vnc.tigervnc.user/759:
"There are major known issues with using the server on Windows 7-- service mode does not work properly, for instance, and it won't ever work without a complete rearchitecting of the code. Basically, the service needs to be separated into a different process from the GUI, since services under Win7 can no longer access GUI functions."
After running the installer, the service has been started properly. However, there's no password configured, and there's no easy way of configuring one: Selecting "Configure VNC Server" under the "service mode" folder on the start menu gives you an "insufficient permission" error.