The type of the CMake variable for TJPEG_LIBRARY is incorrect, which confuses cmake-gui. Currently, its type, PATH, indicates that it is a directory, which means that cmake-gui won't let TJPEG_LIBRARY be set to "/path/to/libturbojpeg.a", only to the containing directory, "/path/to/" (at least when using cmake-gui's file dialog to set the variable).
To fix this, change the type of TJPEG_LIBRARY to "FILEPATH".
Fixed in both VirtualGL and TurboVNC. I should point out, though, that technically TJPEG_LIBRARY doesn't have to just be a file. It can be a set of linker flags as well, such as "-L{TurboJPEG_dir} -lturbojpeg". But since the most common usage is as a file, it makes sense for it to be FILEPATH rather than STRING.
I actually ended up having to change it to a STRING. Whenever you use FILEPATH, CMake will try to prepend the build directory to it, so setting -DTJPEG_LIBRARY=-lturbojpeg on the command line was causing the variable to end up as ~/src/vgl/linux64/-lturbojpeg in my case.