There are lots of things that would require vnl and vgl to work together (ie rotating a vector by multiplying it by a matrix) : vnl_matrix<double> * vgl_vector_3d<double>.  I have written a whole bunch of "helpers" like vgl_vector_to_vnl_vector, vnl_vector_to_vgl_vector, vgl_point_to_vgl_vector, etc that are pretty nice to have around. Could something like these be included in vxl?