I looked at your suggestion;
The adjoint which is present in SparseMatrix of Eigen is actually the "conjugate transpose" or "hermitian transpose" of the matrix, not the "adjugate" in the sense of the "cofactors" matrix.
So this seems like a terminology misunderstanding here.
For large matrices it is more likely that what is needed is the conjugate transpose; see e.g. the conjugate_transpose() method of vnl_matrix<T>.
This can indeed be easily added to vnl_sparse_matrix<T>, together with a transpose() method.