All of this is basically unchanged from VirtualDub 1.10
Enable Direct3D mode: it uses GPU to convert most formats in displayable form.
Enable Multi-monitor mode: otherwise Direct3D may be completely disabled if you put window on wrong monitor!
Use DirectX: just need it to reach Direct3D.
Use Direct3D 11: might be as good as Direct3D 9. Don't know any benefit.
Use OpenGL: almost same as no acceleration.
Use overlays: this has barely any advantage over normal Direct3D and often has problems in drivers.
Disable accelerated display: this option when enabled is wasting more than it is trying to save.