As a workaround you can try the vdub2 32bit version enabling the LAA (LargeAddressAware) using the extra\auxsetup.exe. In that way vdub2x32 will use 4gb of ram (instead of 2gb) on 64 bit systems solving all of the usual out of memory problems.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As a workaround you can try the vdub2 32bit version enabling the LAA (LargeAddressAware) using the extra\auxsetup.exe. In that way vdub2x32 will use 4gb of ram (instead of 2gb) on 64 bit systems solving all of the usual out of memory problems.