I want to propose a couple improvements that might help in applications to larger matrices:
- numpy's save (load) instead of savetxt (loadtxt), much faster to write/read from binary format.
- mpi4py's uppercase methods comm.Scatter, comm.Gather, etc. (versus their lowercase counterparts). The lowercase methods use pickle to serialize the inputs for communication and this limits them to communicating objects <=2GB. On the other hand the uppercase methods are intended for communicating numpy arrays.
I'm willing to work on making these changes, at least in the lsdmap subpackage.
The reason I am proposing these changes is that I am trying to compute lsdmap on a trajectory of 1.5E6 frames by a combination of downsampling, lsdmap, and embedding using rbf.