Menu

#1 Saving and communicating large arrays

1.0
open
None
2015-04-12
2015-04-12
No

I want to propose a couple improvements that might help in applications to larger matrices:

  1. numpy's save (load) instead of savetxt (loadtxt), much faster to write/read from binary format.
  2. mpi4py's uppercase methods comm.Scatter, comm.Gather, etc. (versus their lowercase counterparts). The lowercase methods use pickle to serialize the inputs for communication and this limits them to communicating objects <=2GB. On the other hand the uppercase methods are intended for communicating numpy arrays.

I'm willing to work on making these changes, at least in the lsdmap subpackage.

The reason I am proposing these changes is that I am trying to compute lsdmap on a trajectory of 1.5E6 frames by a combination of downsampling, lsdmap, and embedding using rbf.

Discussion


Log in to post a comment.

MongoDB Logo MongoDB