[Pytables-users] Speed of CArray writing sparse matrices

[Giovanni L. C. <glc...@gm...>]

Hi all,

I have a sparse 3.4M x 3.4M adjacency matrix with nnz = 23M and wanted 
to see if CArray was an appropriate solution for storing it. Right now I 
am using the NumPy binary format for storing the data in coordinate 
format and loading the matrix with Scipy's sparse coo_matrix class. As 
far as I understand, with CArray the matrix would be written in full 
(zeros included) but a) since it's chunked accessing it does not take 
memory and b) with compression enabled it would possible to keep the 
size of the file reasonable.

If my assumptions are correct, then here is my problem: I am running 
into problems when writing the CArray to disk. I adapted the example 
from the documentation [1] and when I run the code on a 6000x6000 matrix 
with nnz = 17K I achieve a decent speed of roughly 4100 elements/s. 
However, when I try it on the full matrix the writing speed drops to 4 
elements/s. Am I doing something wrong? Any feedback would be greatly 
appreciated!

Code: https://gist.github.com/junkieDolphin/5843064

Cheers,

Giovanni

[1] 
http://pytables.github.io/usersguide/libref/homogenous_storage.html#the-carray-class

-- 
Giovanni Luca Ciampaglia

☞ http://www.inf.usi.ch/phd/ciampaglia/
✆ (812) 287-3471
✉ glc...@gm...