From: Alain F. <ala...@fr...> - 2005-12-17 18:46:38
|
Hello Francesc, Yes indeed compression applies. But I was misleaded by the fact that I also tryed to simulate virtual len= gth=20 arrays using tables, and table compression ratio gives much more compress= ed=20 files. For simulating 1000 objects with an array containing 10 000 values, I had= a=20 file size of 1 353 Ko using tables, and 58 687 using VLarrays. Plus I tryed to experiment row modification in VLarray, with my algorythm= =20 define after and in attached prova.py I need to run a ptrepack on my h5=20 file, to come the optimum size. If you have some recommendations to implement such behavior, they are=20 welcome. Kind regards, Alain ------------------------------------------------------------------- Algorythm to modify row 5 000, in a VLarray of 30 000 rows: ------------------------------------------------------------------- a1 =3D array([5]) a2 =3D array([5, 6]) a3 =3D array([5, 6, 7]) for i in range(10000): vlarray.append(a1) vlarray.append(a2) vlarray.append(a3) # MODIFICATION OF A ROW AT 5000 IN THE VLARRAY tmp_vlarray =3D fileh.copyNode(where=3D'/vlarray1', newname=3D'tmp_vl_array', start=3D0, stop=3D5000) tmp_vlarray.append(array([1, 2, 3])) for i in range(5001,vlarray.nrows): tmp=3Dvlarray[i] tmp_vlarray.append(tmp) fileh.removeNode(where=3D'/', name=3D'vlarray1') fileh.renameNode(where=3D'/', newname=3D'vlarray1', name=3D'tmp_vl_ar= ray') ----- Original Message -----=20 From: "Francesc Altet" <fa...@ca...> To: <pyt...@li...> Cc: "Alain Fagot" <ala...@fr...> Sent: Friday, December 16, 2005 1:20 PM Subject: Re: [Pytables-users] Does compression applie to VLarray? Hello Alain, A Dijous 15 Desembre 2005 14:03, Alain Fagot va escriure: > Hello, > > Looking at pyTables 1.2 documentation, it seems to me that compression > should applies to VLarray. The File.createVLarray() method having a=20 > filters > parameters. > When experimenting it, I have no compression for VLarray. Using the attached script, I get: -rw-r--r-- 1 faltet faltet 1300528 Dec 16 13:10 prova.h5 -rw-r--r-- 1 faltet faltet 827440 Dec 16 13:09 prova-zlib1.h5 It does not achieve a lot of compression (maybe there is an issue related with variable length arrays management in HDF5), but it seems to work anyway. Regards, --=20 >0,0< Francesc Altet http://www.carabos.com/ V V C=E1rabos Coop. V. Enjoy Data "-" |