pytables-users

[Pytables-users] nctoh5 with lzo compression

[<fri...@am...>]

Thank you for optimizing the nctoh5 script! 

I have two questions concerning compression: 

1) The effect of zlib-compression
---------------------------------
Tests are made with a 30 MB nc-file resulting in h5 files of

156 MB without any compression
 16 MB with zlib compression level 9

...but using external zip or lzo program it results in approx.
  2 MB

(I get nearly the same result when I compress the original nc file.)

Is there a chance to get smaller h5 files out of nctoh5.py ?

2) internal lzo compression
---------------------------
I placed lzo.dll resp. lzo1.dll in ~system32 directory where zlib.dll and msvcrt.dll live too.
But I get a UserWarning from Leaf.py line 90: lzo compression library is not available...

What I have to do?

Thanks in advance,
Friedemann Gehrt

Re: [Pytables-users] nctoh5 with lzo compression

[Francesc A. <fa...@py...>]

A Divendres 05 Novembre 2004 15:15, fri...@am... va escriure:
> I have two questions concerning compression: 
> 
> 1) The effect of zlib-compression
> ---------------------------------
> Tests are made with a 30 MB nc-file resulting in h5 files of
> 
> 156 MB without any compression
>  16 MB with zlib compression level 9
> 
> ...but using external zip or lzo program it results in approx.
>   2 MB

That's weird... Which compression flags are you using? In particular, are
you using the shuffle filter?

My experience is that a compressor+shuffle *always* give better results than
using the compressor alone. Perhaps your data is an exception to this rule.
Can you try:

$ nctoh5 --complib=zlib --shuffle=1 ncfile h5file

and

$ nctoh5 --complib=zlib --shuffle=0 ncfile h5file

so as to see if shuffle is adding entropy to data instead of reducing it?

> 2) internal lzo compression
> ---------------------------
> I placed lzo.dll resp. lzo1.dll in ~system32 directory where zlib.dll and msvcrt.dll live too.
> But I get a UserWarning from Leaf.py line 90: lzo compression library is not available...
> 
> What I have to do?

Did you install the LZO & UCL aware PyTables auto-installable? I mean,
something like tables-0.8.1-LU.win32-py2.3.exe instead of
tables-0.8.1.win32-py2.3.exe

Cheers,

-- 
Francesc Altet