Re: [Pytables-users] .h5 Files robustness

[Francesc A. <fa...@ca...>]

Hi Francesco,

A Dissabte 08 Octubre 2005 10:06, Francesco Del Degan va escriure:
> I would like to write some hdf5/pytables extension to recovery corrupted
> files, in your opinion it's wasted work?

Well, I can't tell you exactly if this is hard or not, because I
simply don't know too much about the intrincacies of HDF5 in this
regard. You should look at:

https://sourceforge.net/mailarchive/forum.php?forum_id=3D13760&max_rows=3D2=
5&style=3Dnested&viewmonth=3D200409

for some messages exchanged about this issue back in September 2004.
If you read them, you will see that the main problem is the existence
of caches in the HDF5 library for both data and metadata. Those caches
can be disabled for bettering the chances that the file does not get
corrupted, but indeed, at the cost of speed.

In any case, the recomendation by Norbert Nemec of flushing frequently
is good, but of course, you don't have to expect too much of it.

> It's obvious that this is possible only if i append data, because if i
> change rows, i need to track also rows changes in file (more difficult
> task).

The other suggestion by Norbert is intriguing and perhaps worth to try
it up. If you book a big space for your data, and then start to save
rows until the dataset is full, I'd bet that you will be minimizing
the changes of corrupting the metadata headers. CArray objects already
let's you to book a large chunk of space in file for your data. So,
what Norbert proposes is doing that (i.e. setting the metadata) and
fill this space afterwards; if something goes wrong later on, perhaps
some data would be lost, but the file might still be readable.

Despite that Table ojects does not support (yet) booking large amounts
of space without feeding it with data, I suggest you to try CArrays
first in order to see if what I said before actually minimize the
probability of corruption. If it does, then, it's just a matter of
allowing Table objects to support that capability as well.

> Thanks for your job,

You are welcome!

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"