Fwd: Re: [Pytables-users] Danger of file corruption?

[Francesc A. <fa...@py...>]

Someone at the list of HDF5 has answered. It looks like the danger of
corruption is real. Matt suggest writing a number of dumps to avoid the
worst to happen.

Cheers,

-- 
Francesc Alted

----------  Missatge transmes  ----------

Subject: RE: Re: [Pytables-users] Danger of file corruption?
Date: Dijous 30 Setembre 2004 15:57
From: "Matthew Street" <Mat...@aw...>
To: fa...@py..., hd...@nc...


Hi,

My experience has been that the file does indeed become corrupted, and the
data in it unaccessable, if the code is terminated for some reason.  We have
found this to happen if the code crashes, is removed by a scheduler, and
also if the filesystem it is writing to becomes temporarily unavailable.
These kinds of issues would affect most file operations though - not just
HDF5 ones.

A lot of the users I support write timestep dumps, and I always advise
writing a number of dumps containing N timesteps, rather than putting them
all in one, just so that all data is not lost if the worst happens.  I
always get them to close and re-open the file between dumps too as leaving
it open can be dangerous - I think the H5Fclose routine needs to complete
properly for the file to be non-corrupt, if this is the case 'atomic'-type
operations probably wont help.

Hope this helps,

Matt


--
_______________________________________________________________________
Matt Street
Parallel Technology Support	Tel:	+44 (0) 118 982 4528
Applied Computer Science 	AWE, Aldermaston, Reading, RG7 4PR. UK.


> -----Original Message-----
> From: Francesc Alted
> Sent: 30 September 2004 12:42
> 
> 
> Hi,
> 
> A PyTables user has sent me couple of questions about HDF5 
> behaviour in case of a sudden sutdown (or crash) of the 
> program that is writing a file.
> 
> I'm forwarding the questions more related with HDF5, 
> toghether with my answers. Can anybody give us a bit of light 
> on this issues?
> 
> Thanks,
> 
> ----------  Missatge transmes  ----------
> 
> Subject: Re: [Pytables-users] Danger of file corruption? 
> (was: Bug? flush() ignores new tables)
> Date: Dijous 30 Setembre 2004 11:00
> From: Francesc Alted <fa...@py...>
> To: pyt...@li...
> 
> A Dijous 30 Setembre 2004 10:27, Norbert Nemec va escriure:
> > * Are the elementary operations of the underlying hdf5 
> library atomic? 
> > I.e.:
> > If I shut down the program in the middle of a pytables 
> write action, might 
> > the resulting file be fundamentally corrupted at the hdf5 
> level, or would the 
> > corruption be well localized and reparable?
> 
> I don't really know that.  I do know that HDF5 has a quite 
> sophisticated cache system (both for data and metadata), but 
> I know nothing about possible corruption scenarios and 
> neither if they would be reparable or not. That might be a 
> good question for the HDF5 list. Do you want me to ask this? 
> Anyway, if you want to ask yourself, the HDF5 list is 
> hd...@nc....
> 
> > All leading to the third question:
> > 
> > * Might there be some simple way to make a set of changes 
> atomic? Even
> > if that means restricting the use of pytables?
> 
> By atomic you mean in a way that always let the file 
> uncorrupted?. I just don't know, it depends on how HDF5 deals 
> with this situations, but a priori my feeling is negative.
> 
> Cheers,
> 
> -- 
> Francesc Alted
--
_______________________________________________________________________________

The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited.  Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer.  While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected.

AWE Plc
Registered in England and Wales
Registration No 02763902
AWE, Aldermaston, Reading, RG7 4PR



-------------------------------------------------------