Menu
▾
▴
pytables-users
|
From: Francesc A. <fa...@py...> - 2004-09-29 11:51:58
|
Hi Nobbi, A Dimecres 29 Setembre 2004 11:57, vareu escriure: > Hi there, > > just starting to use pytables. I was rather surprised by the fact that flush() > does not write new tables to disk. Instead, I have to close() the file and > then reopen it again. Yes, this is is a known bug. This is solved in CVS version, and some people has made a patch for 0.8, and I guess it will apply well to 0.8.1 as well (see patch #947006 at the pytables project: http://sourceforge.net/tracker/?group_id=63486&atid=504146) > Maybe there is another way to go? My intention is to have one script that > opens an existing hdf5 datafile or creates it from scratch if it does not > exist and then starts a lengthy calculation saving the current state every > few minutes. Usually, the script would never finish but run indefinitely > until interrupted by Ctrl-C (therefore close() is never called) > > Another script would run interactively in parallel (maybe even on another > machine) reading the current state from disk and evaluating the data. I do think so. At least there should be someone in the pytables users list that are using pytables for doing similar things. > Is this setup possible with pytables at all? I read somewhere that HDF5 > supports parallel file access in some way. Does PyTables allow this as well? No, pytables does not support parallel file access as of now. Cheers, -- Francesc Alted |
|
From: Ivan V. i B. <iv...@ca...> - 2004-09-29 17:58:00
|
On Wed, Sep 29, 2004 at 01:51:32PM +0200, Francesc Alted wrote:
> A Dimecres 29 Setembre 2004 11:57, vareu escriure:
> [...]
> > Maybe there is another way to go? My intention is to have one script th=
at=20
> > opens an existing hdf5 datafile or creates it from scratch if it does n=
ot=20
> > exist and then starts a lengthy calculation saving the current state ev=
ery=20
> > few minutes. Usually, the script would never finish but run indefinitel=
y=20
> > until interrupted by Ctrl-C (therefore close() is never called)
> >=20
> > Another script would run interactively in parallel (maybe even on anoth=
er=20
> > machine) reading the current state from disk and evaluating the data.
>=20
> I do think so. At least there should be someone in the pytables users list
> that are using pytables for doing similar things.
> [...]
Anyway, I would suggest enclosing the loop in a =E2=80=9Ctry=E2=80=9D=
clause to
try and catch the KeyboardInterrupt exception, or maybe use a
=E2=80=9Cfinally=E2=80=9C clause to close the file in any situation. Bye!
--=20
Ivan Vilata i Balaguer @ =C2=A1Banda ancha asequible para Espa=C3=B1a ya!=
@
http://www.selidor.net/ @ http://www.petitiononline.com/prot1/ @
|
|
From: Norbert N. <Nor...@gm...> - 2004-09-30 07:32:44
|
On Wednesday 29 September 2004 20:10, Ivan Vilata i Balaguer wrote:
> On Wed, Sep 29, 2004 at 01:51:32PM +0200, Francesc Alted wrote:
> > A Dimecres 29 Setembre 2004 11:57, vareu escriure:
> > [...]
> >
> > > Maybe there is another way to go? My intention is to have one script
> > > that opens an existing hdf5 datafile or creates it from scratch if it
> > > does not exist and then starts a lengthy calculation saving the curre=
nt
> > > state every few minutes. Usually, the script would never finish but r=
un
> > > indefinitely until interrupted by Ctrl-C (therefore close() is never
> > > called)
> > >
> > > Another script would run interactively in parallel (maybe even on
> > > another machine) reading the current state from disk and evaluating t=
he
> > > data.
> >
> > I do think so. At least there should be someone in the pytables users
> > list that are using pytables for doing similar things.
> > [...]
>
> Anyway, I would suggest enclosing the loop in a =E2=80=9Ctry=E2=80=
=9D clause to
> try and catch the KeyboardInterrupt exception, or maybe use a
> =E2=80=9Cfinally=E2=80=9C clause to close the file in any situation. B=
ye!
Good idea, thanks!
Even though: I would strongly hope that it is not necessary. Does=20
h5file.close() do anything on the file but flush() it and then close the=20
physical file? In other words: is there any danger of file corruption if th=
e=20
python program is just shut down cold without a chance to save anything? Li=
ke=20
=2D when the file is written via NFS and the computer running the program i=
s=20
switched off hard?
=2D-=20
_________________________________________Norbert Nemec
Altdorfer Str. 9a ... D-93049 Regensburg
Tel: 0941 / 2009638 ... Mobile: 0179 / 7475199
eMail: <No...@Ne...>
|
|
From: Francesc A. <fa...@py...> - 2004-09-30 07:56:37
|
A Dijous 30 Setembre 2004 09:32, Norbert Nemec va escriure: > Even though: I would strongly hope that it is not necessary. Does > h5file.close() do anything on the file but flush() it and then close the > physical file? In other words: is there any danger of file corruption if the > python program is just shut down cold without a chance to save anything? Like > - when the file is written via NFS and the computer running the program is > switched off hard? I'm not completely sure, but muy guts say that this would not be a problem. If your program crashes (or is shut down) and there are buffers that are not written yet, then the file will miss this information, but will not get corrupted. The only problem may appear if you shut down the program *during* a write process. In that case I guess you will end with a corrupted file, as there are no provision of I/O transactions yet. Cheers, -- Francesc Alted |
|
From: Norbert N. <Nor...@gm...> - 2004-09-30 08:27:49
|
On Thursday 30 September 2004 09:56, Francesc Alted wrote:
> A Dijous 30 Setembre 2004 09:32, Norbert Nemec va escriure:
> > Even though: I would strongly hope that it is not necessary. Does
> > h5file.close() do anything on the file but flush() it and then close the
> > physical file? In other words: is there any danger of file corruption if
> > the python program is just shut down cold without a chance to save
> > anything? Like - when the file is written via NFS and the computer
> > running the program is switched off hard?
>
> I'm not completely sure, but muy guts say that this would not be a problem.
> If your program crashes (or is shut down) and there are buffers that are
> not written yet, then the file will miss this information, but will not get
> corrupted.
>
> The only problem may appear if you shut down the program *during* a write
> process. In that case I guess you will end with a corrupted file, as there
> are no provision of I/O transactions yet.
OK, that leaves me two specific questions:
* Does pytables ever initiate writes except in explicit flush() and close()
commands?
* Are the elementary operations of the underlying hdf5 library atomic? I.e.:
If I shut down the program in the middle of a pytables write action, might
the resulting file be fundamentally corrupted at the hdf5 level, or would the
corruption be well localized and reparable?
All leading to the third question:
* Might there be some simple way to make a set of changes atomic? Even if that
means restricting the use of pytables?
Ciao,
Nobbi
--
_________________________________________Norbert Nemec
Altdorfer Str. 9a ... D-93049 Regensburg
Tel: 0941 / 2009638 ... Mobile: 0179 / 7475199
eMail: <No...@Ne...>
|
|
From: Francesc A. <fa...@py...> - 2004-09-30 09:01:14
|
A Dijous 30 Setembre 2004 10:27, Norbert Nemec va escriure: > * Does pytables ever initiate writes except in explicit flush() and close() > commands? Well, there are two levels of cache buffers: table buffers in PyTables and HDF5 buffers. Now, PyTables triggers a write operation by calling an HDF5 function in two situations: 1) When table buffers are full 2) In any other write operation (writing attributes or *Array elements) then, is responsability of the underlying HDF5 library to write the data on disk. > * Are the elementary operations of the underlying hdf5 library atomic? I.e.: > If I shut down the program in the middle of a pytables write action, might > the resulting file be fundamentally corrupted at the hdf5 level, or would the > corruption be well localized and reparable? I don't really know that. I do know that HDF5 has a quite sophisticated cache system (both for data and metadata), but I know nothing about possible corruption scenarios and neither if they would be reparable or not. That might be a good question for the HDF5 list. Do you want me to ask this? Anyway, if you want to ask yourself, the HDF5 list is hd...@nc.... > All leading to the third question: > > * Might there be some simple way to make a set of changes atomic? Even > if that means restricting the use of pytables? By atomic you mean in a way that always let the file uncorrupted?. I just don't know, it depends on how HDF5 deals with this situations, but a priori my feeling is negative. Cheers, -- Francesc Alted |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X