pytables-users — PyTables users discussion list

[Pytables-users] using pickle strings in VLArray

[Stefan K. <pon...@ya...>]

I am trying to put a pickle string into a VLArray using VLStringAtom. 
If the string is larger than a certain amount, I get an error like this
( the full error is below )

ValueError: Problems when converting the object '[snip]' to the
encoding 'utf-8'. The error was: 'ascii' codec can't decode byte 0x86
in position 4: ordinal not in range(128)
 
If it is a very short pickle string is works, but perhaps that is a red
herring. Here is the example code..

from tables import *
import cPickle

fp = openFile( 'tmp.h5', 'w' )
vlarray = fp.createVLArray( fp.root,
                            'test',
                            VLStringAtom())
vlarray.append( cPickle.dumps( ['aaa' * 130], 1 ))


and here is the full error ( displaying the pickle string )

 File "/apps/lib/python2.4/site-packages/tables/VLArray.py", line 327,
in append
    raise ValueError, "Problems when converting the object '%s' to the
encoding 'ut\
f-8'. The error was: %s" % (object, value)
ValueError: Problems when converting the object
']q^AT\206^A^@^@aaaaaaaaaaaaaaaaaaa\
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaq^Ba.' to the encoding 'utf-8'.
The error wa\
s: 'ascii' codec can't decode byte 0x86 in position 4: ordinal not in
range(128)

thanks,
Stefan


		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

Re: [Pytables-users] RFC: Natural naming proposal for PyTables 2.

[Francesc A. <fa...@ca...>]

A Dissabte 15 Octubre 2005 15:25, Norbert Nemec va escriure:
> What is wrong about checking whether a node is bound and say
> "manipulations to unbound nodes not yet implemented" if people to
> anything to them before they are assigned to a on-disk group?

Indeed, it's a possibility.

> Even if unbound nodes are still a far way off, there is nothing wrong
> about following the design idea now. I think the idea of unbound nodes
> is something very clear to understand for the user, even if - for the
> time being - these nodes are seriously limited until they are actually
> written to disk.
>
> Of course, checking whether a node is bound costs a tiny bit of
> performance, but that certainly can be minimized.

One can always use an "assert" instructuction, and if maximum speed is
needed, pass the -O option to python.

What other people think? Implementing this would improve readability
of the code? My opinion is starting to change and I think it does.

Cheers,

=2D-=20
>0,0<   Francesc Altet =C2=A0 =C2=A0 http://www.carabos.com/
V   V   C=C3=A1rabos Coop. V. =C2=A0=C2=A0Enjoy Data
 "-"

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Norbert N. <Nor...@gm...>]

If I understand correctly, Ubuntu just takes packages from
Debian-testing automatically into universe.

The reasons why pytables-1.1.1 is not in testing yet can be found at:
    http://bjorn.haxx.se/debian/testing.pl?package=3Dpytables

Seems like a jungle of dependencies on various architectures. Ironically
pytables waits for python-numarray, which again conflicts with
python2.2-tables. (Which should be resolved automatically, when the new
pytables is promoted)

So it seems, the hdf5 problem indirectly even stalls python-numarray...

Further checking seems to indicate that the ultimate problem is in xmpi,
which has been rebuilt only 6 days ago. So it seems that the problem is
being worked on, but in general, it may take a long time before a
dependency jungle like this is worked out on Debian... :-(


Francesc Altet wrote:

>A Dijous 13 Octubre 2005 10:39, v=E0reu escriure:
> =20
>
>>That's peculiar:
>>
>>$ cat /etc/issue
>>Ubuntu 5.10 "Breezy Badger" \n \l
>>
>>(the release candidate, actually)
>>
>>$ apt-cache show python-tables
>>Package: python-tables
>>Priority: optional
>>Section: universe/python
>>Installed-Size: 1024
>>Maintainer: Francesc Alted <fa...@py...>
>>...
>>
>>It looks like the maintainer is... you! :-) How can this be?
>>   =20
>>
>
>Oh, really? This was completely unexpected for me. Perhaps this is due
>to that I'm the maintainer for Debian, and because Ubuntu is so close
>to Debian the package my name remains as the original maintainer.
>
>Anyway, PyTables 1.1.1 depends on HDF5 1.6.4 and I'm not sure whether
>this is in Ubuntu or not, but I strongly suspect that it is not (it
>should be 1.6.2 instead). HDF5 1.6.4 is even having problems to enter
>testing distribution in Debian (they are waiting for a recent lam
>package there).
>
>So, all in all, I don't think that PyTables would be ready for Ubuntu
>Breezy anyway. Some time ago, I've received some patch from an Ubuntu
>developer (Sebastian Dr=C3ge <sl...@ub...>). Maybe someone might
>like to get in touch with him.
>
>Cheers,
>
> =20
>

Re: [Pytables-users] RFC: Natural naming proposal for PyTables 2.

[Norbert N. <Nor...@gm...>]

What is wrong about checking whether a node is bound and say
"manipulations to unbound nodes not yet implemented" if people to
anything to them before they are assigned to a on-disk group?

Even if unbound nodes are still a far way off, there is nothing wrong
about following the design idea now. I think the idea of unbound nodes
is something very clear to understand for the user, even if - for the
time being - these nodes are seriously limited until they are actually
written to disk.

Of course, checking whether a node is bound costs a tiny bit of
performance, but that certainly can be minimized.

Greetings,
Norbert


Francesc Altet wrote:

>A Dijous 13 Octubre 2005 00:46, Norbert Nemec va escriure:
>  
>
>>OK, this opens a different topic: unbound nodes. Personally, I think
>>that creating a node like
>>
>>    h5file.root.somenode = Array(somedata)
>>
>>is the most natural way to do the job. It is always said that unbound
>>nodes should be avoided. I believe, however, that this is a move in the
>>wrong direction.
>>    
>>
>
>I also do like *very much* this way of creating nodes. However, as you
>already said on a previous message, the problem is what it is supposed
>that the user is going to do with such an unbound node other than
>bounding it to a node on a file. If we could force something like:
>
>     h5file.root.somenode = Array(somedata)
>
>as the only thing that is allowed to the user, then I'd gladly vote
>for this. However, I'm afraid that we can't prevent the user doing
>things like:
>
>node = EArray(somedata)
>node.append(someotherdata)
><lot of bad message errors appearing....>
>  
>
>So, at least until nodes in-memory would be supported, I don't think
>this is going to be a good idea. You can still use it, if you want,
>but documenting it and stimulating its use for every kind of user
>could be an unnecessary source of confusion, IMO.
>
>Cheers,
>
>  
>

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Antonio V. <val...@co...>]

Alle 13:18, gioved=C3=AC 13 ottobre 2005, Ivan Vilata i Balaguer ha scritto:

> You should install the ``dpkg-dev`` and ``fakeroot`` packages, and once
> you have the three source files of the PyTables Debian package, just
> ``dpkg-source -x python-tables_....dsc``, enter the newly created
> directory and ``dpkg-buildpackage -b -rfakeroot``.  You may need to
> install additional development packages.  Once you are done, you will
> get the nice *.deb packages ready to install with ``dpkg -i``.  You can
> get more information in deb...@li....

Thank you for the info.

=2D-=20
Antonio Valentino
Consorzio Innova
via della Scienza - Zona Paip I
75100 Matera (MT) Italy

Re: [Pytables-users] RFC: Natural naming proposal for PyTables 2.

[Francesc A. <fa...@ca...>]

A Dijous 13 Octubre 2005 00:46, Norbert Nemec va escriure:
> OK, this opens a different topic: unbound nodes. Personally, I think
> that creating a node like
>
>     h5file.root.somenode =3D Array(somedata)
>
> is the most natural way to do the job. It is always said that unbound
> nodes should be avoided. I believe, however, that this is a move in the
> wrong direction.

I also do like *very much* this way of creating nodes. However, as you
already said on a previous message, the problem is what it is supposed
that the user is going to do with such an unbound node other than
bounding it to a node on a file. If we could force something like:

     h5file.root.somenode =3D Array(somedata)

as the only thing that is allowed to the user, then I'd gladly vote
for this. However, I'm afraid that we can't prevent the user doing
things like:

node =3D EArray(somedata)
node.append(someotherdata)
<lot of bad message errors appearing....>

So, at least until nodes in-memory would be supported, I don't think
this is going to be a good idea. You can still use it, if you want,
but documenting it and stimulating its use for every kind of user
could be an unnecessary source of confusion, IMO.

Cheers,

=2D-=20
>0,0<   Francesc Altet =C2=A0 =C2=A0 http://www.carabos.com/
V   V   C=C3=A1rabos Coop. V. =C2=A0=C2=A0Enjoy Data
 "-"

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Nicola L. <ni...@te...>]

> Anyway, I'm going to see if I can accelerate the process of making
> pytables 1.1.1 to enter testing.  I don't know if we would be in time
> for Breezy though.

Well, the announcements just came out, so no, we wouldn't. :-)

Anyway, pytables is in the "universe" repository, not in the main one, so
it isn't included in the release anyway. I don't what the versioning
guidelines for "universe" are, maybe there's still a possibility to put it
in. Otherwise, we'll have to wait for the next release.

-- 
Nicola Larosa - ni...@te...

...I sometimes think it would be nice to be a Tom Cruise or Mel Gibson
but like Rome, it's probably a nice place to visit but I wouldn't want
to live there. -- jgoldblog, May 2002

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Francesc A. <fa...@ca...>]

I'm thinking now that Antonio should not bother too much about getting
pytables 1.1.1 ready in Ubuntu. I think it would be preferable to wait
until pytables 1.1.1 make its way to Debian testing, and then ask the
Ubuntu developers to include the new pytables in testing.

Anyway, I'm going to see if I can accelerate the process of making
pytables 1.1.1 to enter testing.  I don't know if we would be in time
for Breezy though.

Cheers,

A Dijous 13 Octubre 2005 13:48, Francesc Altet va escriure:
> Antonio,
>
> Also, have a look at this:
>
> http://www.debian.org/doc/manuals/maint-guide/index.en.html
>
> in case you want to create good packages that would susceptible of
> being uploaded to distributions.
>
> A Dijous 13 Octubre 2005 13:18, Ivan Vilata i Balaguer va escriure:
> > En/na Valentino Antonio ha escrit::
> > > Hi Francesc,
> > > I recently switched to Ubuntu but I have no idea of how to build a
> > > debian package :(
> > > I have still a lot of things to learn about debian-based systems!
> > > Anyway if you think I can help you in some way please send me some
> > > instruction.
> >
> > You should install the ``dpkg-dev`` and ``fakeroot`` packages, and once
> > you have the three source files of the PyTables Debian package, just
> > ``dpkg-source -x python-tables_....dsc``, enter the newly created
> > directory and ``dpkg-buildpackage -b -rfakeroot``.  You may need to
> > install additional development packages.  Once you are done, you will
> > get the nice *.deb packages ready to install with ``dpkg -i``.  You can
> > get more information in deb...@li....
> >
> > Good luck!
> >
> >
> >
> > 	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
> > 	       C=E1rabos Coop. V.  V  V   Enjoy Data
> > 	                          ""

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Francesc A. <fa...@ca...>]

Antonio,

Also, have a look at this:

http://www.debian.org/doc/manuals/maint-guide/index.en.html

in case you want to create good packages that would susceptible of
being uploaded to distributions.

A Dijous 13 Octubre 2005 13:18, Ivan Vilata i Balaguer va escriure:
> En/na Valentino Antonio ha escrit::
> > Hi Francesc,
> > I recently switched to Ubuntu but I have no idea of how to build a
> > debian package :(
> > I have still a lot of things to learn about debian-based systems!
> > Anyway if you think I can help you in some way please send me some
> > instruction.
>
> You should install the ``dpkg-dev`` and ``fakeroot`` packages, and once
> you have the three source files of the PyTables Debian package, just
> ``dpkg-source -x python-tables_....dsc``, enter the newly created
> directory and ``dpkg-buildpackage -b -rfakeroot``.  You may need to
> install additional development packages.  Once you are done, you will
> get the nice *.deb packages ready to install with ``dpkg -i``.  You can
> get more information in deb...@li....
>
> Good luck!
>
>
>
> 	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
> 	       C=E1rabos Coop. V.  V  V   Enjoy Data
> 	                          ""

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Ivan V. i B. <iv...@ca...>]

En/na Valentino Antonio ha escrit::

> Hi Francesc,
> I recently switched to Ubuntu but I have no idea of how to build a
> debian package :(
> I have still a lot of things to learn about debian-based systems!
> Anyway if you think I can help you in some way please send me some
> instruction.

You should install the ``dpkg-dev`` and ``fakeroot`` packages, and once
you have the three source files of the PyTables Debian package, just
``dpkg-source -x python-tables_....dsc``, enter the newly created
directory and ``dpkg-buildpackage -b -rfakeroot``.  You may need to
install additional development packages.  Once you are done, you will
get the nice *.deb packages ready to install with ``dpkg -i``.  You can
get more information in deb...@li....

Good luck!

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       C=E1rabos Coop. V.  V  V   Enjoy Data
	                          ""

Re: [Pytables-users] save image in an hdf5 file

[Francesc A. <fa...@ca...>]

Hi Philippe,

A Dimecres 12 Octubre 2005 15:13, phi...@ho... va escriure:
> The Hdf5 library is able to store image (file with .png, .jpg extension
> i guess).

Well, HDF5 has a high level API for graphics:

http://hdf.ncsa.uiuc.edu/HDF5/hdf5_hl/doc/RM_hdf5im.html

But this is not meant for dealing with .png or .jpg formats, but
rather with raw RGB image formats. In any case, this API is not
supported yet on PyTables.

> Is there some routines in pytables to do this easily?

If you want to use PyTables to save images right now you have a couple
of possibilities:

1.- If you want to save pure PNG or JPG or whatever format natively,
use the FileNode module:

http://pytables.sourceforge.net/html-doc/usersguide5.html

Of course, your application will have to provide the machinery to
decode this afterwards.

2.- If you want to use native RGB codifications, use an *Array
container with several dimensions (one per color channel) in order to
keep it.

HTH,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Francesc A. <fa...@ca...>]

A Dijous 13 Octubre 2005 10:39, v=E0reu escriure:
> That's peculiar:
>
> $ cat /etc/issue
> Ubuntu 5.10 "Breezy Badger" \n \l
>
> (the release candidate, actually)
>
> $ apt-cache show python-tables
> Package: python-tables
> Priority: optional
> Section: universe/python
> Installed-Size: 1024
> Maintainer: Francesc Alted <fa...@py...>
> ...
>
> It looks like the maintainer is... you! :-) How can this be?

Oh, really? This was completely unexpected for me. Perhaps this is due
to that I'm the maintainer for Debian, and because Ubuntu is so close
to Debian the package my name remains as the original maintainer.

Anyway, PyTables 1.1.1 depends on HDF5 1.6.4 and I'm not sure whether
this is in Ubuntu or not, but I strongly suspect that it is not (it
should be 1.6.2 instead). HDF5 1.6.4 is even having problems to enter
testing distribution in Debian (they are waiting for a recent lam
package there).

So, all in all, I don't think that PyTables would be ready for Ubuntu
Breezy anyway. Some time ago, I've received some patch from an Ubuntu
developer (Sebastian Dr=C3ge <sl...@ub...>). Maybe someone might
like to get in touch with him.

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Valentino A. <val...@co...>]

----- Original Message ----- 
From: "Francesc Altet" <fa...@ca...>
To: <pyt...@li...>
Cc: "Nicola Larosa" <ni...@te...>
Sent: Thursday, October 13, 2005 10:23 AM
Subject: Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?


>A Dimecres 12 Octubre 2005 17:38, Nicola Larosa va escriure:
>> Hi everybody,
>> while the PyTables packages in Debian unstable are at v.1.1.1, the ones 
>> in
>> Ubuntu Breezy are still at 0.9.1, how come?
>>
>> Breezy is due any day now, I don't think anybody would like an old 
>> PyTables
>> version in it, right? :-)
>
>That would be a good thing, of course. Here in Cárabos we don't have
>resources enough to provide support for every distribution. Some
>volunteer for asking the Ubuntu maintainer about that?

Hi Francesc,
I recently switched to Ubuntu but I have no idea of how to build a debian 
package :(
I have still a lot of things to learn about debian-based systems!
Anyway if you think I can help you in some way please send me some 
instruction.

Ciao
Antonio

--
Antonio VALENTINO
Consorzio Innova
via della Scienza - Zona Paip I
75100 Matera (MT) Italy

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Nicola L. <ni...@te...>]

>> Hi everybody,
>> while the PyTables packages in Debian unstable are at v.1.1.1, the ones in
>> Ubuntu Breezy are still at 0.9.1, how come?
>> 
>> Breezy is due any day now, I don't think anybody would like an old PyTables
>> version in it, right? :-)

> That would be a good thing, of course. Here in Cárabos we don't have
> resources enough to provide support for every distribution. Some
> volunteer for asking the Ubuntu maintainer about that?

That's peculiar:

$ cat /etc/issue
Ubuntu 5.10 "Breezy Badger" \n \l

(the release candidate, actually)

$ apt-cache show python-tables
Package: python-tables
Priority: optional
Section: universe/python
Installed-Size: 1024
Maintainer: Francesc Alted <fa...@py...>
...

It looks like the maintainer is... you! :-) How can this be?

-- 
Nicola Larosa - ni...@te...

...I sometimes think it would be nice to be a Tom Cruise or Mel Gibson
but like Rome, it's probably a nice place to visit but I wouldn't want
to live there. -- jgoldblog, May 2002

Re: [Pytables-users] What about PyTables version in Ubuntu Breezy?

[Francesc A. <fa...@ca...>]

A Dimecres 12 Octubre 2005 17:38, Nicola Larosa va escriure:
> Hi everybody,
> while the PyTables packages in Debian unstable are at v.1.1.1, the ones in
> Ubuntu Breezy are still at 0.9.1, how come?
>
> Breezy is due any day now, I don't think anybody would like an old PyTabl=
es
> version in it, right? :-)

That would be a good thing, of course. Here in C=E1rabos we don't have
resources enough to provide support for every distribution. Some
volunteer for asking the Ubuntu maintainer about that?

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Pytables-users] RFC: Natural naming proposal for PyTables 2.

[Norbert N. <Nor...@gm...>]

Ivan Vilata i Balaguer wrote:

>>One idea: would it make sense to consider __getitem__/__setitem__ as
>>alternative access method for children? It cannot offer tab-completion,
>>but for non-interactive use, it might come in handy.
>>    
>>
>
>I have been quite convinced for a while that this would be nice to have
>in PyTables, but I am not so sure now.  At first, ``__setitem__()``
>would require unbound nodes, and idea we do not like much (this is why
>``__setattr__()`` of nodes no longer works).
>
OK, this opens a different topic: unbound nodes. Personally, I think
that creating a node like

    h5file.root.somenode = Array(somedata)

is the most natural way to do the job. It is always said that unbound
nodes should be avoided. I believe, however, that this is a move in the
wrong direction.

Well - matter of taste and of discussion. In the end it is up to you to
decide.

As for the original question: should child nodes be accessible via
getitem/setitem? Thinking about your words, I notice that the
Python-principle actually answers the question: "There should be one -
and only one - obvious way to do the job." Guess, my proposal is killed
by exactly that principle.

True, we already have two ways of accessing child nodes. Those are well
distinguished and both have their purpose. My alternative might at best
save some typing. I'm withdrawing my proposal.

Greetings,
Norbert

[Pytables-users] What about PyTables version in Ubuntu Breezy?

[Nicola L. <ni...@te...>]

Hi everybody,
while the PyTables packages in Debian unstable are at v.1.1.1, the ones in
Ubuntu Breezy are still at 0.9.1, how come?

Breezy is due any day now, I don't think anybody would like an old PyTables
version in it, right? :-)

-- 
Nicola Larosa - ni...@te...

...I sometimes think it would be nice to be a Tom Cruise or Mel Gibson
but like Rome, it's probably a nice place to visit but I wouldn't want
to live there. -- jgoldblog, May 2002

[Pytables-users] save image in an hdf5 file

[<phi...@ho...>]

Hi fransesc,

The Hdf5 library is able to store image (file with .png, .jpg extension 
i guess).
Is there some routines in pytables to do this easily?

regards,
Philippe Collet

Re: [Pytables-users] RFC: Natural naming proposal for PyTables 2.

[Ivan V. i B. <iv...@ca...>]

En/na Norbert Nemec ha escrit::

> The proposal sounds reasonable to me. The only "uglification" that it
> introduces is the trailing underscore for regular items. I think people=

> can bear that. All the escaping should probably not appear in everyday
> use. (Unless somebody wants to punish him/herself.) It certainly is goo=
d
> to have the predictability and the full power of tab-expansion.

The trailing underscore may be ugly but I find it quite interesting that
it makes child nodes instantly distinguishable from instance members.
As you noted, escaping will not happen in most cases, since people tends
to use id-like names.  We people using accents and such in computing
know we will come across trouble sooner or later... :-/

En/na Norbert Nemec ha escrit::

> One idea: would it make sense to consider __getitem__/__setitem__ as
> alternative access method for children? It cannot offer tab-completion,=

> but for non-interactive use, it might come in handy.

I have been quite convinced for a while that this would be nice to have
in PyTables, but I am not so sure now.  At first, ``__setitem__()``
would require unbound nodes, and idea we do not like much (this is why
``__setattr__()`` of nodes no longer works).  Then, I have developed
some ideas against using the item interface:

* I think the semantics of items is not related with the semantics of
  child nodes, so it may not be obvious to the user what
  ``group['nodename']`` does.

* The natural naming interface is related with interactive usage, and
  programmatic usage would already have ``getChild()`` and similarly
  named methods.  These are already two ways of doing the same thing, a
  third one is only confusing.

I *really* would like to discuss these points a little more.  Should
groups, in your opinion, behave more like dictionaries?  If that was the
case, the item interface would make more sense, but then node attributes
would look more strange to be placed in a separate ``attrs`` object.

Cheers,

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       C=C3=A1rabos Coop. V.  V  V   Enjoy Data
	                          ""

Re: [Pytables-users] RFC: Natural naming proposal for PyTables 2.

[Norbert N. <Nor...@gm...>]

The proposal sounds reasonable to me. The only "uglification" that it
introduces is the trailing underscore for regular items. I think people
can bear that. All the escaping should probably not appear in everyday
use. (Unless somebody wants to punish him/herself.) It certainly is good
to have the predictability and the full power of tab-expansion.

One idea: would it make sense to consider __getitem__/__setitem__ as
alternative access method for children? It cannot offer tab-completion,
but for non-interactive use, it might come in handy.





Ivan Vilata i Balaguer wrote:

>Hi list!  I'm posting this document as a Request For Comments for a
>possible new natural naming approach for a future version of PyTables.
>Please note that this is *not* going to be implemented in a near future,=

>but I would like to know your opinion indeed, to know if we will be
>moving in the right direction.  All criticism and opinions welcome,
>thank you!
>
>::
>
>	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
>	       C=C3=A1rabos Coop. V.  V  V   Enjoy Data
>	                          ""
> =20
>

Re: [Pytables-users] .h5 Files robustness

[Francesc A. <fa...@ca...>]

Hi Francesco,

A Dissabte 08 Octubre 2005 10:06, Francesco Del Degan va escriure:
> I would like to write some hdf5/pytables extension to recovery corrupted
> files, in your opinion it's wasted work?

Well, I can't tell you exactly if this is hard or not, because I
simply don't know too much about the intrincacies of HDF5 in this
regard. You should look at:

https://sourceforge.net/mailarchive/forum.php?forum_id=3D13760&max_rows=3D2=
5&style=3Dnested&viewmonth=3D200409

for some messages exchanged about this issue back in September 2004.
If you read them, you will see that the main problem is the existence
of caches in the HDF5 library for both data and metadata. Those caches
can be disabled for bettering the chances that the file does not get
corrupted, but indeed, at the cost of speed.

In any case, the recomendation by Norbert Nemec of flushing frequently
is good, but of course, you don't have to expect too much of it.

> It's obvious that this is possible only if i append data, because if i
> change rows, i need to track also rows changes in file (more difficult
> task).

The other suggestion by Norbert is intriguing and perhaps worth to try
it up. If you book a big space for your data, and then start to save
rows until the dataset is full, I'd bet that you will be minimizing
the changes of corrupting the metadata headers. CArray objects already
let's you to book a large chunk of space in file for your data. So,
what Norbert proposes is doing that (i.e. setting the metadata) and
fill this space afterwards; if something goes wrong later on, perhaps
some data would be lost, but the file might still be readable.

Despite that Table ojects does not support (yet) booking large amounts
of space without feeding it with data, I suggest you to try CArrays
first in order to see if what I said before actually minimize the
probability of corruption. If it does, then, it's just a matter of
allowing Table objects to support that capability as well.

> Thanks for your job,

You are welcome!

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

[Pytables-users] RFC: Natural naming proposal for PyTables 2.

[Ivan V. i B. <iv...@ca...>]

Hi list!  I'm posting this document as a Request For Comments for a
possible new natural naming approach for a future version of PyTables.
Please note that this is *not* going to be implemented in a near future,
but I would like to know your opinion indeed, to know if we will be
moving in the right direction.  All criticism and opinions welcome,
thank you!

::

	Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
	       Cárabos Coop. V.  V  V   Enjoy Data
	                          ""

Re: [Pytables-users] .h5 Files robustness

[Norbert N. <Nor...@gm...>]

Hi there,

I did some web-research on the topic a little while ago, but I could not
find any details about it:

As it seems, the HDF5 library does not take any precautions about
possible power failures. Modern database systems usually have a concepts
of "atomic operations" and "journalling" that protect the data against
corruption. HDF5 does not provide this, so pytables is out of luck.
There is no perfectly safe way of handling a HDF5 file.

What I usually do is to limit the chance of breaking by doing write
operations all at once at the end of a big calculation loop and flush
the data immediately. Of course, this only makes sense for programs
(like mine) where outputting data takes a negligible fraction of the
total computation time. The chance that the program breaks in this short
while is negligible.

Alternatively, it might be possible to carefully select the data
structures in such a way that a write operation never changes the
control data in the file. (i.e. no dynamic data structures, no growing
tables, etc.) this way, it should be possible to write the data in such
a way that the file is in a sane conditition at any point of time, no
matter where it breaks half-way in an operation.

Data compression should probably be avoided in any case in such a situation.

Of course, in either way, you throw away much of the power of pytables,
but unfortunately, that's the price you have to pay for safety.

Greetings,
Norbert




Francesco Del Degan wrote:

>Hi, i've a question:
>
>Under certain circumstances, i can running into corrupting of .h5 file,
>for instance when machine crash, power failure, etc... , causing
>lost of all data (i'm working into gigabytes files).
>
>Perhaps this is an hdf5 question, but i would like to know in what manner
>do you manage this in practice.
>
>Some considerations:
>
>1) Copying/rsyncing the file before writing is possible but is very very
>slow on big files.
>2) Truncating the file to size before writing doesn't resolve the problem,
>hdf5 remains corrupted (the header is changed, hdf5 isn't pure append file)
>
>It's possible to implement a recovery/checkpoint system?
>
>I've noticed that hdf5 uses some headers in file
>(http://hdf.ncsa.uiuc.edu/HDF5/doc/H5.format.html),
>if i try to save (i need to study hdf5 file format) headers before
>writing, for instance
>in a master register/file, i can achieve recovery?
>
>I would like to write some hdf5/pytables extension to recovery corrupted
>files, in your
>opinion it's wasted work?
>
>It's obvious that this is possible only if i append data, because if i
>change
>rows, i need to track also rows changes in file (more difficult task).
>
>Thanks for your job,
>Francesco
>
>
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by:
>Power Architecture Resource Center: Free content, downloads, discussions,
>and more. http://solutions.newsforge.com/ibmarch.tmpl
>_______________________________________________
>Pytables-users mailing list
>Pyt...@li...
>https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
>  
>

[Pytables-users] .h5 Files robustness

[Francesco D. D. <ke...@li...>]

Hi, i've a question:

Under certain circumstances, i can running into corrupting of .h5 file,
for instance when machine crash, power failure, etc... , causing
lost of all data (i'm working into gigabytes files).

Perhaps this is an hdf5 question, but i would like to know in what manner
do you manage this in practice.

Some considerations:

1) Copying/rsyncing the file before writing is possible but is very very
slow on big files.
2) Truncating the file to size before writing doesn't resolve the problem,
hdf5 remains corrupted (the header is changed, hdf5 isn't pure append file)

It's possible to implement a recovery/checkpoint system?

I've noticed that hdf5 uses some headers in file
(http://hdf.ncsa.uiuc.edu/HDF5/doc/H5.format.html),
if i try to save (i need to study hdf5 file format) headers before
writing, for instance
in a master register/file, i can achieve recovery?

I would like to write some hdf5/pytables extension to recovery corrupted
files, in your
opinion it's wasted work?

It's obvious that this is possible only if i append data, because if i
change
rows, i need to track also rows changes in file (more difficult task).

Thanks for your job,
Francesco

[Pytables-users] ANN: PyTables-1.2-pre2

[Francesc A. <fa...@ca...>]

Hi List,

We are in the final stages before releasing PyTables 1.2, which
offers, as I already said here, a new LRU cache for keeping only the
most used nodes in-memory, and not all the nodes as before.

One of the main consequences of that is very fast opening of files,
iregardingly of how many nodes (groups + leaves) has on it. For
example, with PyTables 1.1.1, a file with 3000 nodes took 2.4 seconds
(5.5 seconds in PyTables 1.1) to be opened; now, PyTables 1.2 takes
just 0.03 seconds, which is very good news, specially for
interactively working with files with such a large number of nodes.

I'd liked to include the patch sent by travlr, to be able to use
indexing to get and set values in *Array, Table and Column objects
(i.e. "indexing a la numarray"); but provided he sent a patch just for
getting values in Table and Column objects and that we are rather busy
right now, we've decided to not incorporate this functionality until a
next version, when all the objects would receive complete support for
both getting and setting values "a la numarray".

You can download the current tarball from:

http://www.carabos.com/downloads/pytables/preliminary/pytables-1.2-pre2.tar=
=2Egz

Please, help us to test this new version before it would be released!

Have a nice weekend!

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"