pytables-users — PyTables users discussion list

[Pytables-users] Pytables file reading

[Juan M. V. T. <jmv...@gm...>]

Hello all,

I´m managing a file close to 26 Gb size. It´s main structure is  a table
with a bit more than 8 million rows. The table is made by four columns, the
first two columns store names, the 3rd one has a 53 items array in each
cell and the last column has a 133x6 matrix in each cell.
I use to work with a Linux workstation with 24 Gb. My usual way of working
with the file is to retrieve, from each cell in the 4th column of the
table, the same row from the 133x6 matrix.
I store the information in a bumpy array with shape 8e6x6. In this process
I almost use the whole workstation memory.
Is there anyway to optimize the memory usage?
If not, I have been thinking about splitting the file.

Thank you,

Juanma

Re: [Pytables-users] advice on data representation

[Anthony S. <sc...@gm...>]

On Wed, Aug 1, 2012 at 1:10 AM, <ben...@lf...> wrote:

> > There might be an easier way to do this with numpy dtypes.  In pseudo-
> > code:
> >
> > np.dtype([(colname, np.int16) for colname in colnames])
> >
>
> Can we use time and enum kinds that way as well?
>

Ahh you might not be able to use these types, depending on your numpy
version.   Instead I would use dictionary comprehensions of columns:

{colname: UInt16Col(pos=i) for i, colname in enumerate(colnames)}

This makes me think I should probably flatten my table.
> Having nested columns is quite natural to group cells under a data item
> (it's nice to access a field like I030_180/SPEED).
> But that's more a naming convention and I can't perform in-kernel searches
> with nested columns.
>

It would sure be nice if we could though ;)


>
> >
> > If you want to mark the whole column as valid, you can use a boolean
> > attribute on the table itself for each column.  They could be named
> > like colname_valid.
> >
> > See
> > http://pytables.github.com/usersguide/libref.html#tables.Leaf.setAttr
> > and http://pytables.github.com/usersguide/libref.html#the-attributeset-
> > class for more info.
>
> Good to know.
>
> >
> > This is more for flagging individual cells as valid or not.  For
> > integers you need  to pick a values which means invalid (like -999999).
> >
> >
>
> Yes, you are right. It's not a column I want to flag but a group of cells
> in a row (all cells of a particular data item).
> Problem is that if I have a uint8 for a cell, there is no invalid value I
> can use.
> Maybe I could use a "larger type" (uint16 in this case) to be able to pick
> an invalid value.
> Or a bool for this group of cells.
>

Yes, if you used a bool column next to it you could use this as a valid
mask.  Note that since bools are stored with a full byte, the bool column +
the uint8 column the same size as uint16.  However, it will be much quicker
to query over the bool column.

Be Well
Anthony


>
> >
> > If you don't want to use a VLArray, then maxlen is probably your best
> > option.
> >
> > If you want to do something a little more sophisticated, you could
> > break you data out into a main table and then a helper VLarray.  Every
> > row in the table is matched by the same row in vlarray.  Then when you
> > want to get your full data back out, you have to go to the table and
> > the vlarray.  This makes things a little more annoying to work with,
> > but it does what you want.
>
> Interesting. I'll look more into it.
>
> >
> > Hope this helps.  Feel free to ask more questions!
>
> Yes, it does.
> Thanks!
>
>
> Benjamin
>
>
> > Be Well
> > Anthony
> >
> >
> >
> >
> >
> >
> >       Cheers,
> >
> >
> >
> >       Benjamin
> >
> >
> >
> >
> >
> >       class I030_180_DESC(tables.IsDescription):
> >
> >           """Calculated Track Velocity (Polar)"""
> >
> >           SPEED = tables.UInt16Col(pos=0)
> >
> >           HEADING = tables.UInt16Col(pos=1)
> >
> >
> >
> >       class I030_181_DESC(tables.IsDescription):
> >
> >           """Calculated Track Velocity (Cartesian)"""
> >
> >           X = tables.Int16Col(pos=0)
> >
> >           Y = tables.Int16Col(pos=1)
> >
> >
> >
> >       class I030_340_DESC(tables.IsDescription):
> >
> >           """Last Measured Mode 3/A"""
> >
> >           V = tables.EnumCol(tables.Enum({
> >
> >               "Code validated": 0,
> >
> >               "Code not validated": 1,
> >
> >               "uninitialized": 255
> >
> >               }), "uninitialized",
> >
> >               base="uint8",
> >
> >               pos=0)
> >
> >           G = tables.EnumCol(tables.Enum({
> >
> >               "Default": 0,
> >
> >               "Garbled code": 1,
> >
> >               "uninitialized": 255
> >
> >               }), "uninitialized",
> >
> >               base="uint8",
> >
> >               pos=1)
> >
> >           L = tables.EnumCol(tables.Enum({
> >
> >               "MODE 3/A code as derived from the reply of the
> > transponder,": 0,
> >
> >               "Smoothed MODE 3/A code as provided by a local
> > tracker": 1
> >
> >               "uninitialized": 255
> >
> >               }), "uninitialized",
> >
> >               base="uint8",
> >
> >               pos=2)
> >
> >           sb = tables.UInt8Col(pos=3)
> >
> >           mode_3_a = tables.UInt16Col(pos=4)
> >
> >
> >
> >       class I030_400_DESC(tables.IsDescription):
> >
> >           """Callsign"""
> >
> >           callsign = tables.StringCol(7, pos=0)
> >
> >
> >
> >       class I030_050_DESC(tables.IsDescription):
> >
> >           """Artas Track Number"""
> >
> >           AUI = tables.UInt8Col(pos=0)
> >
> >           unused = tables.UInt8Col(pos=1)
> >
> >           STN = tables.UInt16Col(pos=2)
> >
> >           FX = tables.EnumCol(tables.Enum({
> >
> >               "end of data item": 0,
> >
> >               "extension into next extent": 1,
> >
> >               "uninitialized": 255
> >
> >               }), "uninitialized",
> >
> >               base="uint8",
> >
> >               pos=3)
> >
> >
> >
> >       class I030Record(tables.IsDescription):
> >
> >           """Cat 030 record"""
> >
> >           ff_timestamp = tables.Time32Col()
> >
> >           I030_010 = I030_010_DESC()
> >
> >           I030_015 = I030_015_DESC()
> >
> >           I030_030 = I030_030_DESC()
> >
> >           I030_035 = I030_035_DESC()
> >
> >           I030_040 = I030_040_DESC()
> >
> >           I030_070 = I030_070_DESC()
> >
> >           I030_170 = I030_170_DESC()
> >
> >           I030_100 = I030_100_DESC()
> >
> >           I030_180 = I030_180_DESC()
> >
> >           I030_181 = I030_181_DESC()
> >
> >           I030_060 = I030_060_DESC()
> >
> >           I030_150 = I030_150_DESC()
> >
> >           I030_140 = I030_140_DESC()
> >
> >           I030_340 = I030_340_DESC()
> >
> >           I030_400 = I030_400_DESC()
> >
> >       ...
> >
> >           I030_210 = I030_210_DESC()
> >
> >           I030_120 = I030_120_DESC()
> >
> >           I030_050 = I030_050_DESC()
> >
> >           I030_270 = I030_270_DESC()
> >
> >           I030_370 = I030_370_DESC()
> >
> >
> >
> >
> >
> >       Från: Anthony Scopatz [mailto:sc...@gm...]
> >       Skickat: den 12 juli 2012 00:02
> >       Till: Discussion list for PyTables
> >       Ämne: Re: [Pytables-users] advice on using PyTables
> >
> >
> >
> >       Hello Benjamin,
> >
> >
> >
> >       Not knowing to much about the ASTERIX format, other than
> > what you said and what is in the links, I would say that this is a good
> > fit for HDF5 and PyTables.  PyTables will certainly help you read in
> > the data and manipulate it.
> >
> >
> >
> >       However, before you abandon hachoir completely, I will say
> > it is a lot easier to write hdf5 files in PyTables than to use the HDF5
> > C API.   If hachoir is too slow, have you tried profiling the code to
> > see what is taking up the most time?  Maybe you could just rewrite
> > these parts in C?  Have you looked into Cythonizing it?  Also, you
> > don't seem to be using numpy to read in the data... (there are some
> > tricks given ASTERIX here, but not insurmountable).
> >
> >
> >
> >       I ask the above, just so you don't have to completely
> > rewrite everything.  You are correct though that pure python is
> > probably not sufficient.  Feel free to ask more questions here.
> >
> >
> >
> >       Be Well
> >
> >       Anthony
> >
> >
> >
> >       On Wed, Jul 11, 2012 at 6:52 AM, <ben...@lf...>
> > wrote:
> >
> >       Hi,
> >
> >       I'm working with Air Traffic Management and would like to
> > perform checks / compute statistics on ASTERIX data.
> >       ASTERIX is an ATM Surveillance Data Binary Messaging Format
> > (http://www.eurocontrol.int/asterix/public/standard_page/overview.html)
> >
> >       The data consist of a concatenation of consecutive data
> > blocks.
> >       Each data block consists of data category + length +
> > records.
> >       Each record is of variable length and consists of several
> > data items (that are well defined for each category).
> >       Some data items might be present or not depending on a field
> > specification (bitfield).
> >
> >       I started to write a parser using hachoir
> > (https://bitbucket.org/haypo/hachoir/overview) a pure python library.
> >       But the parsing was really too slow and taking a lot of
> > memory.
> >       That's not really useable.
> >
> >       >From what I read, PyTables could really help to manipulate
> > and analyze the data.
> >       So I've been thinking about writing a tool (probably in C)
> > to convert my ASTERIX format to HDF5.
> >
> >       Before I start, I'd like confirmation that this seems like a
> > suitable application for PyTables.
> >       Is there another approach than writing a conversion tool to
> > HDF5?
> >
> >       Thanks in advance
> >
> >       Benjamin
> >
> >
> >
> >
> >       ------------------------------------------------------------
> > ------------------
> >       Live Security Virtual Conference
> >       Exclusive live event will cover all the ways today's
> > security and
> >       threat landscape has changed and how IT managers can
> > respond. Discussions
> >       will include endpoint security, mobile security and the
> > latest in malware
> >       threats.
> > http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> >       _______________________________________________
> >       Pytables-users mailing list
> >       Pyt...@li...
> >       https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
> >
> >
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

Re: [Pytables-users] advice on data representation

[<ben...@lf...>]

> There might be an easier way to do this with numpy dtypes.  In pseudo-
> code:
> 
> np.dtype([(colname, np.int16) for colname in colnames])
> 

Can we use time and enum kinds that way as well?

This makes me think I should probably flatten my table.
Having nested columns is quite natural to group cells under a data item (it's nice to access a field like I030_180/SPEED).
But that's more a naming convention and I can't perform in-kernel searches with nested columns.

> 
> If you want to mark the whole column as valid, you can use a boolean
> attribute on the table itself for each column.  They could be named
> like colname_valid.
> 
> See
> http://pytables.github.com/usersguide/libref.html#tables.Leaf.setAttr
> and http://pytables.github.com/usersguide/libref.html#the-attributeset-
> class for more info.

Good to know.

> 
> This is more for flagging individual cells as valid or not.  For
> integers you need  to pick a values which means invalid (like -999999).
> 
> 

Yes, you are right. It's not a column I want to flag but a group of cells in a row (all cells of a particular data item).
Problem is that if I have a uint8 for a cell, there is no invalid value I can use.
Maybe I could use a "larger type" (uint16 in this case) to be able to pick an invalid value.
Or a bool for this group of cells.

> 
> If you don't want to use a VLArray, then maxlen is probably your best
> option.
> 
> If you want to do something a little more sophisticated, you could
> break you data out into a main table and then a helper VLarray.  Every
> row in the table is matched by the same row in vlarray.  Then when you
> want to get your full data back out, you have to go to the table and
> the vlarray.  This makes things a little more annoying to work with,
> but it does what you want.

Interesting. I'll look more into it.

> 
> Hope this helps.  Feel free to ask more questions!

Yes, it does.
Thanks!


Benjamin


> Be Well
> Anthony
> 
> 
> 
> 
> 
> 
> 	Cheers,
> 
> 
> 
> 	Benjamin
> 
> 
> 
> 
> 
> 	class I030_180_DESC(tables.IsDescription):
> 
> 	    """Calculated Track Velocity (Polar)"""
> 
> 	    SPEED = tables.UInt16Col(pos=0)
> 
> 	    HEADING = tables.UInt16Col(pos=1)
> 
> 
> 
> 	class I030_181_DESC(tables.IsDescription):
> 
> 	    """Calculated Track Velocity (Cartesian)"""
> 
> 	    X = tables.Int16Col(pos=0)
> 
> 	    Y = tables.Int16Col(pos=1)
> 
> 
> 
> 	class I030_340_DESC(tables.IsDescription):
> 
> 	    """Last Measured Mode 3/A"""
> 
> 	    V = tables.EnumCol(tables.Enum({
> 
> 	        "Code validated": 0,
> 
> 	        "Code not validated": 1,
> 
> 	        "uninitialized": 255
> 
> 	        }), "uninitialized",
> 
> 	        base="uint8",
> 
> 	        pos=0)
> 
> 	    G = tables.EnumCol(tables.Enum({
> 
> 	        "Default": 0,
> 
> 	        "Garbled code": 1,
> 
> 	        "uninitialized": 255
> 
> 	        }), "uninitialized",
> 
> 	        base="uint8",
> 
> 	        pos=1)
> 
> 	    L = tables.EnumCol(tables.Enum({
> 
> 	        "MODE 3/A code as derived from the reply of the
> transponder,": 0,
> 
> 	        "Smoothed MODE 3/A code as provided by a local
> tracker": 1
> 
> 	        "uninitialized": 255
> 
> 	        }), "uninitialized",
> 
> 	        base="uint8",
> 
> 	        pos=2)
> 
> 	    sb = tables.UInt8Col(pos=3)
> 
> 	    mode_3_a = tables.UInt16Col(pos=4)
> 
> 
> 
> 	class I030_400_DESC(tables.IsDescription):
> 
> 	    """Callsign"""
> 
> 	    callsign = tables.StringCol(7, pos=0)
> 
> 
> 
> 	class I030_050_DESC(tables.IsDescription):
> 
> 	    """Artas Track Number"""
> 
> 	    AUI = tables.UInt8Col(pos=0)
> 
> 	    unused = tables.UInt8Col(pos=1)
> 
> 	    STN = tables.UInt16Col(pos=2)
> 
> 	    FX = tables.EnumCol(tables.Enum({
> 
> 	        "end of data item": 0,
> 
> 	        "extension into next extent": 1,
> 
> 	        "uninitialized": 255
> 
> 	        }), "uninitialized",
> 
> 	        base="uint8",
> 
> 	        pos=3)
> 
> 
> 
> 	class I030Record(tables.IsDescription):
> 
> 	    """Cat 030 record"""
> 
> 	    ff_timestamp = tables.Time32Col()
> 
> 	    I030_010 = I030_010_DESC()
> 
> 	    I030_015 = I030_015_DESC()
> 
> 	    I030_030 = I030_030_DESC()
> 
> 	    I030_035 = I030_035_DESC()
> 
> 	    I030_040 = I030_040_DESC()
> 
> 	    I030_070 = I030_070_DESC()
> 
> 	    I030_170 = I030_170_DESC()
> 
> 	    I030_100 = I030_100_DESC()
> 
> 	    I030_180 = I030_180_DESC()
> 
> 	    I030_181 = I030_181_DESC()
> 
> 	    I030_060 = I030_060_DESC()
> 
> 	    I030_150 = I030_150_DESC()
> 
> 	    I030_140 = I030_140_DESC()
> 
> 	    I030_340 = I030_340_DESC()
> 
> 	    I030_400 = I030_400_DESC()
> 
> 	...
> 
> 	    I030_210 = I030_210_DESC()
> 
> 	    I030_120 = I030_120_DESC()
> 
> 	    I030_050 = I030_050_DESC()
> 
> 	    I030_270 = I030_270_DESC()
> 
> 	    I030_370 = I030_370_DESC()
> 
> 
> 
> 
> 
> 	Från: Anthony Scopatz [mailto:sc...@gm...]
> 	Skickat: den 12 juli 2012 00:02
> 	Till: Discussion list for PyTables
> 	Ämne: Re: [Pytables-users] advice on using PyTables
> 
> 
> 
> 	Hello Benjamin,
> 
> 
> 
> 	Not knowing to much about the ASTERIX format, other than
> what you said and what is in the links, I would say that this is a good
> fit for HDF5 and PyTables.  PyTables will certainly help you read in
> the data and manipulate it.
> 
> 
> 
> 	However, before you abandon hachoir completely, I will say
> it is a lot easier to write hdf5 files in PyTables than to use the HDF5
> C API.   If hachoir is too slow, have you tried profiling the code to
> see what is taking up the most time?  Maybe you could just rewrite
> these parts in C?  Have you looked into Cythonizing it?  Also, you
> don't seem to be using numpy to read in the data... (there are some
> tricks given ASTERIX here, but not insurmountable).
> 
> 
> 
> 	I ask the above, just so you don't have to completely
> rewrite everything.  You are correct though that pure python is
> probably not sufficient.  Feel free to ask more questions here.
> 
> 
> 
> 	Be Well
> 
> 	Anthony
> 
> 
> 
> 	On Wed, Jul 11, 2012 at 6:52 AM, <ben...@lf...>
> wrote:
> 
> 	Hi,
> 
> 	I'm working with Air Traffic Management and would like to
> perform checks / compute statistics on ASTERIX data.
> 	ASTERIX is an ATM Surveillance Data Binary Messaging Format
> (http://www.eurocontrol.int/asterix/public/standard_page/overview.html)
> 
> 	The data consist of a concatenation of consecutive data
> blocks.
> 	Each data block consists of data category + length +
> records.
> 	Each record is of variable length and consists of several
> data items (that are well defined for each category).
> 	Some data items might be present or not depending on a field
> specification (bitfield).
> 
> 	I started to write a parser using hachoir
> (https://bitbucket.org/haypo/hachoir/overview) a pure python library.
> 	But the parsing was really too slow and taking a lot of
> memory.
> 	That's not really useable.
> 
> 	>From what I read, PyTables could really help to manipulate
> and analyze the data.
> 	So I've been thinking about writing a tool (probably in C)
> to convert my ASTERIX format to HDF5.
> 
> 	Before I start, I'd like confirmation that this seems like a
> suitable application for PyTables.
> 	Is there another approach than writing a conversion tool to
> HDF5?
> 
> 	Thanks in advance
> 
> 	Benjamin
> 
> 
> 
> 
> 	------------------------------------------------------------
> ------------------
> 	Live Security Virtual Conference
> 	Exclusive live event will cover all the ways today's
> security and
> 	threat landscape has changed and how IT managers can
> respond. Discussions
> 	will include endpoint security, mobile security and the
> latest in malware
> 	threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> 	_______________________________________________
> 	Pytables-users mailing list
> 	Pyt...@li...
> 	https://lists.sourceforge.net/lists/listinfo/pytables-users
> 
> 
> 

Re: [Pytables-users] advice on data representation

[Anthony S. <sc...@gm...>]

On Tue, Jul 31, 2012 at 3:14 AM, <ben...@lf...> wrote:

>  Hello,****
>
> ** **
>
> I ’ve started to write a parser to convert ASTERIX data to HDF5, but I
> have some problem to represent all the data.****
>
> ** **
>
> I use table objects. I’ve defined a class for each category record (a
> record is made of different data items).****
>
> See below as an example for category 30.
>

There might be an easier way to do this with numpy dtypes.  In pseudo-code:

np.dtype([(colname, np.int16) for colname in colnames])


> ****
>
> ** 1. Some data items are optional. Is there a good way to mark a column
> as valid?
>

If you want to mark the whole column as valid, you can use a boolean
attribute on the table itself for each column.  They could be named like
colname_valid.

See http://pytables.github.com/usersguide/libref.html#tables.Leaf.setAttr
and http://pytables.github.com/usersguide/libref.html#the-attributeset-class
for
more info.


>  For enum, I can easily add an “uninitialized” default value.****
>
> For (U)Int, most of the time there is no good default value that could
> tell me if the data is valid or not.****
>
> I thought about using np.nan, but that’s only for float.
>

This is more for flagging individual cells as valid or not.  For integers
you need  to pick a values which means invalid (like -999999).


>  ****
>
> Or I could add a bool variable (valid) to each column.****
>
> Is there another way?
>

Yes, attributes, as above.

****
>
> ** 2. Some data items have a variable length (in fact some fields can be
> repeated).
>
> In the class I030_050_DESC for example, if FX is set to 1, then the fields
> are repeated.****
>
> I know that fields cannot be of a variable length in a table object.****
>
> I could try to use a shape (max_length,) for those columns but the max
> length would be a bit arbitrary as there is no theoretical limit (even if
> in practice, it is often quite low).****
>
> Or should I try to represent the data using a VLArray?****
>
> I found it quite natural to represent my data as a table and I don’t
> really see how I could do the same with an array.
>

If you don't want to use a VLArray, then maxlen is probably your best
option.

If you want to do something a little more sophisticated, you could break
you data out into a main table and then a helper VLarray.  Every row in the
table is matched by the same row in vlarray.  Then when you want to get
your full data back out, you have to go to the table and the vlarray.  This
makes things a little more annoying to work with, but it does what you want.

Hope this helps.  Feel free to ask more questions!
Be Well
Anthony


> ****
>
> ** **
>
> Cheers,****
>
> ** **
>
> Benjamin****
>
> ** **
>
> ** **
>
> class I030_180_DESC(tables.IsDescription):****
>
>     """Calculated Track Velocity (Polar)"""****
>
>     SPEED = tables.UInt16Col(pos=0)****
>
>     HEADING = tables.UInt16Col(pos=1)****
>
> ** **
>
> class I030_181_DESC(tables.IsDescription):****
>
>     """Calculated Track Velocity (Cartesian)"""****
>
>     X = tables.Int16Col(pos=0)****
>
>     Y = tables.Int16Col(pos=1)****
>
> ** **
>
> class I030_340_DESC(tables.IsDescription):****
>
>     """Last Measured Mode 3/A"""****
>
>     V = tables.EnumCol(tables.Enum({****
>
>         "Code validated": 0, ****
>
>         "Code not validated": 1,****
>
>         "uninitialized": 255****
>
>         }), "uninitialized",****
>
>         base="uint8",****
>
>         pos=0)****
>
>     G = tables.EnumCol(tables.Enum({****
>
>         "Default": 0, ****
>
>         "Garbled code": 1,****
>
>         "uninitialized": 255****
>
>         }), "uninitialized",****
>
>         base="uint8",****
>
>         pos=1)****
>
>     L = tables.EnumCol(tables.Enum({****
>
>         "MODE 3/A code as derived from the reply of the transponder,": 0,
> ****
>
>         "Smoothed MODE 3/A code as provided by a local tracker": 1****
>
>         "uninitialized": 255****
>
>         }), "uninitialized",****
>
>         base="uint8",****
>
>         pos=2)****
>
>     sb = tables.UInt8Col(pos=3)****
>
>     mode_3_a = tables.UInt16Col(pos=4)****
>
> ** **
>
> class I030_400_DESC(tables.IsDescription):****
>
>     """Callsign"""****
>
>     callsign = tables.StringCol(7, pos=0)****
>
> ** **
>
> class I030_050_DESC(tables.IsDescription):****
>
>     """Artas Track Number"""****
>
>     AUI = tables.UInt8Col(pos=0)****
>
>     unused = tables.UInt8Col(pos=1)****
>
>     STN = tables.UInt16Col(pos=2)****
>
>     FX = tables.EnumCol(tables.Enum({****
>
>         "end of data item": 0, ****
>
>         "extension into next extent": 1,****
>
>         "uninitialized": 255****
>
>         }), "uninitialized",****
>
>         base="uint8",****
>
>         pos=3)****
>
> ** **
>
> class I030Record(tables.IsDescription):****
>
>     """Cat 030 record"""****
>
>     ff_timestamp = tables.Time32Col()****
>
>     I030_010 = I030_010_DESC()****
>
>     I030_015 = I030_015_DESC()****
>
>     I030_030 = I030_030_DESC()****
>
>     I030_035 = I030_035_DESC()****
>
>     I030_040 = I030_040_DESC()****
>
>     I030_070 = I030_070_DESC()****
>
>     I030_170 = I030_170_DESC()****
>
>     I030_100 = I030_100_DESC()****
>
>     I030_180 = I030_180_DESC()****
>
>     I030_181 = I030_181_DESC()****
>
>     I030_060 = I030_060_DESC()****
>
>     I030_150 = I030_150_DESC()****
>
>     I030_140 = I030_140_DESC()****
>
>     I030_340 = I030_340_DESC()****
>
>     I030_400 = I030_400_DESC()****
>
> ...****
>
>     I030_210 = I030_210_DESC()****
>
>     I030_120 = I030_120_DESC()****
>
>     I030_050 = I030_050_DESC()****
>
>     I030_270 = I030_270_DESC()****
>
>     I030_370 = I030_370_DESC()****
>
> ** **
>
> ** **
>
> *Från:* Anthony Scopatz [mailto:sc...@gm...]
> *Skickat:* den 12 juli 2012 00:02
> *Till:* Discussion list for PyTables
> *Ämne:* Re: [Pytables-users] advice on using PyTables****
>
> ** **
>
> Hello Benjamin, ****
>
> ** **
>
> Not knowing to much about the ASTERIX format, other than what you said and
> what is in the links, I would say that this is a good fit for HDF5 and
> PyTables.  PyTables will certainly help you read in the data and manipulate
> it.  ****
>
> ** **
>
> However, before you abandon hachoir completely, I will say it is a lot
> easier to write hdf5 files in PyTables than to use the HDF5 C API.   If
> hachoir is too slow, have you tried profiling the code to see what is
> taking up the most time?  Maybe you could just rewrite these parts in C?
>  Have you looked into Cythonizing it?  Also, you don't seem to be using
> numpy to read in the data... (there are some tricks given ASTERIX here, but
> not insurmountable).****
>
> ** **
>
> I ask the above, just so you don't have to completely rewrite everything.
>  You are correct though that pure python is probably not sufficient.  Feel
> free to ask more questions here.****
>
> ** **
>
> Be Well****
>
> Anthony****
>
> ** **
>
> On Wed, Jul 11, 2012 at 6:52 AM, <ben...@lf...> wrote:****
>
> Hi,
>
> I'm working with Air Traffic Management and would like to perform checks /
> compute statistics on ASTERIX data.
> ASTERIX is an ATM Surveillance Data Binary Messaging Format (
> http://www.eurocontrol.int/asterix/public/standard_page/overview.html)
>
> The data consist of a concatenation of consecutive data blocks.
> Each data block consists of data category + length + records.
> Each record is of variable length and consists of several data items (that
> are well defined for each category).
> Some data items might be present or not depending on a field specification
> (bitfield).
>
> I started to write a parser using hachoir (
> https://bitbucket.org/haypo/hachoir/overview) a pure python library.
> But the parsing was really too slow and taking a lot of memory.
> That's not really useable.
>
> >From what I read, PyTables could really help to manipulate and analyze
> the data.
> So I've been thinking about writing a tool (probably in C) to convert my
> ASTERIX format to HDF5.
>
> Before I start, I'd like confirmation that this seems like a suitable
> application for PyTables.
> Is there another approach than writing a conversion tool to HDF5?
>
> Thanks in advance
>
> Benjamin****
>
> ** **
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>

[Pytables-users] advice on data representation

[<ben...@lf...>]

Hello,

I 've started to write a parser to convert ASTERIX data to HDF5, but I have some problem to represent all the data.

I use table objects. I've defined a class for each category record (a record is made of different data items).
See below as an example for category 30.

1. Some data items are optional. Is there a good way to mark a column as valid?
For enum, I can easily add an "uninitialized" default value.
For (U)Int, most of the time there is no good default value that could tell me if the data is valid or not.
I thought about using np.nan, but that's only for float.
Or I could add a bool variable (valid) to each column.
Is there another way?

2. Some data items have a variable length (in fact some fields can be repeated).
In the class I030_050_DESC for example, if FX is set to 1, then the fields are repeated.
I know that fields cannot be of a variable length in a table object.
I could try to use a shape (max_length,) for those columns but the max length would be a bit arbitrary as there is no theoretical limit (even if in practice, it is often quite low).
Or should I try to represent the data using a VLArray?
I found it quite natural to represent my data as a table and I don't really see how I could do the same with an array.

Cheers,

Benjamin


class I030_180_DESC(tables.IsDescription):
    """Calculated Track Velocity (Polar)"""
    SPEED = tables.UInt16Col(pos=0)
    HEADING = tables.UInt16Col(pos=1)

class I030_181_DESC(tables.IsDescription):
    """Calculated Track Velocity (Cartesian)"""
    X = tables.Int16Col(pos=0)
    Y = tables.Int16Col(pos=1)

class I030_340_DESC(tables.IsDescription):
    """Last Measured Mode 3/A"""
    V = tables.EnumCol(tables.Enum({
        "Code validated": 0,
        "Code not validated": 1,
        "uninitialized": 255
        }), "uninitialized",
        base="uint8",
        pos=0)
    G = tables.EnumCol(tables.Enum({
        "Default": 0,
        "Garbled code": 1,
        "uninitialized": 255
        }), "uninitialized",
        base="uint8",
        pos=1)
    L = tables.EnumCol(tables.Enum({
        "MODE 3/A code as derived from the reply of the transponder,": 0,
        "Smoothed MODE 3/A code as provided by a local tracker": 1
        "uninitialized": 255
        }), "uninitialized",
        base="uint8",
        pos=2)
    sb = tables.UInt8Col(pos=3)
    mode_3_a = tables.UInt16Col(pos=4)

class I030_400_DESC(tables.IsDescription):
    """Callsign"""
    callsign = tables.StringCol(7, pos=0)

class I030_050_DESC(tables.IsDescription):
    """Artas Track Number"""
    AUI = tables.UInt8Col(pos=0)
    unused = tables.UInt8Col(pos=1)
    STN = tables.UInt16Col(pos=2)
    FX = tables.EnumCol(tables.Enum({
        "end of data item": 0,
        "extension into next extent": 1,
        "uninitialized": 255
        }), "uninitialized",
        base="uint8",
        pos=3)

class I030Record(tables.IsDescription):
    """Cat 030 record"""
    ff_timestamp = tables.Time32Col()
    I030_010 = I030_010_DESC()
    I030_015 = I030_015_DESC()
    I030_030 = I030_030_DESC()
    I030_035 = I030_035_DESC()
    I030_040 = I030_040_DESC()
    I030_070 = I030_070_DESC()
    I030_170 = I030_170_DESC()
    I030_100 = I030_100_DESC()
    I030_180 = I030_180_DESC()
    I030_181 = I030_181_DESC()
    I030_060 = I030_060_DESC()
    I030_150 = I030_150_DESC()
    I030_140 = I030_140_DESC()
    I030_340 = I030_340_DESC()
    I030_400 = I030_400_DESC()
...
    I030_210 = I030_210_DESC()
    I030_120 = I030_120_DESC()
    I030_050 = I030_050_DESC()
    I030_270 = I030_270_DESC()
    I030_370 = I030_370_DESC()


Från: Anthony Scopatz [mailto:sc...@gm...]
Skickat: den 12 juli 2012 00:02
Till: Discussion list for PyTables
Ämne: Re: [Pytables-users] advice on using PyTables

Hello Benjamin,

Not knowing to much about the ASTERIX format, other than what you said and what is in the links, I would say that this is a good fit for HDF5 and PyTables.  PyTables will certainly help you read in the data and manipulate it.

However, before you abandon hachoir completely, I will say it is a lot easier to write hdf5 files in PyTables than to use the HDF5 C API.   If hachoir is too slow, have you tried profiling the code to see what is taking up the most time?  Maybe you could just rewrite these parts in C?  Have you looked into Cythonizing it?  Also, you don't seem to be using numpy to read in the data... (there are some tricks given ASTERIX here, but not insurmountable).

I ask the above, just so you don't have to completely rewrite everything.  You are correct though that pure python is probably not sufficient.  Feel free to ask more questions here.

Be Well
Anthony

On Wed, Jul 11, 2012 at 6:52 AM, <ben...@lf...<mailto:ben...@lf...>> wrote:
Hi,

I'm working with Air Traffic Management and would like to perform checks / compute statistics on ASTERIX data.
ASTERIX is an ATM Surveillance Data Binary Messaging Format (http://www.eurocontrol.int/asterix/public/standard_page/overview.html)

The data consist of a concatenation of consecutive data blocks.
Each data block consists of data category + length + records.
Each record is of variable length and consists of several data items (that are well defined for each category).
Some data items might be present or not depending on a field specification (bitfield).

I started to write a parser using hachoir (https://bitbucket.org/haypo/hachoir/overview) a pure python library.
But the parsing was really too slow and taking a lot of memory.
That's not really useable.

>From what I read, PyTables could really help to manipulate and analyze the data.
So I've been thinking about writing a tool (probably in C) to convert my ASTERIX format to HDF5.

Before I start, I'd like confirmation that this seems like a suitable application for PyTables.
Is there another approach than writing a conversion tool to HDF5?

Thanks in advance

Benjamin

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Andy W. <wil...@gm...>]

+1 on the changes


On Wed, Jul 25, 2012 at 6:10 PM, Anthony Scopatz <sc...@gm...> wrote:

> On Wed, Jul 25, 2012 at 5:48 PM, Maarten Sneep <maa...@xs...>wrote:
>>
>>
>> Although for the example you give here I would prefer to change to
>> tb.open(). Namespaces exist for a reason, I think.
>>
>
> This is a good point, and I would be in favor of this rename if other
> people were as well...
>
> If you do a "from tables import *" and the tb.open() overwrites the
> builtin open(), you can always regain the builtin by doing "del open".
>  That said, I agree with you that we shouldn't be 'import *'ing.  It sets a
> bad example ;)
>


As a compromise, maybe open() and open_file() could both be defined and
equivalent but only add open_file() as part of the module's __all__ list?
Namespaces are great but if someone is using import *, they probably don't
know that yet and this would be a rude introduction.

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Anthony S. <sc...@gm...>]

Hello Francesc, David,

This is a good point and worth noting.  But that was also sort of the point
of this poll.  If there are major API changes that people would like to
see, or if there are parts which people find difficult or non-intuitive,
please speak up!  (For example tb.openFile() -> tb.open() was a great
suggestion.)

David, you mentioned that you hit some rough spots yourself.   What where
these?

Be Well
Anthony

On Fri, Jul 27, 2012 at 4:12 AM, Francesc Alted <fa...@py...> wrote:

> I think David raises a good point on this message (probably sent from un
> unsubscribed address):
>
>
> ForwardedMessage.eml
> Subject:
> RE: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!
> From:
> "David Briant" <da...@bi...>
> Date:
> 7/27/12 8:13 AM
>
> To:
> "'Discussion list for PyTables'" <pyt...@li...>
>
>
> It’s been a while since I used PyTablesl pro but when I was learning it
> I found it quite tricky to get to grips with. If you’re gonna be
> changing the public interface – you could do so much more than just be
> PEP 8 compliant which while is a great guideline cannot address the real
> issues of usability. I’d talk to people and find the areas they found
> hard to pickup and address those first. A dress over of things like
> openFile to open_file will not add much value but will create a lot of
> work maintenance wise. Joel on Software is a recommended resource.
>
> Just my two penneth.
>
> Kind regards
>
> David
>
> *From:*Anthony Scopatz [mailto:sc...@gm...]
> *Sent:* 26 July 2012 00:11
> *To:* Discussion list for PyTables
> *Subject:* Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal -
> Please respond!
>
> On Wed, Jul 25, 2012 at 5:48 PM, Maarten Sneep <maa...@xs...
> <mailto:maa...@xs...>> wrote:
>
> On 24 jul. 2012, at 18:39, Anthony Scopatz wrote:
>
>  > The next version of PyTables that will be released will be v3.0 and
> will be Python 3 compliant. I thought that this would be an excellent
> (if not the only) chance to bring the PyTables API into full PEP8
> compliance [1]. This would mean changing function and attribute names like:
>  >
>  > tb.openFile() -> tb.open_file()
>  >
>  > For the next couple of releases BOTH the new and old API would be
> available to facilitate this transition (ie tb.openFile() and
> tb.open_file() would both work). To ease migration, we would also
> provide a 2to3-like utility for you to use on your code base that would
> update the API for you automatically. At some fixed point in the future
> (v3.2?), the old API would go away, but you would have had ample time to
> run this script. The developers either feel positively or neutral to
> these changes.
>  >
>  > The question for you the user is then, would this be something that
> you would like to see? How valuable would you find this change? Is
> bundling this with the Python 3 change too much overhead?
>
> +1
>
> Although for the example you give here I would prefer to change to
> tb.open(). Namespaces exist for a reason, I think.
>
> This is a good point, and I would be in favor of this rename if other
> people were as well...
>
> If you do a "from tables import *" and the tb.open() overwrites the
> builtin open(), you can always regain the builtin by doing "del open".
> That said, I agree with you that we shouldn't be 'import *'ing. It sets
> a bad example ;)
>
>     And yes, that would mean that a significant part of the examples and
>     documentation would need to be written with 'import tables as tb'
>     instead of 'from tables import *'. I find the former much better for
>     learning. Since most of the functionality is accessed through
>     instance methods anyway, the namespace isn't too big anyway.
>
>     This change will require careful thought, but I think it is worth it.
>
>     About the timing. Since many of the packages I need aren't on py3, I
>     haven't given much thought to py3. I have of course followed the
>     recommended practices (use the print function, not the statement,
>     ...) for py 2.7, but other than that I have yet to figure out 2to3.
>     Since this change will require attention to older code anyway, we
>     might as well fold this PEP8 change into the Python 3 one.
>
> To be clear as well, the Python 2 version of PyTables would probably
> also include both new & old APIs to facilitate the transition.
>
> Be Well
>
> Anthony
>
>
>     Best,
>
>     Maarten
>
>
>
>
> ------------------------------------------------------------------------------
>     Live Security Virtual Conference
>     Exclusive live event will cover all the ways today's security and
>     threat landscape has changed and how IT managers can respond.
>     Discussions
>     will include endpoint security, mobile security and the latest in
>     malware
>     threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>     _______________________________________________
>     Pytables-users mailing list
>     Pyt...@li...
>     <mailto:Pyt...@li...>
>     https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
>
> On 7/24/12 6:39 PM, Anthony Scopatz wrote:
> > Dear PyTables Community,
> >
> > The next version of PyTables that will be released will be v3.0 and
> > will be Python 3 compliant. I thought that this would be an excellent
> > (if not the only) chance to bring the PyTables API into full PEP8
> > compliance [1]. This would mean changing function and attribute names
> > like:
> >
> >     tb.openFile() -> tb.open_file()
> >
> >
> > For the next couple of releases BOTH the new and old API would be
> > available to facilitate this transition (ie tb.openFile()and
> > tb.open_file() would both work). To ease migration, we would also
> > provide a 2to3-like utility for you to use on your code base that
> > would update the API for you automatically. At some fixed point in the
> > future (v3.2?), the old API would go away, but you would have had
> > ample time to run this script. The developers either feel positively
> > or neutral to these changes.
> >
> > The question for you the user is then, would this be something that
> > you would like to see? How valuable would you find this change? Is
> > bundling this with the Python 3 change too much overhead?
> >
> > *Please respond with a +1, +0, -0, or -1 and any comments you might
> > have.* I look forward to hearing from you!
> >
> > Be Well
> > Anthony
> >
> > 1. http://www.python.org/dev/peps/pep-0008/
> >
> >
> >
> ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> >
> >
> > _______________________________________________
> > Pytables-users mailing list
> > Pyt...@li...
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
> --
> Francesc Alted
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Francesc A. <fa...@py...>]

I think David raises a good point on this message (probably sent from un 
unsubscribed address):


ForwardedMessage.eml
Subject:
RE: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!
From:
"David Briant" <da...@bi...>
Date:
7/27/12 8:13 AM

To:
"'Discussion list for PyTables'" <pyt...@li...>


It’s been a while since I used PyTablesl pro but when I was learning it 
I found it quite tricky to get to grips with. If you’re gonna be 
changing the public interface – you could do so much more than just be 
PEP 8 compliant which while is a great guideline cannot address the real 
issues of usability. I’d talk to people and find the areas they found 
hard to pickup and address those first. A dress over of things like 
openFile to open_file will not add much value but will create a lot of 
work maintenance wise. Joel on Software is a recommended resource.

Just my two penneth.

Kind regards

David

*From:*Anthony Scopatz [mailto:sc...@gm...]
*Sent:* 26 July 2012 00:11
*To:* Discussion list for PyTables
*Subject:* Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - 
Please respond!

On Wed, Jul 25, 2012 at 5:48 PM, Maarten Sneep <maa...@xs... 
<mailto:maa...@xs...>> wrote:

On 24 jul. 2012, at 18:39, Anthony Scopatz wrote:

 > The next version of PyTables that will be released will be v3.0 and 
will be Python 3 compliant. I thought that this would be an excellent 
(if not the only) chance to bring the PyTables API into full PEP8 
compliance [1]. This would mean changing function and attribute names like:
 >
 > tb.openFile() -> tb.open_file()
 >
 > For the next couple of releases BOTH the new and old API would be 
available to facilitate this transition (ie tb.openFile() and 
tb.open_file() would both work). To ease migration, we would also 
provide a 2to3-like utility for you to use on your code base that would 
update the API for you automatically. At some fixed point in the future 
(v3.2?), the old API would go away, but you would have had ample time to 
run this script. The developers either feel positively or neutral to 
these changes.
 >
 > The question for you the user is then, would this be something that 
you would like to see? How valuable would you find this change? Is 
bundling this with the Python 3 change too much overhead?

+1

Although for the example you give here I would prefer to change to 
tb.open(). Namespaces exist for a reason, I think.

This is a good point, and I would be in favor of this rename if other 
people were as well...

If you do a "from tables import *" and the tb.open() overwrites the 
builtin open(), you can always regain the builtin by doing "del open". 
That said, I agree with you that we shouldn't be 'import *'ing. It sets 
a bad example ;)

    And yes, that would mean that a significant part of the examples and
    documentation would need to be written with 'import tables as tb'
    instead of 'from tables import *'. I find the former much better for
    learning. Since most of the functionality is accessed through
    instance methods anyway, the namespace isn't too big anyway.

    This change will require careful thought, but I think it is worth it.

    About the timing. Since many of the packages I need aren't on py3, I
    haven't given much thought to py3. I have of course followed the
    recommended practices (use the print function, not the statement,
    ...) for py 2.7, but other than that I have yet to figure out 2to3.
    Since this change will require attention to older code anyway, we
    might as well fold this PEP8 change into the Python 3 one.

To be clear as well, the Python 2 version of PyTables would probably 
also include both new & old APIs to facilitate the transition.

Be Well

Anthony


    Best,

    Maarten



    ------------------------------------------------------------------------------
    Live Security Virtual Conference
    Exclusive live event will cover all the ways today's security and
    threat landscape has changed and how IT managers can respond.
    Discussions
    will include endpoint security, mobile security and the latest in
    malware
    threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
    _______________________________________________
    Pytables-users mailing list
    Pyt...@li...
    <mailto:Pyt...@li...>
    https://lists.sourceforge.net/lists/listinfo/pytables-users



On 7/24/12 6:39 PM, Anthony Scopatz wrote:
> Dear PyTables Community,
>
> The next version of PyTables that will be released will be v3.0 and 
> will be Python 3 compliant. I thought that this would be an excellent 
> (if not the only) chance to bring the PyTables API into full PEP8 
> compliance [1]. This would mean changing function and attribute names 
> like:
>
>     tb.openFile() -> tb.open_file()
>
>
> For the next couple of releases BOTH the new and old API would be 
> available to facilitate this transition (ie tb.openFile()and 
> tb.open_file() would both work). To ease migration, we would also 
> provide a 2to3-like utility for you to use on your code base that 
> would update the API for you automatically. At some fixed point in the 
> future (v3.2?), the old API would go away, but you would have had 
> ample time to run this script. The developers either feel positively 
> or neutral to these changes.
>
> The question for you the user is then, would this be something that 
> you would like to see? How valuable would you find this change? Is 
> bundling this with the Python 3 change too much overhead?
>
> *Please respond with a +1, +0, -0, or -1 and any comments you might 
> have.* I look forward to hearing from you!
>
> Be Well
> Anthony
>
> 1. http://www.python.org/dev/peps/pep-0008/
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users


-- 
Francesc Alted

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Anthony S. <sc...@gm...>]

On Wed, Jul 25, 2012 at 5:48 PM, Maarten Sneep <maa...@xs...>wrote:

> On 24 jul. 2012, at 18:39, Anthony Scopatz wrote:
>
> > The next version of PyTables that will be released will be v3.0 and will
> be Python 3 compliant.  I thought that this would be an excellent (if not
> the only) chance to bring the PyTables API into full PEP8 compliance [1].
>  This would mean changing function and attribute names like:
> >
> > tb.openFile() -> tb.open_file()
> >
> > For the next couple of releases BOTH the new and old API would be
> available to facilitate this transition (ie tb.openFile() and
> tb.open_file() would both work).    To ease migration, we would also
> provide a 2to3-like utility for you to use on your code base that would
> update the API for you automatically.  At some fixed point in the future
> (v3.2?), the old API would go away, but you would have had ample time to
> run this script.  The developers either feel positively or neutral to these
> changes.
> >
> > The question for you the user is then, would this be something that you
> would like to see?  How valuable would you find this change?  Is bundling
> this with the Python 3 change too much overhead?
>
> +1
>
> Although for the example you give here I would prefer to change to
> tb.open(). Namespaces exist for a reason, I think.
>

This is a good point, and I would be in favor of this rename if other
people were as well...

If you do a "from tables import *" and the tb.open() overwrites the builtin
open(), you can always regain the builtin by doing "del open".  That said,
I agree with you that we shouldn't be 'import *'ing.  It sets a bad example
;)


> And yes, that would mean that a significant part of the examples and
> documentation would need to be written with 'import tables as tb' instead
> of 'from tables import *'. I find the former much better for learning.
> Since most of the functionality is accessed through instance methods
> anyway, the namespace isn't too big anyway.
>
> This change will require careful thought, but I think it is worth it.
>
> About the timing. Since many of the packages I need aren't on py3, I
> haven't given much thought to py3. I have of course followed the
> recommended practices (use the print function, not the statement, ...) for
> py 2.7, but other than that I have yet to figure out 2to3. Since this
> change will require attention to older code anyway, we might as well fold
> this PEP8 change into the Python 3 one.
>

To be clear as well, the Python 2 version of PyTables would probably also
include both new & old APIs to facilitate the transition.

Be Well
Anthony


>
> Best,
>
> Maarten
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Maarten S. <maa...@xs...>]

On 24 jul. 2012, at 18:39, Anthony Scopatz wrote:

> The next version of PyTables that will be released will be v3.0 and will be Python 3 compliant.  I thought that this would be an excellent (if not the only) chance to bring the PyTables API into full PEP8 compliance [1].  This would mean changing function and attribute names like:
> 
> tb.openFile() -> tb.open_file()
> 
> For the next couple of releases BOTH the new and old API would be available to facilitate this transition (ie tb.openFile() and tb.open_file() would both work).    To ease migration, we would also provide a 2to3-like utility for you to use on your code base that would update the API for you automatically.  At some fixed point in the future (v3.2?), the old API would go away, but you would have had ample time to run this script.  The developers either feel positively or neutral to these changes.
> 
> The question for you the user is then, would this be something that you would like to see?  How valuable would you find this change?  Is bundling this with the Python 3 change too much overhead?

+1

Although for the example you give here I would prefer to change to tb.open(). Namespaces exist for a reason, I think. 

And yes, that would mean that a significant part of the examples and documentation would need to be written with 'import tables as tb' instead of 'from tables import *'. I find the former much better for learning. Since most of the functionality is accessed through instance methods anyway, the namespace isn't too big anyway.

This change will require careful thought, but I think it is worth it. 

About the timing. Since many of the packages I need aren't on py3, I haven't given much thought to py3. I have of course followed the recommended practices (use the print function, not the statement, ...) for py 2.7, but other than that I have yet to figure out 2to3. Since this change will require attention to older code anyway, we might as well fold this PEP8 change into the Python 3 one. 

Best,

Maarten

Re: [Pytables-users] Copy real files/ directories as fileNodes to hdf5 file with pytables...

[Anthony S. <sc...@gm...>]

On Wed, Jul 25, 2012 at 3:35 PM, Christian Werner <
fre...@go...> wrote:

> Great!
>
> Seems like this is just what I need... I'm currently trying to come up
> with the reverse script...
> Seems that the File.walkNodes() function does not work for fileNodes,
> right?
>

File.walkNodes() is what you should be using.  It should work for filenodes
as well... just leave classname=None.


> Also, I might need to create some metadata for attributes and such...
>
> Thanks for the pointer,
>

No problem!

Be Well
Anthony


> C
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

Re: [Pytables-users] Copy real files/ directories as fileNodes to hdf5 file with pytables...

[Christian W. <fre...@go...>]

Great!

Seems like this is just what I need... I'm currently trying to come up with the reverse script... 
Seems that the File.walkNodes() function does not work for fileNodes, right?

Also, I might need to create some metadata for attributes and such...

Thanks for the pointer,
C

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Dr. L. J. W. <lou...@no...>]

+1 from me.

though I wont be using Py3.0 for a while yet.

Lou

On 7/25/12 1:48 PM, Richard Llewellyn wrote:
> +1
> I'm already making the 2 to 3 transition, so have left pytables for
> awhile ;[.  It's great news that the next version will be py3k
> compatible!  Style change will be a good thing as far as I'm
> concerned.
>
> On Wed, Jul 25, 2012 at 12:39 PM, Andre' Walker-Loud
> <wal...@gm...> wrote:
>> +1
>>
>> while it will be painful to switch now, it will be more painful to switch when I have had time to write more code in the old style.
>>
>> Andre
>>
>>
>>
>>
>> On Jul 25, 2012, at 12:37 AM, <ben...@lf...> <ben...@lf...> wrote:
>>
>>> +1
>>>
>>> /Benjamin
>>>
>>>> -----Ursprungligt meddelande-----
>>>> Från: Anthony Scopatz [mailto:sc...@gm...]
>>>> Skickat: den 24 juli 2012 18:39
>>>> Till: Discussion list for PyTables
>>>> Ämne: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please
>>>> respond!
>>>>
>>>> Dear PyTables Community,
>>>>
>>>> The next version of PyTables that will be released will be v3.0 and
>>>> will be Python 3 compliant.  I thought that this would be an excellent
>>>> (if not the only) chance to bring the PyTables API into full PEP8
>>>> compliance [1].  This would mean changing function and attribute names
>>>> like:
>>>>
>>>>
>>>>       tb.openFile() -> tb.open_file()
>>>>
>>>>
>>>> For the next couple of releases BOTH the new and old API would be
>>>> available to facilitate this transition (ie tb.openFile() and
>>>> tb.open_file() would both work).    To ease migration, we would also
>>>> provide a 2to3-like utility for you to use on your code base that would
>>>> update the API for you automatically.  At some fixed point in the
>>>> future (v3.2?), the old API would go away, but you would have had ample
>>>> time to run this script.  The developers either feel positively or
>>>> neutral to these changes.
>>>>
>>>> The question for you the user is then, would this be something that you
>>>> would like to see?  How valuable would you find this change?  Is
>>>> bundling this with the Python 3 change too much overhead?
>>>>
>>>> Please respond with a +1, +0, -0, or -1 and any comments you might
>>>> have.  I look forward to hearing from you!
>>>>
>>>> Be Well
>>>> Anthony
>>>>
>>>> 1. http://www.python.org/dev/peps/pep-0008/
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

-- 
----------------------------------------------------------------------------
| Dr. Louis J. Wicker
| NSSL/FRDD  Rm 4368
| National Weather Center
| 120 David L. Boren Boulevard, Norman, OK 73072
|
| E-mail:   Lou...@no...
| HTTP:    www.nssl.noaa.gov/~lwicker
| Phone:    (405) 325-6340
| Fax:        (405) 325-6780
|
| Yesterday I was so clever, so I wanted to change the world.
| Today I am wiser, so I am changing myself.
|
|    Rumi
|
----------------------------------------------------------------------------

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Richard L. <ll...@gm...>]

+1
I'm already making the 2 to 3 transition, so have left pytables for
awhile ;[.  It's great news that the next version will be py3k
compatible!  Style change will be a good thing as far as I'm
concerned.

On Wed, Jul 25, 2012 at 12:39 PM, Andre' Walker-Loud
<wal...@gm...> wrote:
> +1
>
> while it will be painful to switch now, it will be more painful to switch when I have had time to write more code in the old style.
>
> Andre
>
>
>
>
> On Jul 25, 2012, at 12:37 AM, <ben...@lf...> <ben...@lf...> wrote:
>
>> +1
>>
>> /Benjamin
>>
>>> -----Ursprungligt meddelande-----
>>> Från: Anthony Scopatz [mailto:sc...@gm...]
>>> Skickat: den 24 juli 2012 18:39
>>> Till: Discussion list for PyTables
>>> Ämne: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please
>>> respond!
>>>
>>> Dear PyTables Community,
>>>
>>> The next version of PyTables that will be released will be v3.0 and
>>> will be Python 3 compliant.  I thought that this would be an excellent
>>> (if not the only) chance to bring the PyTables API into full PEP8
>>> compliance [1].  This would mean changing function and attribute names
>>> like:
>>>
>>>
>>>      tb.openFile() -> tb.open_file()
>>>
>>>
>>> For the next couple of releases BOTH the new and old API would be
>>> available to facilitate this transition (ie tb.openFile() and
>>> tb.open_file() would both work).    To ease migration, we would also
>>> provide a 2to3-like utility for you to use on your code base that would
>>> update the API for you automatically.  At some fixed point in the
>>> future (v3.2?), the old API would go away, but you would have had ample
>>> time to run this script.  The developers either feel positively or
>>> neutral to these changes.
>>>
>>> The question for you the user is then, would this be something that you
>>> would like to see?  How valuable would you find this change?  Is
>>> bundling this with the Python 3 change too much overhead?
>>>
>>> Please respond with a +1, +0, -0, or -1 and any comments you might
>>> have.  I look forward to hearing from you!
>>>
>>> Be Well
>>> Anthony
>>>
>>> 1. http://www.python.org/dev/peps/pep-0008/
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Andre' Walker-L. <wal...@gm...>]

+1

while it will be painful to switch now, it will be more painful to switch when I have had time to write more code in the old style.

Andre




On Jul 25, 2012, at 12:37 AM, <ben...@lf...> <ben...@lf...> wrote:

> +1
> 
> /Benjamin
> 
>> -----Ursprungligt meddelande-----
>> Från: Anthony Scopatz [mailto:sc...@gm...]
>> Skickat: den 24 juli 2012 18:39
>> Till: Discussion list for PyTables
>> Ämne: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please
>> respond!
>> 
>> Dear PyTables Community,
>> 
>> The next version of PyTables that will be released will be v3.0 and
>> will be Python 3 compliant.  I thought that this would be an excellent
>> (if not the only) chance to bring the PyTables API into full PEP8
>> compliance [1].  This would mean changing function and attribute names
>> like:
>> 
>> 
>> 	tb.openFile() -> tb.open_file()
>> 
>> 
>> For the next couple of releases BOTH the new and old API would be
>> available to facilitate this transition (ie tb.openFile() and
>> tb.open_file() would both work).    To ease migration, we would also
>> provide a 2to3-like utility for you to use on your code base that would
>> update the API for you automatically.  At some fixed point in the
>> future (v3.2?), the old API would go away, but you would have had ample
>> time to run this script.  The developers either feel positively or
>> neutral to these changes.
>> 
>> The question for you the user is then, would this be something that you
>> would like to see?  How valuable would you find this change?  Is
>> bundling this with the Python 3 change too much overhead?
>> 
>> Please respond with a +1, +0, -0, or -1 and any comments you might
>> have.  I look forward to hearing from you!
>> 
>> Be Well
>> Anthony
>> 
>> 1. http://www.python.org/dev/peps/pep-0008/
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

[Pytables-users] Copy real files/ directories as fileNodes to hdf5 file with potables...

[Christian W. <fre...@go...>]

Hi group

I'm currently trying to store a directory with all files contained as
filesNodes in a h5 file. The reason is that I currently store a single
configuration file and model output from this model as tables in one h5
file to a) document the simulation settings and b) store the model results.

This works and everything is fine, but I was wondering if there is some
elegant way to just copy a set of files/ subfolders into the h5 and also
store the filenames, attributes et al. ...
It would be even better if I could store my entire model setup (incl. the
model executable), too, as this would really mean I or someone else would
always be sure that I/ he could reproduce the shared data with the original
model setup)...

Consider this file structure (should be copied as file nodes), in addition
there will be tables storing the model results...:

-+- model (binary)
 |
 +- setupdir, contains fileA, fileB, ...
 |
 +- parameterdir, contains parA, parB, ...

Is it possible to just copy them without knowing a) filenames, b) number of
files, ...? Without specifying each file, creating a node etc?
Is there some API for this?

Cheers,
C

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[<ben...@lf...>]

+1

/Benjamin

> -----Ursprungligt meddelande-----
> Från: Anthony Scopatz [mailto:sc...@gm...]
> Skickat: den 24 juli 2012 18:39
> Till: Discussion list for PyTables
> Ämne: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please
> respond!
> 
> Dear PyTables Community,
> 
> The next version of PyTables that will be released will be v3.0 and
> will be Python 3 compliant.  I thought that this would be an excellent
> (if not the only) chance to bring the PyTables API into full PEP8
> compliance [1].  This would mean changing function and attribute names
> like:
> 
> 
> 	tb.openFile() -> tb.open_file()
> 
> 
> For the next couple of releases BOTH the new and old API would be
> available to facilitate this transition (ie tb.openFile() and
> tb.open_file() would both work).    To ease migration, we would also
> provide a 2to3-like utility for you to use on your code base that would
> update the API for you automatically.  At some fixed point in the
> future (v3.2?), the old API would go away, but you would have had ample
> time to run this script.  The developers either feel positively or
> neutral to these changes.
> 
> The question for you the user is then, would this be something that you
> would like to see?  How valuable would you find this change?  Is
> bundling this with the Python 3 change too much overhead?
> 
> Please respond with a +1, +0, -0, or -1 and any comments you might
> have.  I look forward to hearing from you!
> 
> Be Well
> Anthony
> 
> 1. http://www.python.org/dev/peps/pep-0008/

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Jacob B. <jac...@gm...>]

+1 of course! ;)

-Jacob

On Tue, Jul 24, 2012 at 12:30 PM, Fernando Paolo <fs...@gm...> wrote:

> +1
>
> I fully agree on following the Python (standard) naming convention
> (PEP8 compliance). This will make the PyTables API even more
> "pythonic" and, therefore, more intuitive (if that is possible).
>
> Thanks!
>
> -fernando
>
>
>
> On Tue, Jul 24, 2012 at 9:39 AM, Anthony Scopatz <sc...@gm...>
> wrote:
> > Dear PyTables Community,
> >
> > The next version of PyTables that will be released will be v3.0 and will
> be
> > Python 3 compliant.  I thought that this would be an excellent (if not
> the
> > only) chance to bring the PyTables API into full PEP8 compliance [1].
>  This
> > would mean changing function and attribute names like:
> >
> > tb.openFile() -> tb.open_file()
> >
> >
> > For the next couple of releases BOTH the new and old API would be
> available
> > to facilitate this transition (ie tb.openFile() and tb.open_file() would
> > both work).    To ease migration, we would also provide a 2to3-like
> utility
> > for you to use on your code base that would update the API for you
> > automatically.  At some fixed point in the future (v3.2?), the old API
> would
> > go away, but you would have had ample time to run this script.  The
> > developers either feel positively or neutral to these changes.
> >
> > The question for you the user is then, would this be something that you
> > would like to see?  How valuable would you find this change?  Is bundling
> > this with the Python 3 change too much overhead?
> >
> > Please respond with a +1, +0, -0, or -1 and any comments you might have.
>  I
> > look forward to hearing from you!
> >
> > Be Well
> > Anthony
> >
> > 1. http://www.python.org/dev/peps/pep-0008/
> >
> >
> ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Pytables-users mailing list
> > Pyt...@li...
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
>
>
>
> --
> Fernando Paolo
> Institute of Geophysics & Planetary Physics
> Scripps Institution of Oceanography
> University of California, San Diego
>
> http://fspaolo.net
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>



-- 
Jacob Bennett
Massachusetts Institute of Technology
Department of Electrical Engineering and Computer Science
Class of 2014| ben...@mi...

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Fernando P. <fs...@gm...>]

+1

I fully agree on following the Python (standard) naming convention
(PEP8 compliance). This will make the PyTables API even more
"pythonic" and, therefore, more intuitive (if that is possible).

Thanks!

-fernando



On Tue, Jul 24, 2012 at 9:39 AM, Anthony Scopatz <sc...@gm...> wrote:
> Dear PyTables Community,
>
> The next version of PyTables that will be released will be v3.0 and will be
> Python 3 compliant.  I thought that this would be an excellent (if not the
> only) chance to bring the PyTables API into full PEP8 compliance [1].  This
> would mean changing function and attribute names like:
>
> tb.openFile() -> tb.open_file()
>
>
> For the next couple of releases BOTH the new and old API would be available
> to facilitate this transition (ie tb.openFile() and tb.open_file() would
> both work).    To ease migration, we would also provide a 2to3-like utility
> for you to use on your code base that would update the API for you
> automatically.  At some fixed point in the future (v3.2?), the old API would
> go away, but you would have had ample time to run this script.  The
> developers either feel positively or neutral to these changes.
>
> The question for you the user is then, would this be something that you
> would like to see?  How valuable would you find this change?  Is bundling
> this with the Python 3 change too much overhead?
>
> Please respond with a +1, +0, -0, or -1 and any comments you might have.  I
> look forward to hearing from you!
>
> Be Well
> Anthony
>
> 1. http://www.python.org/dev/peps/pep-0008/
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>



-- 
Fernando Paolo
Institute of Geophysics & Planetary Physics
Scripps Institution of Oceanography
University of California, San Diego

http://fspaolo.net

Re: [Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Jon W. <js...@fn...>]

+1 !

PEP8 (or any widely used (and diligently applied) naming convention) 
drastically improves discoverability and usability of an interface.

Regards,
Jon Wilson

On 07/24/2012 11:39 AM, Anthony Scopatz wrote:
> Dear PyTables Community,
>
> The next version of PyTables that will be released will be v3.0 and 
> will be Python 3 compliant.  I thought that this would be an excellent 
> (if not the only) chance to bring the PyTables API into full PEP8 
> compliance [1].  This would mean changing function and attribute names 
> like:
>
>     tb.openFile() -> tb.open_file()
>
>
> For the next couple of releases BOTH the new and old API would be 
> available to facilitate this transition (ie tb.openFile()and 
> tb.open_file() would both work).    To ease migration, we would also 
> provide a 2to3-like utility for you to use on your code base that 
> would update the API for you automatically.  At some fixed point in 
> the future (v3.2?), the old API would go away, but you would have had 
> ample time to run this script.  The developers either feel positively 
> or neutral to these changes.
>
> The question for you the user is then, would this be something that 
> you would like to see?  How valuable would you find this change?  Is 
> bundling this with the Python 3 change too much overhead?
>
> *Please respond with a +1, +0, -0, or -1 and any comments you might 
> have.*  I look forward to hearing from you!
>
> Be Well
> Anthony
>
> 1. http://www.python.org/dev/peps/pep-0008/
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users

[Pytables-users] [POLL] Fully Adopt PEP8 Proposal - Please respond!

[Anthony S. <sc...@gm...>]

Dear PyTables Community,

The next version of PyTables that will be released will be v3.0 and will be
Python 3 compliant.  I thought that this would be an excellent (if not the
only) chance to bring the PyTables API into full PEP8 compliance [1].  This
would mean changing function and attribute names like:

tb.openFile() -> tb.open_file()


For the next couple of releases BOTH the new and old API would be available
to facilitate this transition (ie tb.openFile() and tb.open_file() would
both work).    To ease migration, we would also provide a 2to3-like utility
for you to use on your code base that would update the API for you
automatically.  At some fixed point in the future (v3.2?), the old API
would go away, but you would have had ample time to run this script.  The
developers either feel positively or neutral to these changes.

The question for you the user is then, would this be something that you
would like to see?  How valuable would you find this change?  Is bundling
this with the Python 3 change too much overhead?

*Please respond with a +1, +0, -0, or -1 and any comments you might
have.* I look forward to hearing from you!

Be Well
Anthony

1. http://www.python.org/dev/peps/pep-0008/

Re: [Pytables-users] ANN: PyTables 2.4.0 released

[Jacob B. <jac...@gm...>]

This is great! Much gratitude must go to you all for your hard work.

Thanks,
Jacob Bennett

On Fri, Jul 20, 2012 at 3:38 PM, Anthony Scopatz <sc...@gm...> wrote:

> Excellent!  Congrats Everybody!  Thanks a ton for all of your hard work.
>  I am going to send this out to numpy and scipy.
>
>
> On Fri, Jul 20, 2012 at 3:36 PM, Antonio Valentino <
> ant...@ti...> wrote:
>
>> ==========================
>> Announcing PyTables 2.4.0
>> ==========================
>>
>> We are happy to announce PyTables 2.4.0.
>>
>> This is an incremental release which includes many changes to prepare
>> for future Python 3 support.
>>
>>
>> What's new
>> ==========
>>
>> This release includes support for the float16 data type and read-only
>> support for variable length string attributes.
>>
>> The handling of HDF5 errors has been improved.  The user will no longer
>> see HDF5 error stacks dumped to the console.  All HDF5 error messages
>> are trapped and attached to a proper Python exception.
>>
>> Now PyTables only supports HDF5 v1.8.4+. All the code has been updated
>> to the new HDF5 API.  Supporting only HDF5 1.8 series is beneficial for
>> future development.
>>
>> Documentation has been improved.
>>
>> As always, a large amount of bugs have been addressed and squashed as
>> well.
>>
>> In case you want to know more in detail what has changed in this
>> version, please refer to:
>> http://pytables.github.com/release_notes.html
>>
>> You can download a source package with generated PDF and HTML docs, as
>> well as binaries for Windows, from:
>> http://sourceforge.net/projects/pytables/files/pytables/2.4.0
>>
>> For an online version of the manual, visit:
>> http://pytables.github.com/usersguide/index.html
>>
>>
>> What it is?
>> ===========
>>
>> PyTables is a library for managing hierarchical datasets and designed to
>> efficiently cope with extremely large amounts of data with support for
>> full 64-bit file addressing.  PyTables runs on top of the HDF5 library
>> and NumPy package for achieving maximum throughput and convenient use.
>> PyTables includes OPSI, a new indexing technology, allowing to perform
>> data lookups in tables exceeding 10 gigarows (10**10 rows) in less than
>> a tenth of a second.
>>
>>
>> Resources
>> =========
>>
>> About PyTables: http://www.pytables.org
>>
>> About the HDF5 library: http://hdfgroup.org/HDF5/
>>
>> About NumPy: http://numpy.scipy.org/
>>
>>
>> Acknowledgments
>> ===============
>>
>> Thanks to many users who provided feature improvements, patches, bug
>> reports, support and suggestions.  See the ``THANKS`` file in the
>> distribution package for a (incomplete) list of contributors.  Most
>> specially, a lot of kudos go to the HDF5 and NumPy (and numarray!)
>> makers.  Without them, PyTables simply would not exist.
>>
>>
>> Share your experience
>> =====================
>>
>> Let us know of any bugs, suggestions, gripes, kudos, etc. you may have.
>>
>>
>> ----
>>
>>   **Enjoy data!**
>>
>>   -- The PyTables Team
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>


-- 
Jacob Bennett
Massachusetts Institute of Technology
Department of Electrical Engineering and Computer Science
Class of 2014| ben...@mi...

Re: [Pytables-users] ANN: PyTables 2.4.0 released

[Anthony S. <sc...@gm...>]

Excellent!  Congrats Everybody!  Thanks a ton for all of your hard work.  I
am going to send this out to numpy and scipy.

On Fri, Jul 20, 2012 at 3:36 PM, Antonio Valentino <
ant...@ti...> wrote:

> ==========================
> Announcing PyTables 2.4.0
> ==========================
>
> We are happy to announce PyTables 2.4.0.
>
> This is an incremental release which includes many changes to prepare
> for future Python 3 support.
>
>
> What's new
> ==========
>
> This release includes support for the float16 data type and read-only
> support for variable length string attributes.
>
> The handling of HDF5 errors has been improved.  The user will no longer
> see HDF5 error stacks dumped to the console.  All HDF5 error messages
> are trapped and attached to a proper Python exception.
>
> Now PyTables only supports HDF5 v1.8.4+. All the code has been updated
> to the new HDF5 API.  Supporting only HDF5 1.8 series is beneficial for
> future development.
>
> Documentation has been improved.
>
> As always, a large amount of bugs have been addressed and squashed as well.
>
> In case you want to know more in detail what has changed in this
> version, please refer to:
> http://pytables.github.com/release_notes.html
>
> You can download a source package with generated PDF and HTML docs, as
> well as binaries for Windows, from:
> http://sourceforge.net/projects/pytables/files/pytables/2.4.0
>
> For an online version of the manual, visit:
> http://pytables.github.com/usersguide/index.html
>
>
> What it is?
> ===========
>
> PyTables is a library for managing hierarchical datasets and designed to
> efficiently cope with extremely large amounts of data with support for
> full 64-bit file addressing.  PyTables runs on top of the HDF5 library
> and NumPy package for achieving maximum throughput and convenient use.
> PyTables includes OPSI, a new indexing technology, allowing to perform
> data lookups in tables exceeding 10 gigarows (10**10 rows) in less than
> a tenth of a second.
>
>
> Resources
> =========
>
> About PyTables: http://www.pytables.org
>
> About the HDF5 library: http://hdfgroup.org/HDF5/
>
> About NumPy: http://numpy.scipy.org/
>
>
> Acknowledgments
> ===============
>
> Thanks to many users who provided feature improvements, patches, bug
> reports, support and suggestions.  See the ``THANKS`` file in the
> distribution package for a (incomplete) list of contributors.  Most
> specially, a lot of kudos go to the HDF5 and NumPy (and numarray!)
> makers.  Without them, PyTables simply would not exist.
>
>
> Share your experience
> =====================
>
> Let us know of any bugs, suggestions, gripes, kudos, etc. you may have.
>
>
> ----
>
>   **Enjoy data!**
>
>   -- The PyTables Team
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>

[Pytables-users] ANN: PyTables 2.4.0 released

[Antonio V. <ant...@ti...>]

==========================
Announcing PyTables 2.4.0
==========================

We are happy to announce PyTables 2.4.0.

This is an incremental release which includes many changes to prepare
for future Python 3 support.


What's new
==========

This release includes support for the float16 data type and read-only
support for variable length string attributes.

The handling of HDF5 errors has been improved.  The user will no longer
see HDF5 error stacks dumped to the console.  All HDF5 error messages
are trapped and attached to a proper Python exception.

Now PyTables only supports HDF5 v1.8.4+. All the code has been updated
to the new HDF5 API.  Supporting only HDF5 1.8 series is beneficial for
future development.

Documentation has been improved.

As always, a large amount of bugs have been addressed and squashed as well.

In case you want to know more in detail what has changed in this
version, please refer to:
http://pytables.github.com/release_notes.html

You can download a source package with generated PDF and HTML docs, as
well as binaries for Windows, from:
http://sourceforge.net/projects/pytables/files/pytables/2.4.0

For an online version of the manual, visit:
http://pytables.github.com/usersguide/index.html


What it is?
===========

PyTables is a library for managing hierarchical datasets and designed to
efficiently cope with extremely large amounts of data with support for
full 64-bit file addressing.  PyTables runs on top of the HDF5 library
and NumPy package for achieving maximum throughput and convenient use.
PyTables includes OPSI, a new indexing technology, allowing to perform
data lookups in tables exceeding 10 gigarows (10**10 rows) in less than
a tenth of a second.


Resources
=========

About PyTables: http://www.pytables.org

About the HDF5 library: http://hdfgroup.org/HDF5/

About NumPy: http://numpy.scipy.org/


Acknowledgments
===============

Thanks to many users who provided feature improvements, patches, bug
reports, support and suggestions.  See the ``THANKS`` file in the
distribution package for a (incomplete) list of contributors.  Most
specially, a lot of kudos go to the HDF5 and NumPy (and numarray!)
makers.  Without them, PyTables simply would not exist.


Share your experience
=====================

Let us know of any bugs, suggestions, gripes, kudos, etc. you may have.


----

  **Enjoy data!**

  -- The PyTables Team

Re: [Pytables-users] How to get the name of a current table instance?

[Jacob B. <jac...@gm...>]

Cool, thanks Anthony!

-Jake

On Fri, Jul 20, 2012 at 11:24 AM, Anthony Scopatz <sc...@gm...> wrote:

> Hey Jacob,
>
> There is always Node._v_name and Node._v_pathname for all nodes in the
> tree [1].
>
> Be Well
> Anthony
>
>
> http://pytables.github.com/usersguide/libref.html?highlight=path#tables.Node._v_name
>
> On Fri, Jul 20, 2012 at 11:08 AM, Jacob Bennett <jac...@gm...
> > wrote:
>
>> Hello PyTables Gurus,
>>
>> I am trying to look up the name of a particular table when I am iterating
>> through all of the tables in my file; however, there doesn't seem to be a
>> name attribute accessible or a public method that will return the name for
>> me (this is all according to the current documentation) There must be this
>> attribute, maybe its table.name?
>>
>> Thanks,
>> Jacob
>>
>> --
>> Jacob Bennett
>> Massachusetts Institute of Technology
>> Department of Electrical Engineering and Computer Science
>> Class of 2014| ben...@mi...
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>


-- 
Jacob Bennett
Massachusetts Institute of Technology
Department of Electrical Engineering and Computer Science
Class of 2014| ben...@mi...