Re: [Pytables-users] advice on using PyTables

[Anthony S. <sc...@gm...>]

On Thu, Jul 12, 2012 at 10:17 AM, <ben...@lf...> wrote:

> > Hello Benjamin,
> >
> > Not knowing to much about the ASTERIX format, other than what you said
> > and what is in the links, I would say that this is a good fit for HDF5
> > and PyTables.  PyTables will certainly help you read in the data and
> > manipulate it.
> >
> > However, before you abandon hachoir completely, I will say it is a lot
> > easier to write hdf5 files in PyTables than to use the HDF5 C API.   If
> > hachoir is too slow, have you tried profiling the code to see what is
> > taking up the most time?  Maybe you could just rewrite these parts in
> > C?  Have you looked into Cythonizing it?  Also, you don't seem to be
> > using numpy to read in the data... (there are some tricks given ASTERIX
> > here, but not insurmountable).
> >
> > I ask the above, just so you don't have to completely rewrite
> > everything.  You are correct though that pure python is probably not
> > sufficient.  Feel free to ask more questions here.
>
> Hello Anthony,
>
> Thanks for your answer.
> Now that I discovered ipython and the line_profiler extension, I've done
> more profiling.
> Most of the time is spent in creating the field objects defined in hachoir.
> A field object has nice attributes (size, value, description, address...)
> but it adds a lot of overhead.
>

Understood!  Good luck.


> Contrary to my intuition, not that much time is spent in reading the data
> (despite some bitfields to read).
> So it's probably worth trying to parse and write an hdf5 file directly
> with PyTables.
> I need to read more PyTables doc and examples.
> I might ask more questions when I have :-)
>

Feel free ;)


>
> Regards,
>
> Benjamin
>
>
> >
> > Be Well
> > Anthony
> >
> > On Wed, Jul 11, 2012 at 6:52 AM, <ben...@lf...> wrote:
> >
> >
> >       Hi,
> >
> >       I'm working with Air Traffic Management and would like to
> > perform checks / compute statistics on ASTERIX data.
> >       ASTERIX is an ATM Surveillance Data Binary Messaging Format
> > (http://www.eurocontrol.int/asterix/public/standard_page/overview.html)
> >
> >       The data consist of a concatenation of consecutive data
> > blocks.
> >       Each data block consists of data category + length +
> > records.
> >       Each record is of variable length and consists of several
> > data items (that are well defined for each category).
> >       Some data items might be present or not depending on a field
> > specification (bitfield).
> >
> >       I started to write a parser using hachoir
> > (https://bitbucket.org/haypo/hachoir/overview) a pure python library.
> >       But the parsing was really too slow and taking a lot of
> > memory.
> >       That's not really useable.
> >
> >       >From what I read, PyTables could really help to manipulate
> > and analyze the data.
> >       So I've been thinking about writing a tool (probably in C)
> > to convert my ASTERIX format to HDF5.
> >
> >       Before I start, I'd like confirmation that this seems like a
> > suitable application for PyTables.
> >       Is there another approach than writing a conversion tool to
> > HDF5?
> >
> >       Thanks in advance
> >
> >       Benjamin
> >
> >       ------------------------------------------------------------
> > ------------------
> >       Live Security Virtual Conference
> >       Exclusive live event will cover all the ways today's
> > security and
> >       threat landscape has changed and how IT managers can
> > respond. Discussions
> >       will include endpoint security, mobile security and the
> > latest in malware
> >       threats.
> > http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> >       _______________________________________________
> >       Pytables-users mailing list
> >       Pyt...@li...
> >       https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
> >
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>