Re: [Pytables-users] Combining in-kernel queries with out-of-core computations

[Anthony S. <sc...@gm...>]

On Tue, Jun 5, 2012 at 10:32 PM, Jon Wilson <js...@fn...> wrote:

[snip]


>  I think something like
> histogram(tables.Expr('col0 + col1**2', mytable.where('col2 > 15 &
> abs(col3) < 5')).eval())
> would be ideal, but since where() returns a row iterator, and not
> something that I can extract Column objects from, I don't see any way to
> make it work.
>

You are probably looking for the readWhere()
method<http://pytables.github.com/usersguide/libref.html#tables.Table.readWhere>
which
normally returns a numpy structured array.  The line you are looking for is
thus:

histogram(tables.Expr('col0 + col1**2', mytable.readWhere('col2 > 15 &
abs(col3) < 5')).eval())

This will likely be fast in both cases.  I hope this helps.

Be Well
Anthony


>
> So, am I missing some way to compute the histogram variable in the numexpr
> kernel, but only for rows I'm interested in?
> Regards,
> Jon
>
>
> On 06/05/2012 09:45 PM, Anthony Scopatz wrote:
>
> Hello Jon,
>
>  I believe that the where() method just uses the Expr / numexpr
> functionality under the covers.  Anything that you can do in Expr you
> should be able to do in where().  Can you provide a short example where
> this is not the case?
>
>  Be Well
> Anthony
>
> On Tue, Jun 5, 2012 at 6:17 PM, Jon Wilson <js...@fn...> wrote:
>
>> Hi all,
>> In looking through the docs, I see two very nice features: the .where()
>> query method, and the tables.Expr computation mechanism.  But, it
>> doesn't appear to be possible to combine the two.  It appears that, if I
>> want to compute some function of my columns, but only for certain rows,
>> I have two options.
>>  - I can use tables.Expr to compute the function, and then filter the
>> results in python
>>  - I can use mytable.where() to select the rows I'm interested in, and
>> then compute the function in python
>>
>> Am I missing anything?  Is it possible to perform fast out-of-core
>> computations with numexpr, but only on a subset of the existing rows?
>> Regards,
>> Jon Wilson
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
>
> _______________________________________________
> Pytables-users mailing lis...@li...://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>