Re: [Pytables-users] Combining in-kernel queries with out-of-core computations

[Anthony S. <sc...@gm...>]

On Wed, Jun 6, 2012 at 11:18 AM, Jon Wilson <js...@fn...> wrote:

>  Hi Anthony,
>
>  Well I think the issue at hand is that you are trying to support two
> disparate cases with one expression: sparse and dense selection.  We have
> tools for dealing with these cases individually and performing out-of-core
> calculations.  And if you know a priori which case you are going to fall
> into, you can do the right thing.  So without doing anything special, I
> think medium-fast is probably the best and easiest thing that you can
> expect right now.  (Though I would be delighted to be proved wrong on this
> point.)
>
> True enough.  Sometimes I can't know anything a priori about the density
> of the selection, of course.  And I'm happy to worry about the internals,
> but some of my colleagues, not so much ;)
>

I fully understand your position!


>  but I think the ideal would be to have a .where type query operator that
>> returns Column objects or a Table object, with a "view" imposed in either
>> case.
>>
>
>  We are very open to pull requests if you come up with an implementation
> of this that you like more ;).
>
> Very fair.  We'll see if I can get to it.  Is there any sort of
> guide-to-the-source to help me get started when and if that happens?  I
> guess just the reference guide?
>

Not really as such.  There is this list, and then there is
pyt...@go... for development specific questions and issues.


>
> I'll have a lot to learn before I can contribute usefully, I'm sure.  I
> don't know enough about the implementation details yet to know: would a
> selection make the out-of-core performance gains from chunking and other
> things moot because you'd have to skip around too much?
>

Basically, I would say that if you were to write a function that does what
you described, we could take care of / help with the PyTables integration
issues.  You could start by looking at the code for Expr [1] and seeing if
you could track down the one field issue.  Or maybe there is a way to add
this functionality easily...  Don't hesitate to ask if you run into
problems.

Be Well
Anthony


> Regards,
> Jon
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>