Re: [Pytables-users] pyTable index from c++

[Anthony S. <sc...@gm...>]

On Thu, Nov 8, 2012 at 10:19 PM, Jim Knoll <jim...@sp...>wrote:

>   I love the index function and promote the internal use of PyTables at
> my company.  The availability of a indexed method to speed the search is
> the main reason why.****
>
> ** **
>
> We are a mixed shop using c++ to create H5 (just for the raw speed …  need
> to keep up with streaming data)  End users start with python pyTables to
> consume the data.  (Often after we have created indexes from python
> pytables.col.col1.createIndex())   ****
>
> ** **
>
> Sometimes the users come up with something we want to do thousands of
> times and performance is critical.  But then we are falling back to c++ We
> can use our own index method but would like to make dbl use of the PyTables
> index.  ****
>
> ** **
>
> I know the python table.where(   is implemented in C.
>

Hi Jim,

This is only kind of true.  Querying (ie all of the where*() methods) are
actually mostly written in Python in the tables.py and expressions.py
files.  However, they make use of numexpr [1].


>  ****
>
> ** Is there a way to access that from c or c++?    Don’t mind if I need
> to do work to get the result I think in my case the work may be worth it.
>

*PLAN 1:* One possibility is that the parts of PyTables are written in
Cython.  We could maybe try (without making any edits to these files) to
convert them to Cython.  This has the advantage that for Cython files, if
you write the appropriate C++ header file and link against the shared
library correctly, it is possible to access certain functions from C/C++.
BUT, I am not sure how much of speed boost you would get out of this since
you would still be calling out to the Python interpreter to get these
result.  You are just calling Python's virtual machine from C++ rather than
calling it from Python (like normal).   This has the advantage that you
would basically get access to these functions acting on tables from C++.

*PLAN 2:* Alternatively, numexpr itself is mostly written in C++ already.
 You should be able to call core numexpr functions directly.  However, you
would have to feed it data that you read from the tables yourself.  These
could even be table indexes.  On a personal note, if you get code working
that does this, I would be interested in seeing your implementation.  (I
have another project where I have tables that I want to query from C++)

Let us know what route you ultimately end up taking or if you have any
further questions!

Be Well
Anthony

1. http://code.google.com/p/numexpr/source/browse/#hg%2Fnumexpr


>
>
> ------------------------------
>
> *    Jim Knoll*     *
>      **Data Developer*
>
>      Spot Trading L.L.C
>      440 South LaSalle St., Suite 2800
>      Chicago, IL 60605
>      Office: 312.362.4550
>      Direct: 312-362-4798
>      Fax: 312.362.4551
>      jim...@sp...
>      www.spottradingllc.com
> ------------------------------
>
> The information contained in this message may be privileged and
> confidential and protected from disclosure. If the reader of this message
> is not the intended recipient, or an employee or agent responsible for
> delivering this message to the intended recipient, you are hereby notified
> that any dissemination, distribution or copying of this communication is
> strictly prohibited. If you have received this communication in error,
> please notify us immediately by replying to the message and deleting it
> from your computer. Thank you. Spot Trading, LLC
>
>
>
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_d2d_nov
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>