[Pytables-users] Restrict searching to particular rows; in-kernel?

[Gerrit H. <ger...@lt...>]

Hi,

I have two tables in my dataset, a large one and a small one. The
smaller table contains (among other data) pointers to row-numbers in
the larger one. I would like to restrict the search to those rows.

This approach works:

    indices = t.root.collocs.mean_collocations.read(start=1,stop=1494,field="FIRST")
    S = set(indices) # faster searching
    res = (row for row in
t.root.collocs.collocations.where(condition="(B_LAT < 0)",
                                      start=1, stop=65508)
           if row.nrow in S)

However, I wonder if I can do this in-kernel somehow.
mytable[indices].where(...) doesn't work, because mytable[indices] is
a numpy.ndarray already, not a table that supports in-kernel
searching. I tried something like mytable.where("(B_LAT < 0) & (NO in
S)") (where I need to add NO explicitly to the table?), but this
results in an exception "TypeError: variable ``S`` has data type
``object``, not allowed in conditions". If I use "...NO in indices..."
instead (should be much slower) I get "TypeError: argument of type
'VariableNode' is not iterable"

Can this be done in-kernel?

I could rearrange my data in a quite different way, and rather than
have those 'pointers', use an embedded table. However, quite often I
am only interested in the larger table, and searching in this one
would become much slower. Yet another approach that would allow
in-kernel searches would be to repeat the entire row from the larger
table in the smaller table, but that would be inefficient
storage-wise.

Any hints?

Gerrit.

-- 
Gerrit Holl
PhD student at Department of Space Science, Luleå University of
Technology, Kiruna, Sweden
http://www.sat.ltu.se/members/gerrit/