Re: [Pytables-users] Some method like a "table.readWhereSorted"

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

On Mon, Apr 15, 2013 at 9:40 AM, Julio Trevisan <jul...@gm...>wrote:

> Hi Anthony
>
> Thanks for adding this issue.
>
> Is there a way to use CS indexes to get the row coordinates satisfying a
> simple condition sorted by the column in the condition? I would like to
> avoid using numpy.sort() since the sorting order is probably already
> available within the index information.
>
> My condition is simply (timestamp >= %d) & (timestamp <= %d)" % (ts1, ts2)
>
> If you could please give me some guidelines so that I could put together
> such a method, that would be great.
>
> Julio
>
>
> (like using getWhereList()), but using an index to get the coordinate list
> ordered by column indexes to get the coordinates ? I nhas a *sort*parameter that uses numpy.sort() to do the work, and that readSorted() uses
> a full index to get the sorted sequence. I couldn't make complete sense yet
> of "chunkmaps" being passed to numexpr evaluator inside _where().
>

Hi Julio,

Thanks for taking this on!  You probably what to read [1] to figure out how
numexpr works and what chunkmaps is doing, if you haven't already.

However, probably the simplest implementation of this method would be
basically part of the read_where() method followed by the read_sorted()
body.   It would look like this, but you'll have to try it:

def read_where_sorted(self, ...):
    self._g_check_open()
    condcoords = set([p.nrow for p in
              self._where(condition, condvars, start, stop, step)])
    self._where_condition = None  # reset the conditions
    index = self._check_sortby_csi(sortby, checkCSI)
    sortcoords = index[start:stop:step]
    coords = [c for c in sortcoords if c in condcoords]
    return self.read_coordinates(coords, field)

There may be faster, more elegant solutions, but I think that something
like this would work.

Be Well
Anthony

1. http://code.google.com/p/numexpr/

>
>
> On Thu, Apr 11, 2013 at 1:14 PM, Anthony Scopatz <sc...@gm...>wrote:
>
>> Thanks for bringing this up, Julio.
>>
>> Hmm I don't think that this exists currently, but since there are
>> readWhere() and readSorted() it shouldn't be too hard to implement.  I have
>> opened issue #225 to this effect.  Pull requests welcome!
>>
>> https://github.com/PyTables/PyTables/issues/225
>>
>> Be Well
>> Anthony
>>
>>
>> On Wed, Apr 10, 2013 at 1:02 PM, Dr. Louis Wicker <lou...@no...>wrote:
>>
>>> I am also interested in the this capability, if it exists in some way...
>>>
>>> Lou
>>>
>>> On Apr 10, 2013, at 12:35 PM, Julio Trevisan <jul...@gm...>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > Is there a way that I could have the ability of readWhere (i.e.,
>>> specify condition, and fast result) but also using a CSIndex so that the
>>> rows come sorted in a particular order?
>>> >
>>> > I checked readSorted() but it is iterative and does not allow to
>>> specify a condition.
>>> >
>>> > Julio
>>> >
>>> ------------------------------------------------------------------------------
>>> > Precog is a next-generation analytics platform capable of advanced
>>> > analytics on semi-structured data. The platform includes APIs for
>>> building
>>> > apps and a phenomenal toolset for data science. Developers can use
>>> > our toolset for easy data analysis & visualization. Get a free account!
>>> >
>>> http://www2.precog.com/precogplatform/slashdotnewsletter_______________________________________________
>>> > Pytables-users mailing list
>>> > Pyt...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>> ----------------------------------------------------------------------------
>>> | Dr. Louis J. Wicker
>>> | NSSL/WRDD  Rm 4366
>>> | National Weather Center
>>> | 120 David L. Boren Boulevard, Norman, OK 73072
>>> |
>>> | E-mail:   Lou...@no...
>>> | HTTP:    http://www.nssl.noaa.gov/~lwicker
>>> | Phone:    (405) 325-6340
>>> | Fax:        (405) 325-6780
>>> |
>>> |
>>> I  For every complex problem, there is a solution that is simple,
>>> |      neat, and wrong.
>>> |
>>> |   -- H. L. Mencken
>>> |
>>>
>>> ----------------------------------------------------------------------------
>>> | "The contents  of this message are mine personally and
>>> | do not reflect any position of  the Government or NOAA."
>>>
>>> ----------------------------------------------------------------------------
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Precog is a next-generation analytics platform capable of advanced
>>> analytics on semi-structured data. The platform includes APIs for
>>> building
>>> apps and a phenomenal toolset for data science. Developers can use
>>> our toolset for easy data analysis & visualization. Get a free account!
>>> http://www2.precog.com/precogplatform/slashdotnewsletter
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pyt...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Precog is a next-generation analytics platform capable of advanced
>> analytics on semi-structured data. The platform includes APIs for building
>> apps and a phenomenal toolset for data science. Developers can use
>> our toolset for easy data analysis & visualization. Get a free account!
>> http://www2.precog.com/precogplatform/slashdotnewsletter
>> _______________________________________________
>> Pytables-users mailing list
>> Pyt...@li...
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>