[Pytables-users] Performance issue?

[Louis W. <Lou...@no...>]

Hi:

I have a 300,000+ line Pytable having 25 columns of various data,  
mostly floats.

I need to do a set of simultaneous searches on this data, based on  
observation type, time, location...

Here is the code I am using to do the search (here I am just  
searching for 5 columns of data, I would like to do all 25 and have  
in other ways, not much diff):

       for o in table.iterrows():

             ob_date = datetime.datetime(o['year'], o['month'], o 
['day'], o['hour'], o['min'], o['sec'])

             if o['status'] == status_in:

                   if o['type'] == obtype_in                         
and  \
                      tgrid[0]  <= ob_date  < tgrid[1]          and  \
                      xgrid[1]  <=  o['x']  < xgrid[2]              
and  \
                      ygrid[1]  <=  o['y']  < ygrid[2]              
and  \
                      zgrid[1]  <=  o['z']  < zgrid[2]:

                         value.append(o['value'])
                         Hxa.append(o['Hxa'])
                         Hxf.append(o['Hxf'])
                         sdHxa.append(o['sdHxa'])
                         sdHxf.append(o['sdHxf'])

the only "funny" deal here is that I am using the python datetime  
module to compare the times, but from my tests and trials with and  
without it I don't think that is the problem (?).

It takes about 7 sec on the Mac G5 (2.5 ghz) and 8.5 sec on an SGI  
Altix.  I think that is not very good, right?

I have also tried "inlining" each variable with a search (really  
slow), or try using the table.where functions, again, slow.

Another problem I am having with this is that I need to search over  
20-30 individual time bins (like above), and I cannot figure out a  
way to bin up the data
in one pass, e.g., I am calling the above loop 20-30 times, so to  
data mine this table I have a longer wait than I thought I would....

Ideas or comments?

Lou Wicker


------------------------------------------------------------------------ 
----
|  Dr. Louis J. Wicker
|  Research Scientist, National Severe Storms Lab
| 1313 Halley Cir, Norman, OK 73069
| E-mail:   Lou...@no...
| HTTP:  www.nssl.noaa.gov/~lwicker
| Phone:    (405) 366-0416
| Fax:       (405) 366-0472
|
|
| Two wrongs don't make a right, but three left's do...
------------------------------------------------------------------------ 
----
|
| "The contents  of this message are mine personally and
| do not reflect any position of  the Government or NOAA."
|
------------------------------------------------------------------------ 
----