Re: [Pytables-users] performance question ( with source )

[Stefan K. <pon...@ya...>]

Sorry for the 2 posts, but I thought an example would help..
    # assume that
    # src_table is source table, filled with data
    # new_table1 and 2 are empty tables, the destination

    # get the rows
    src_rows = src_table.read( start=0, stop=len(src_table))

    # do 'something' to the rows, in reality sort them, but here
    # just put them in a list
    tmp = []
    for row in src_rows:
        tmp.append( row )

    # if we dump out the original rows, it's blazing fast, < second
    new_table1.append( src_rows )
    new_table1.flush()
    
    # but this operation, with the same number of rows, but in 
    # a different order and in a list, takes almost 20 times as long!
    # is there a better way?
    new_table2.append( tmp )
    new_table2.flush()

thanks again,
Stefan

--- Stefan Kuzminski <pon...@ya...> wrote:

> I need an 'external sort' utility for large pytables, since I haven't
> been able to find one I wrote one, but performance is a problem. 
> Specifically if I build up a list of NestedRecords and then append
> them
> to a new table, the write time seems very slow.  If I even use a
> NestedRecArray as a buffer there seems to be the same slowdown just
> setting the new rows in NestedRecArray buffer.  Any pointers would be
> appreciated..
> 
> thanks,
> Stefan
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
> 
> 
> -------------------------------------------------------
> SF.Net email is sponsored by:
> Tame your development challenges with Apache's Geronimo App Server. 
> Download it for free - -and be entered to win a 42" plasma tv or your
> very
> own Sony(tm)PSP.  Click here to play:
> http://sourceforge.net/geronimo.php
> _______________________________________________
> Pytables-users mailing list
> Pyt...@li...
> https://lists.sourceforge.net/lists/listinfo/pytables-users
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com