Re: [Pytables-users] Copying block of rows between tables

[Francesc A. <fa...@py...>]

A Monday 27 September 2010 14:24:21 Koert Kuipers escrigué:
> Hello all,
> I need to copy a large block of rows (start=1,000,000 and
> stop=5,000,000) from table a to table b in the same file. Looking at
> the pytables documentation I came up with a few options.
> 
> Option 1:
> for row in a.iterrows(start, stop):
>     b.append([row[:]])
> 
> However a loop in python over millions of rows doesn't seem optimal.
> 
> Option 2:
> b.append(a.read(start, stop))
> 
> All dthe ata is first copied from a to memory before being written to
> b, which is not scalable.
> 
> Option 3:
> Use whereAppend()
> 
> This seems the best (assuming the copying of rows from table to table
> is done much more efficiently in C than I could do in Python).
> However whereAppend doesn't want to run without a condition, and I
> do not have a condition, just start and stop parameters....

Well, if you want to do some timings on whereAppend() you can always use 
some trivial condition like 'col1 == col1' or similar.  But I agree that 
`condition` should support a ``None`` value meaning ``without 
condition``.  Could you please add a ticket on this?

> Did I miss the obvious alternative? Any suggestions?

Well, I'd say that you forgot the most obvious one, i.e. `Table.copy()` 
:-)

-- 
Francesc Alted