Re: [ctypes-users] copying/slicing ctypes arrays, (c_ulong *n)()

[Ray S <ra...@bl...>]

Hi Thomas,

I made a better direct comparison of ctypes and memmove, map etc., using 
all the methods suggested so far. I think it's fairly valid.(?) Run on a 
2.4GHz WinXP, Py2.3, a number of times.

Parsing numarray,info() was a royal pain, it writes directly to stdout!
I.e.:
 >>> inf = src.info()
class: <class 'numarray.numarraycore.NumArray'>
shape: (32,)
strides: (8,)
byteoffset: 0
bytestride: 8
itemsize: 8
aligned: 1
contiguous: 1
data: <memory at 00815950 with size:256 held by object 00815930 aliasing object
00000000>
byteorder: little
byteswap: 0
type: Float64
 >>> inf
 >>> type(inf)
<type 'NoneType'>


I was surprised that assignment wasn't faster and that numarray assignment 
was consistently ~.5% faster than Numeric.
The N vs. n memmove() flip-flopped for fastest, with array.array always slower.

Since the ctype does support slicing, I was considering leaving the data in 
the device driver buffer (~1MB circluar) and poking into it, but memmove is 
so much faster than slicing the ctype doing memmove()s to numarray.
I presume that I should check for numarray.iscontiguous( ) or is_c_array( 
)  first to be safe...

Results and code below.

Thanks to all for the help,
Ray



 >python test.py
Array address 11443208
n address 17039424
N address 21102656
for loop  2027.0 us ea
map       1781.0 us ea
slice     1704.0 us ea
assign  N 1244.0 us ea
assign  n 1242.0 us ea
memmove   4.3831 us ea
memmove N 3.4773 us ea
memmove n 3.4803 us ea

_____________________________________________________

## test.py
import array
import numarray
import ctypes
import time
import string
import StringIO
import sys

buf = (ctypes.c_long * 2000)()
Array = array.array("l", [0]*10000)
n = numarray.zeros((1000000), numarray.Int32)
N = numarray.zeros((1000000), numarray.Int32)

#!!!!!!!!!!!!!!! arrrrgggg!  !!!!!!!!!!!!!!!!
# n.info() writes directly to stdout!
stdout = sys.stdout
fileo = StringIO.StringIO()
sys.stdout = fileo
n.info()
ninfo = fileo.getvalue( )
fileo.close()
sys.stdout = stdout

stdout = sys.stdout
fileo = StringIO.StringIO()
sys.stdout = fileo
N.info()
Ninfo = fileo.getvalue( )
fileo.close()
sys.stdout = stdout
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

print 'Array address', Array.buffer_info()[0]

ninfo = string.split(ninfo)
nAddress = int(ninfo[20], 16)
print 'n address', nAddress
Ninfo = string.split(Ninfo)
NAddress = int(Ninfo[20], 16)
print 'N address', NAddress

t0 = time.clock()
for loop in range(1000):
     for i in range(2000):
         n[loop+i] = buf[i]
print 'for loop ', round((time.clock()-t0)*1000), 'us ea'

t0 = time.clock()
for loop in range(1000):
     n[loop:loop+2000] = map(None, buf)
print 'map      ', round((time.clock()-t0)*1000), 'us ea'

t0 = time.clock()
for loop in range(1000):
     n[loop:loop+2000] = buf[0:2000]
print 'slice    ', round((time.clock()-t0)*1000), 'us ea'

t0 = time.clock()
for loop in range(10000):
     N[loop:loop+2000] = buf
print 'assign  N', round((time.clock()-t0)*100), 'us ea'

t0 = time.clock()
for loop in range(10000):
     n[loop:loop+2000] = buf
print 'assign  n', round((time.clock()-t0)*100), 'us ea'

t0 = time.clock()
for loop in range(10000):
     ctypes.memmove(10+Array.buffer_info()[0],
                      buf,
                      2000)
print 'memmove  ', round((time.clock()-t0)*1, 4), 'us ea'

t0 = time.clock()
for loop in range(10000):
     ctypes.memmove(10+NAddress,
                      buf,
                      2000)
print 'memmove N', round((time.clock()-t0)*1, 4), 'us ea'

t0 = time.clock()
for loop in range(10000):
     ctypes.memmove(10+nAddress,
                      buf,
                      2000)
print 'memmove n', round((time.clock()-t0)*1, 4), 'us ea'










At 05:10 PM 12/16/2004 +0100, Thomas Heller wrote:
>RayS <ra...@bl...> writes:
>
> > Thanks Florian,
> >
> > My preliminary assessment from trials yesterday (based only on a rough
> > benchmark of idle time) is that
> >         for i in range(2000): numArray[pntr+i] = buf[i]
> > is >10x slower than
> >         numArray[pntr:pntr+2000] = map(None, buf)
> > which is 25% slower than using array.array() & memmove
> >         memmove(array.buffer_info()[0]+pntr, buf)
> >
>
>Do I understand it correctly that memmove is only slightly faster than
>the 'numArray[..] = map(None, buf)' call?  If so, interesting.
>
> > I'll test the direct assignment today, as below, as subsequent
> > operations on numarray should be faster than array. I don't recall why
> > I didn't.
> >
> > The numarray buffer interface is interesting, but I mainly need to do
> > a (deep) copy out of the ctypes memory.  Since the main issue is that
> > src (the ctypes obj) is being updated with 2KB A/D data @125x per
> > second (250K samp./s), I need to get it out of the cytpes obj into
> > storage as efficiently as possible. That storage must then work with
> > slicing and FFT.
> >
> > If the ctypes construct buf = (c_ulong * 2000)() supported slicing, I
> > would just leave it there...
>
>But it *should* support slicing.  Can't you write this instead of the 
>map() call?
>
>     numArray[pntr:pntr+2000] = buf[0:2000]
>
>Another possibility would be to let the ADC write the data directly into
>the numArray, once you know it's address (and assuming it doesn't
>change).
>
>Thomas