Thread: [ctypes-users] copying/slicing ctypes arrays, (c_ulong *n)()
Brought to you by:
theller
Menu
▾
▴
ctypes-users
|
From: RJ <ra...@sa...> - 2004-12-15 04:48:08
|
I have some code that gets a data array from a Win32 A/D dll/driver call at 250,000 samples/sec in 2K chunks. It has the form: buf = (c_ulong * 2000)() which I then copy into a circular numarray buffer like: for i in range(2000): narray[pntr+i] = buf[i] which is a bit slow, but fine for much lower data rates. So, what is the fastest way copy the data? I was re-reading the Python speed FAQ - I'll try narray[pntr:pntr+2000] = map(None, buf) tomorrow, but is that right? Also, since narray is circular I sometimes need to split the ctypes buf in the copy when the pointer starts over, and ctypes does not seem to support slicing. So, I copy to a temp numarray and slice that, which seems sub-optimal. The only "problem" here is that at 250Ksmps/sec it does not leave a lot of processor time to do other things. Would it be best to use a large ctypes array instead of numarray as the circular buffer? I would still need to slice it though, with stepping, to do FFTs etc. Thanks, Ray Secret anti-spam filter-passing text. Include with reply: qwertyuiop |
|
From: Thomas H. <th...@py...> - 2004-12-15 17:30:20
|
RJ <ra...@sa...> writes:
> I have some code that gets a data array from a Win32 A/D dll/driver
> call at 250,000 samples/sec in 2K chunks.
> It has the form:
> buf = (c_ulong * 2000)()
> which I then copy into a circular numarray buffer like:
> for i in range(2000): narray[pntr+i] = buf[i]
> which is a bit slow, but fine for much lower data rates.
>
> So, what is the fastest way copy the data? I was re-reading the
> Python speed FAQ - I'll try
> narray[pntr:pntr+2000] = map(None, buf)
> tomorrow, but is that right?
Should work, but builds a large temporary list. Best would be to avoid
temporary objects at all, like this (for memmove, you need the latest
ctypes release) (I assume that it is possible to get the address of a
numarray buffer in a similar way as is possible for a normal array
instance):
from ctypes import *
import array
src = array.array("i", range(32))
dst = (c_int * 32)()
memmove(dst, *src.buffer_info())
Both ctypes instances and array instances support the buffer interface,
but I don't know of a way to use that from Python code (except for the
readinto() method of file objects).
> Also, since narray is circular I sometimes need to split the ctypes
> buf in the copy when the pointer starts over, and ctypes does not seem
> to support slicing. So, I copy to a temp numarray and slice that,
> which seems sub-optimal.
Currently, ctypes only supports slicing for reading from array and
pointer objects, the next release will probably support writing slices
too (although that's much more dangerous).
> The only "problem" here is that at 250Ksmps/sec it does not leave a
> lot of processor time to do other things.
>
> Would it be best to use a large ctypes array instead of numarray as
> the circular buffer? I would still need to slice it though, with
> stepping, to do FFTs etc.
> Thanks,
> Ray
>
>
> Secret anti-spam filter-passing text. Include with reply:
> qwertyuiop
Thomas
|
|
From: Ray S. <rj...@bl...> - 2004-12-15 18:51:02
|
At 06:31 PM 12/15/2004 +0100, Thomas Heller wrote:
>Should work, but builds a large temporary list. Best would be to avoid
>temporary objects at all, like this (for memmove, you need the latest
>ctypes release) (I assume that it is possible to get the address of a
>numarray buffer in a similar way as is possible for a normal array
>instance):
I'm not sure it will be able to work...
The closest thing to .buffer_info() for numarray-addressing seems to be
info(), but it returns text:
>>> src = numarray.array(range(32), Float)
>>> s = info(src)
class: <class 'numarray.numarraycore.NumArray'>
shape: (32,)
strides: (8,)
byteoffset: 0
bytestride: 8
itemsize: 8
aligned: 1
contiguous: 1
data: <memory at 00815880 with size:256 held by object 00815860 aliasing object
00000000>
byteorder: little
byteswap: 0
type: Float64
>>> src._byteoffset
0
The def from numarraycore.py:
def info(self):
"""info() prints out the key attributes of a numarray."""
_gen.NDArray.info(self)
print "byteorder:", self._byteorder
print "byteswap:", self.isbyteswapped()
print "type:", repr(self._type)
>>> dir(src) does not reveal anything else useable...
>from ctypes import *
>import array
>
>src = array.array("i", range(32))
>dst = (c_int * 32)()
>
>memmove(dst, *src.buffer_info())
>
>Both ctypes instances and array instances support the buffer interface,
>but I don't know of a way to use that from Python code (except for the
>readinto() method of file objects).
>
> > Also, since narray is circular I sometimes need to split the ctypes
> > buf in the copy when the pointer starts over, and ctypes does not seem
> > to support slicing. So, I copy to a temp numarray and slice that,
> > which seems sub-optimal.
>
>Currently, ctypes only supports slicing for reading from array and
>pointer objects, the next release will probably support writing slices
>too (although that's much more dangerous).
I am going to try re-implementing with array instead of numarray and see
how it goes. FFT can digest array.array()s as input anyway.
Since I still need a circular buffer to keep memory/storage use from
running away I'll make the array ~10% longer than needed and just track the
"end" as another variable, endPntr, setting pntr=0 anytime the actual data
saved extends past 100000. Sort of an artificial variable length array.
I'll let you know how the speed goes.
Thanks again Thomas,
Ray
|
|
From: Thomas H. <th...@py...> - 2004-12-15 19:24:33
|
Ray Schumacher <rj...@bl...> writes: > At 06:31 PM 12/15/2004 +0100, Thomas Heller wrote: >>Should work, but builds a large temporary list. Best would be to avoid >>temporary objects at all, like this (for memmove, you need the latest >>ctypes release) (I assume that it is possible to get the address of a >>numarray buffer in a similar way as is possible for a normal array >>instance): > > I'm not sure it will be able to work... > The closest thing to .buffer_info() for numarray-addressing seems to > be info(), but it returns text: > > >>> src = numarray.array(range(32), Float) > >>> s = info(src) > class: <class 'numarray.numarraycore.NumArray'> > shape: (32,) > strides: (8,) > byteoffset: 0 > bytestride: 8 > itemsize: 8 > aligned: 1 > contiguous: 1 > data: <memory at 00815880 with size:256 held by object 00815860 aliasing object > 00000000> > byteorder: little > byteswap: 0 > type: Float64 > >>> src._byteoffset > 0 What a pity. (You could try to parse the <memory at 00815880> to get the address). Maybe you should ask to make sure that there's no way to copy between objects implementing the buffer protocol with some Python function that I do not know about? If such a function doesn't exist, ctypes' memmove function should be changed to accept objects implementing the buffer interface. Thomas |
|
From: Florian S. <flo...@gm...> - 2004-12-16 08:55:30
|
On Wed, 15 Dec 2004 10:57:45 -0800, Ray Schumacher <rj...@bl...> wrote: > At 06:31 PM 12/15/2004 +0100, Thomas Heller wrote: >> Should work, but builds a large temporary list. Best would be to avoid >> temporary objects at all, like this (for memmove, you need the latest >> ctypes release) (I assume that it is possible to get the address of a >> numarray buffer in a similar way as is possible for a normal array >> instance): > > I'm not sure it will be able to work... > The closest thing to .buffer_info() for numarray-addressing seems to be > info(), but it returns text: Try src._data (where src is a numarray.ArrayType, I don't know about Numeric, but it should be similar), it's a Memory object which should be able to function as a buffer. Regards, Florian Schulze |
|
From: Florian S. <flo...@gm...> - 2004-12-16 09:28:52
|
On Thu, 16 Dec 2004 09:55:09 +0100, Florian Schulze <flo...@gm...> wrote: > On Wed, 15 Dec 2004 10:57:45 -0800, Ray Schumacher <rj...@bl...> > wrote: > >> At 06:31 PM 12/15/2004 +0100, Thomas Heller wrote: >>> Should work, but builds a large temporary list. Best would be to avoid >>> temporary objects at all, like this (for memmove, you need the latest >>> ctypes release) (I assume that it is possible to get the address of a >>> numarray buffer in a similar way as is possible for a normal array >>> instance): >> >> I'm not sure it will be able to work... >> The closest thing to .buffer_info() for numarray-addressing seems to be >> info(), but it returns text: > > Try src._data (where src is a numarray.ArrayType, I don't know about > Numeric, but it should be similar), it's a Memory object which should be > able to function as a buffer. > > Regards, > Florian Schulze I posted some experiments on the numeric list, Ray asked the same question there and that forced me to try it out ;) Regards, Florian Schulze |
|
From: RayS <ra...@bl...> - 2004-12-16 15:45:53
|
Thanks Florian,
My preliminary assessment from trials yesterday (based only on a rough benchmark of idle time) is that
for i in range(2000): numArray[pntr+i] = buf[i]
is >10x slower than
numArray[pntr:pntr+2000] = map(None, buf)
which is 25% slower than using array.array() & memmove
memmove(array.buffer_info()[0]+pntr, buf)
I'll test the direct assignment today, as below, as subsequent operations on numarray should be faster than array. I don't recall why I didn't.
The numarray buffer interface is interesting, but I mainly need to do a (deep) copy out of the ctypes memory.
Since the main issue is that src (the ctypes obj) is being updated with 2KB A/D data @125x per second (250K samp./s), I need to get it out of the cytpes obj into storage as efficiently as possible. That storage must then work with slicing and FFT.
If the ctypes construct buf = (c_ulong * 2000)() supported slicing, I would just leave it there...
The small buffer_read thread I made has a time.sleep(.001) for when the A/D buffer is not ready, but even at .001s, it only gets hit once per cycle, if at all. It also seemed that sleep(.0001)==sleep(0); too bad.
Thanks,
Ray
At 10:14 AM 12/16/2004 +0100, Florian Schulze wrote:
>On Wed, 15 Dec 2004 20:16:07 -0800, RJ <ra...@sa...> wrote:
>
>>I'm posting here to see if numarray has a method that could work like array.buffer_info(). numarray.info() returns a text output that can't be used like the array method is to memmove() between ctypes and Python arrays without parsing, apparently.
>>
>>ctypes thread with Thomas Heller below. His main question: "Maybe you should ask to make sure that there's no way to copy between
>>objects implementing the buffer protocol with some Python function that
>>I do not know about?"
>
>I just tried some things:
>>>>import ctypes
>>>>a = (ctypes.c_int * 5)()
>>>>a[0] = 1; a[1] = 2; a[2] = 3; a[3] = 4; a[4] = 5
>>>>list(a)
>[1, 2, 3, 4, 5]
>>>>import numarray
>
>>>>buf = numarray.zeros(shape=20, type='i4')
>>>>buf
>array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
>>>>buf[2:7] = a
>>>>buf
>array([0, 0, 1, 2, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
>
>>>>temp = numarray.array(sequence=buffer(a), shape=5, type='i4')
>>>>temp
>array([1, 2, 3, 4, 5])
>>>>temp._data
><read-only buffer for 0x00FA9230, ptr 0x00BFA668, size 20 at 0x010D6DA0>
>>>>buffer(a)
><read-only buffer for 0x00FA9230, ptr 0x00BFA668, size 20 at 0x00FAF600>
>
>>>>a[2] = 10
>>>>temp
>array([ 1, 2, 10, 4, 5])
>
>The first block just creates the ctypes data.
>
>The second block uses slice assignment to copy the data from the ctypes array into the numarray.array.
>
>The third block uses the buffer interface to create a numarray.array which points to the same memory location as the ctypes array.
>
>The forth block illustrates that it's really the same memory.
>
>You have to benchmark which one of the two solutions is the better one for you.
>
>Regards,
>Florian Schulze
>
>
>______________________________________________
>Numpy-discussion mailing list
>Num...@li...
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion
|
|
From: Thomas H. <th...@py...> - 2004-12-16 16:09:20
|
RayS <ra...@bl...> writes:
> Thanks Florian,
>
> My preliminary assessment from trials yesterday (based only on a rough
> benchmark of idle time) is that
> for i in range(2000): numArray[pntr+i] = buf[i]
> is >10x slower than
> numArray[pntr:pntr+2000] = map(None, buf)
> which is 25% slower than using array.array() & memmove
> memmove(array.buffer_info()[0]+pntr, buf)
>
Do I understand it correctly that memmove is only slightly faster than
the 'numArray[..] = map(None, buf)' call? If so, interesting.
> I'll test the direct assignment today, as below, as subsequent
> operations on numarray should be faster than array. I don't recall why
> I didn't.
>
> The numarray buffer interface is interesting, but I mainly need to do
> a (deep) copy out of the ctypes memory. Since the main issue is that
> src (the ctypes obj) is being updated with 2KB A/D data @125x per
> second (250K samp./s), I need to get it out of the cytpes obj into
> storage as efficiently as possible. That storage must then work with
> slicing and FFT.
>
> If the ctypes construct buf = (c_ulong * 2000)() supported slicing, I
> would just leave it there...
But it *should* support slicing. Can't you write this instead of the map() call?
numArray[pntr:pntr+2000] = buf[0:2000]
Another possibility would be to let the ADC write the data directly into
the numArray, once you know it's address (and assuming it doesn't
change).
Thomas
|
|
From: Florian S. <flo...@gm...> - 2004-12-16 17:07:46
|
On Thu, 16 Dec 2004 17:10:10 +0100, Thomas Heller <th...@py...> wrote: > Another possibility would be to let the ADC write the data directly into > the numArray, once you know it's address (and assuming it doesn't > change). AFAIKT it won't change. One feature of numarray is mmap support and for this the memory must stay where it is. > Thomas Regards, Florian Schulze |
|
From: Ray S <ra...@bl...> - 2004-12-16 21:17:33
|
Hi Thomas,
I made a better direct comparison of ctypes and memmove, map etc., using
all the methods suggested so far. I think it's fairly valid.(?) Run on a
2.4GHz WinXP, Py2.3, a number of times.
Parsing numarray,info() was a royal pain, it writes directly to stdout!
I.e.:
>>> inf = src.info()
class: <class 'numarray.numarraycore.NumArray'>
shape: (32,)
strides: (8,)
byteoffset: 0
bytestride: 8
itemsize: 8
aligned: 1
contiguous: 1
data: <memory at 00815950 with size:256 held by object 00815930 aliasing object
00000000>
byteorder: little
byteswap: 0
type: Float64
>>> inf
>>> type(inf)
<type 'NoneType'>
I was surprised that assignment wasn't faster and that numarray assignment
was consistently ~.5% faster than Numeric.
The N vs. n memmove() flip-flopped for fastest, with array.array always slower.
Since the ctype does support slicing, I was considering leaving the data in
the device driver buffer (~1MB circluar) and poking into it, but memmove is
so much faster than slicing the ctype doing memmove()s to numarray.
I presume that I should check for numarray.iscontiguous( ) or is_c_array(
) first to be safe...
Results and code below.
Thanks to all for the help,
Ray
>python test.py
Array address 11443208
n address 17039424
N address 21102656
for loop 2027.0 us ea
map 1781.0 us ea
slice 1704.0 us ea
assign N 1244.0 us ea
assign n 1242.0 us ea
memmove 4.3831 us ea
memmove N 3.4773 us ea
memmove n 3.4803 us ea
_____________________________________________________
## test.py
import array
import numarray
import ctypes
import time
import string
import StringIO
import sys
buf = (ctypes.c_long * 2000)()
Array = array.array("l", [0]*10000)
n = numarray.zeros((1000000), numarray.Int32)
N = numarray.zeros((1000000), numarray.Int32)
#!!!!!!!!!!!!!!! arrrrgggg! !!!!!!!!!!!!!!!!
# n.info() writes directly to stdout!
stdout = sys.stdout
fileo = StringIO.StringIO()
sys.stdout = fileo
n.info()
ninfo = fileo.getvalue( )
fileo.close()
sys.stdout = stdout
stdout = sys.stdout
fileo = StringIO.StringIO()
sys.stdout = fileo
N.info()
Ninfo = fileo.getvalue( )
fileo.close()
sys.stdout = stdout
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
print 'Array address', Array.buffer_info()[0]
ninfo = string.split(ninfo)
nAddress = int(ninfo[20], 16)
print 'n address', nAddress
Ninfo = string.split(Ninfo)
NAddress = int(Ninfo[20], 16)
print 'N address', NAddress
t0 = time.clock()
for loop in range(1000):
for i in range(2000):
n[loop+i] = buf[i]
print 'for loop ', round((time.clock()-t0)*1000), 'us ea'
t0 = time.clock()
for loop in range(1000):
n[loop:loop+2000] = map(None, buf)
print 'map ', round((time.clock()-t0)*1000), 'us ea'
t0 = time.clock()
for loop in range(1000):
n[loop:loop+2000] = buf[0:2000]
print 'slice ', round((time.clock()-t0)*1000), 'us ea'
t0 = time.clock()
for loop in range(10000):
N[loop:loop+2000] = buf
print 'assign N', round((time.clock()-t0)*100), 'us ea'
t0 = time.clock()
for loop in range(10000):
n[loop:loop+2000] = buf
print 'assign n', round((time.clock()-t0)*100), 'us ea'
t0 = time.clock()
for loop in range(10000):
ctypes.memmove(10+Array.buffer_info()[0],
buf,
2000)
print 'memmove ', round((time.clock()-t0)*1, 4), 'us ea'
t0 = time.clock()
for loop in range(10000):
ctypes.memmove(10+NAddress,
buf,
2000)
print 'memmove N', round((time.clock()-t0)*1, 4), 'us ea'
t0 = time.clock()
for loop in range(10000):
ctypes.memmove(10+nAddress,
buf,
2000)
print 'memmove n', round((time.clock()-t0)*1, 4), 'us ea'
At 05:10 PM 12/16/2004 +0100, Thomas Heller wrote:
>RayS <ra...@bl...> writes:
>
> > Thanks Florian,
> >
> > My preliminary assessment from trials yesterday (based only on a rough
> > benchmark of idle time) is that
> > for i in range(2000): numArray[pntr+i] = buf[i]
> > is >10x slower than
> > numArray[pntr:pntr+2000] = map(None, buf)
> > which is 25% slower than using array.array() & memmove
> > memmove(array.buffer_info()[0]+pntr, buf)
> >
>
>Do I understand it correctly that memmove is only slightly faster than
>the 'numArray[..] = map(None, buf)' call? If so, interesting.
>
> > I'll test the direct assignment today, as below, as subsequent
> > operations on numarray should be faster than array. I don't recall why
> > I didn't.
> >
> > The numarray buffer interface is interesting, but I mainly need to do
> > a (deep) copy out of the ctypes memory. Since the main issue is that
> > src (the ctypes obj) is being updated with 2KB A/D data @125x per
> > second (250K samp./s), I need to get it out of the cytpes obj into
> > storage as efficiently as possible. That storage must then work with
> > slicing and FFT.
> >
> > If the ctypes construct buf = (c_ulong * 2000)() supported slicing, I
> > would just leave it there...
>
>But it *should* support slicing. Can't you write this instead of the
>map() call?
>
> numArray[pntr:pntr+2000] = buf[0:2000]
>
>Another possibility would be to let the ADC write the data directly into
>the numArray, once you know it's address (and assuming it doesn't
>change).
>
>Thomas
|
|
From: Ray S <ra...@bl...> - 2004-12-17 19:41:51
Attachments:
test.py
|
import win32process
si = win32process.STARTUPINFO()
p = win32process.CreateProcess(None, # module
"python test.py", #command line
None, #process security attributes
None, #thread security attributes
0, #handle inheritance flag
#win32process.THREAD_PRIORITY_TIME_CRITICAL, #creation flags REALTIME_PRIORITY_CLASS
win32process.REALTIME_PRIORITY_CLASS,
None, # process new environment setting
None, #start directory
si) #STARTUPINFO object specifying window appearance
|
|
From: Thomas H. <th...@py...> - 2004-12-21 18:49:18
|
Ray S <ra...@bl...> writes: > New results, corrected code... > > >python test.py > Array address: 11784976 > n address: 17039424 <memory at 01040040 with size:4000000 held by object > 01040020 aliasing object 00000000> > N address: 11384616 <built-in method __copy__ of array object at 0x00ADB6D8> > > for loop 1929.0 us ea > map 1630.0 us ea > slice 1589.0 us ea > assign n 1164.0 us ea > assign N 337.0 us ea > memmove A 3.9847 us ea > memmove n 3.2005 us ea The memmove results are probably still somewhat misleading, because of the adress arithmetic done in the timed loop, that *may* explain this: > I'm not sure why a memmove to Array is always slower than numarray. > I could not do a memmove() with Numeric because I couldn't find a > corresponding output to repr(n._data) (thanks for that tip); > repr(N.__copy__) gives the address of an object(array or method?). > dir(N) and dir(Numeric) were no help. In test.py I tried adding 40 > hex (al-la numarray) to get the first data element, then I tried 80. > Ninfo = string.split(repr(N.__copy__)) > NAddress = int(Ninfo[7][:-1], 16)+int(hex(80), 16) > Adding 40 to the address of the object apparently overwrites the > object, as a GC error is thrown. Adding 80 allows the loop to run, but > MS Win2K complains that python23.dll had a problem and wants to send > an error report. I might be overwriting the __copy__ method in the dll > for all I know - the error makes that likely. I have thought about all this a little bit, and it seems that it would be very useful to accept Python objects which implement the buffer interface as function parameters (currently only ctype instances, strings and integers are allowed). Of course, sometimes you don't want the data to be copied just at the beginning, but in this case you should wrap the object into a buffer object (with the buildin buffer() type/function). The buffer() call accepts optional parameters for offset and size, so address arithmetic would be done by the buffer object itself - nicer than to add more parameters to the memmove function and friends. How does that sound? Thomas |
|
From: Florian S. <flo...@gm...> - 2004-12-22 20:55:26
|
On Tue, 21 Dec 2004 19:50:24 +0100, Thomas Heller <th...@py...> wrote: > I have thought about all this a little bit, and it seems that it would > be very useful to accept Python objects which implement the buffer > interface as function parameters (currently only ctype instances, > strings and integers are allowed). > > Of course, sometimes you don't want the data to be copied just at the > beginning, but in this case you should wrap the object into a buffer > object (with the buildin buffer() type/function). The buffer() call > accepts optional parameters for offset and size, so address arithmetic > would be done by the buffer object itself - nicer than to add more > parameters to the memmove function and friends. > > How does that sound? Sounds good. I wonder what the python developers plan to do with buffer in the future, I read it's deprecated and should be redone. Regards, Florian Schulze |
|
From: Thomas H. <th...@py...> - 2004-12-23 15:57:53
|
Florian Schulze <flo...@gm...> writes: > On Tue, 21 Dec 2004 19:50:24 +0100, Thomas Heller <th...@py...> > wrote: > >> I have thought about all this a little bit, and it seems that it would >> be very useful to accept Python objects which implement the buffer >> interface as function parameters (currently only ctype instances, >> strings and integers are allowed). >> >> Of course, sometimes you don't want the data to be copied just at the >> beginning, but in this case you should wrap the object into a buffer >> object (with the buildin buffer() type/function). The buffer() call >> accepts optional parameters for offset and size, so address arithmetic >> would be done by the buffer object itself - nicer than to add more >> parameters to the memmove function and friends. >> >> How does that sound? > > Sounds good. > > I wonder what the python developers plan to do with buffer in the > future, I read it's deprecated and should be redone. The buffer interface and the buffer object are separate things, the latter is not loved very much. There have been changes in Python 2.4 which make buffer objects somewhat safer. The address of the underlying object is no longer stored in the buffer object anymore, it's only retrieved when needed. But I don't have the energy to start another buffer discussion on python-dev right now, and since ctypes is dangerous anyway, the buffer object is fine imo. Thomas |
|
From: RayS <ra...@bl...> - 2004-12-23 00:42:58
|
Hi,
At 05:10 PM 12/16/2004 +0100, Thomas Heller wrote:
>But it *should* support slicing. =20
Yes, I got it going correctly, again...
>Another possibility would be to let the ADC write the data directly into
>the numArray, once you know it's address (and assuming it doesn't
>change).
I'm trying to figure out how this would be accomplished; I currently crea=
te a (c_long*2000)() to receive the result of the driver call
buf =3D usb.EDRE_ADGetDataRaw(Sn,*Buf ,*BufSize)
where=20
usb =3D windll.EDRAPI #driver
Sn 32-bit unsigned integer Board=92s serial number.
*Buf Pointer to an array of 16-bit unsigned words
*BufSize Pointer to an array of 32-bit unsigned integers
As per you suggestion, the address and memmove() work fine, the address c=
alcs are easy.
So, how can I set up a windll call to the driver to plop data directly in=
to the numarray? (The 4us memmove() is actually fast enough; the driver c=
alls and other ops are now the majority of the time spent. I have 8000us =
between 2K buffers.)
Ray
Happy Holidays to all!=20
|
|
From: Thomas H. <th...@py...> - 2004-12-23 16:13:46
|
RayS <ra...@bl...> writes: > Hi, > At 05:10 PM 12/16/2004 +0100, Thomas Heller wrote: >>But it *should* support slicing. > > Yes, I got it going correctly, again... > >>Another possibility would be to let the ADC write the data directly into >>the numArray, once you know it's address (and assuming it doesn't >>change). > > I'm trying to figure out how this would be accomplished; I currently > create a (c_long*2000)() to receive the result of the driver call > buf = usb.EDRE_ADGetDataRaw(Sn,*Buf ,*BufSize) > where > usb = windll.EDRAPI #driver > Sn 32-bit unsigned integer Boards serial number. > *Buf Pointer to an array of 16-bit unsigned words > *BufSize Pointer to an array of 32-bit unsigned integers > > As per you suggestion, the address and memmove() work fine, the > address calcs are easy. So, how can I set up a windll call to the > driver to plop data directly into the numarray? Pass the integer address as second parameter. Thomas |
|
From: RayS <ra...@bl...> - 2005-02-01 16:37:57
|
At 04:56 PM 12/23/2004 +0100, you wrote: >> So, how can I set up a windll call to the >> driver to plop data directly into the numarray? > >Pass the integer address as second parameter. Thanks Thomas, I used that with numarray without apparent ill effects, but Todd Miller has also recently suggested that the pointer might not be reliable (which only means that it needs to be re-read each time). On a related topic, I asked over in Numpy-discussion http://sourceforge.net/mailarchive/forum.php?thread_id=6468875&forum_id=4890 if there was a similar possibility directly with Numeric (we had discussed this a bit here before http://sourceforge.net/mailarchive/message.php?msg_id=10368635) since it is faster for small-ish arrays. Part of it boils down to: could I use ctypes' pointer(offset) to read the Numeric structure's pointer-to-data[0]-value and then use it in memmove()? Is this reasonable? Alternatively, do you expect to implement the acceptance of Python objects which implement the buffer interface in the near future? Ray My post to Numpy-discussion: ################################################## Thanks Todd, Travis, Yes, Travis, I _was_ using ctypes' memmove() with numarray to great benefit. I had also understood that the address of the numarray data was stable. Originally, I was trying to move data from a hardware A/D driver DLL call to Python as quickly as possible and did some benchmarking, so as per Thomas Heller's suggestion on the ctypes list I used memmove(): http://sourceforge.net/mailarchive/forum.php?thread_id=6166311&forum_id=24606 from ctypes import * import array src = array.array("i", range(32)) dst = (c_int * 32)() memmove(dst, *src.buffer_info()) memmove() is orders of magnitude faster than map() or assignments. I then tested and found numarray faster for the rest of the tasks (FFT etc.) http://sourceforge.net/mailarchive/forum.php?thread_id=6205658&forum_id=24606 and so parsed the info() output and plugged the data[0] address into the call. Thomas Heller (in the above) suggested he make a mod to ctypes to accept Python objects which implement the buffer interface as function parameters, which would allow Numeric use once implemented. I like numarray's breadth of methods so I used it at the time, but in another try at speed-squeezing yesterday afternoon I found Numeric's FFT and subtraction to be >30% faster in this case, so I switched that code over (this increase includes the use of map() with Numeric to read the A/D in another thread). Using memmove() with Numeric would speed up the reader thread once again. At 08:06 AM 2/1/2005 -0500, Todd Miller wrote: >> >> What is the actual address of the first element? > >In C, look at a->data. I had read the Numeric API and looked at the PyObject structure, as Travis then suggested, but my question then is: if the offset from the "array object at 0x..." (object address value) to the array[0] address is not fixed and must be read from the pointer in the PyObject structure, can we get that pointer's value directly from Python or ctypes? ctypes pointer() "allows" poking in and directly reading memory: "It is also possible to use indexes different from 0, but you must know what you're doing when you use this: You access or change arbitrary memory locations when you do this." http://starship.python.net/crew/theller/ctypes/tutorial.html So, could I use ctypes' pointer(offset) to read the structure's pointer-to-data[0]-value and then use it in memmove()? I'll cross-post this to ctypes-users... >I think the "fragile data pointer" is generally useful information, but >not completely dependable so I gave it the garish name it has. >Comments? It's quite reasonable. Is the data pointer value really often changed for contiguous arrays in small stand-alone Python apps? If so, I may stick with map() to avoid having to re-read the pointer before each data buffer transfer. If anyone is interested I could post some relevant code snips from the office... Thanks for the help, Ray |
|
From: Thomas H. <th...@py...> - 2005-02-02 21:07:55
|
RayS <ra...@bl...> writes: > Alternatively, do you expect to implement the acceptance of Python > objects which implement the buffer interface in the near future? I hope to be able to make a new release this or next week. So far, I think objects implementing the buffer interface should *not* be allowed as function parameters, but *buffer objects* should be. The reason is that the former would break too much code. IIUC, you should then be able to call buffer() on a Numeric array instance and pass that to a function, or call buffer() on the _data member of a Numarray array instance. The buffer() object has been changed in Python 2.4 (or was it even 2.3?) so that it retrieves the memory block address from the original object each time it is needed, instead of getting it once and then storing it permanently. So it should be even safe if the original object changes its internal memory block. Thomas |
|
From: Ray S. <rj...@bl...> - 2005-02-02 22:15:31
|
At 10:09 PM 2/2/2005 +0100, Thomas Heller wrote: >RayS <ra...@bl...> writes: > > > Alternatively, do you expect to implement the acceptance of Python > > objects which implement the buffer interface in the near future? > >I hope to be able to make a new release this or next week. So far, I >think objects implementing the buffer interface should *not* be allowed >as function parameters, but *buffer objects* should be. The reason is >that the former would break too much code. > >IIUC, you should then be able to call buffer() on a Numeric array >instance and pass that to a function, or call buffer() on the _data >member of a Numarray array instance. > >The buffer() object has been changed in Python 2.4 (or was it even 2.3?) >so that it retrieves the memory block address from the original object >each time it is needed, instead of getting it once and then storing it >permanently. So it should be even safe if the original object changes >its internal memory block. Ahh, thanks for the insight. >>> import Numeric, string >>> N=Numeric.zeros((100,), Numeric.Float) >>> repr(N.__copy__) '<built-in method __copy__ of array object at 0x007B10B8>' >>> buffer(N, 0, 0) <read-only buffer for 0x007B10B8, ptr 0x00809A98, size 0 at 0x0082B0D0> >>> buffer(N, 50, 50) <read-only buffer for 0x007B10B8, ptr 0x00809ACA, size 50 at 0x00828E98> >>> string.split(buf.__repr__())[5] '0x00809A98,' >>> int(string.split(buf.__repr__())[5][:-1], 16) 8428184 I presume (!) that the ptr address could be passed to ctypes.memmove() At least, I'll try; memmove() does work fine with numarray.info address. The desire is to write form the ctypes DLL call into the Numeric array, so actually using the buffer(N) is out. Ray |
|
From: Thomas H. <th...@py...> - 2005-02-03 18:31:29
|
Ray Schumacher <rj...@bl...> writes: > >>> import Numeric, string > >>> N=Numeric.zeros((100,), Numeric.Float) > >>> repr(N.__copy__) > '<built-in method __copy__ of array object at 0x007B10B8>' > >>> buffer(N, 0, 0) > <read-only buffer for 0x007B10B8, ptr 0x00809A98, size 0 at 0x0082B0D0> > >>> buffer(N, 50, 50) > <read-only buffer for 0x007B10B8, ptr 0x00809ACA, size 50 at 0x00828E98> > >>> string.split(buf.__repr__())[5] > '0x00809A98,' > >>> int(string.split(buf.__repr__())[5][:-1], 16) > 8428184 > > I presume (!) that the ptr address could be passed to ctypes.memmove() > At least, I'll try; memmove() does work fine with numarray.info address. > > The desire is to write form the ctypes DLL call into the Numeric > array, so actually using the buffer(N) is out. Ray, it seems I don't understand you. Why should buffer() not work? Don't be confused that the buffer() object says <read-only buffer ...>! The buffer call only asks for readable memory..., but ctypes doesn't care about the readonly attribute - it will happily write into this memory. If this is too confusing, and this may well be, ctypes could expose a memory() function which would insist on read-write memory, but apart from that do the same that buffer does: c:\sf\ctypes\sandbox\tools\codegen>py23 Python 2.3.5c1 (#61, Jan 25 2005, 19:52:06) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. # readline available, tab: complete >>> from Numeric import * >>> import ctypes >>> from _ctypes import memory >>> n = zeros(3, Float) >>> n array([ 0., 0., 0.]) >>> n.__copy__ <built-in method __copy__ of array object at 0x00B2C890> >>> memory(n) <read-write buffer for 0x00B2C890, ptr 0x00869EF0, size 24 at 0x00B1FC00> >>> buffer(n) <read-only buffer for 0x00B2C890, ptr 0x00869EF0, size 24 at 0x00B1FAA0> >>> memory(n, 4) <read-write buffer for 0x00B2C890, ptr 0x00869EF4, size 20 at 0x00B2E8E0> >>> Thomas |
|
From: Ray S <ra...@bl...> - 2005-02-03 20:22:08
|
At 07:32 PM 2/3/2005 +0100, you wrote:
>Don't be confused that the buffer() object says <read-only buffer ...>!
>The buffer call only asks for readable memory..., but ctypes doesn't
>care about the readonly attribute - it will happily write into this
>memory.
Hi Thomas,
Yes, I was thinking of what the shell error said upon assignment...
Upon adding to some working code all is well:
>>> import Numeric, ctypes, string
>>> N = Numeric.zeros((10,), Numeric.Float)
>>> buf = buffer(N)
>>> buf
<read-only buffer for 0x008F9C28, ptr 0x008D7780, size 80 at 0x008FE220>
>>> int(string.split(repr(buf))[5][:-1], 16)
9271168
## numarray version
# nAddress = int(string.split(repr(N._data))[2], 16)
## Numeric version
NAddress = int(string.split(repr(buffer(N)))[5][:-1], 16)
## Load DLL here...
## do this to get data from the USB A/D's DLL
usb.GetData(usb.Sn, (bufferInsertPos * N.itemsize()) + NAddress,
ctypes.byref( (types.c_long *
buffersize)() ) )
Which is faster than getting data into a ctypes array (c_ulong *n)() and
then doing memmove() to Numeric - one less step.
Maybe this snip would be of help to some others, although more so to numpy
people.
Of course, the Python array works the same:
>>> a = array.array('l',[1,2,3])
>>> int(string.split(repr(buffer(a)))[5][:-1], 16)
8380408
>If this is too confusing, and this may well be, ctypes could expose a
>memory() function which would insist on read-write memory, but apart
>from that do the same that buffer does:
No, not confusing, just not clear to a non-expert C person that ctypes
ignores where Python is read-only. A simple note in the tutorial would be
fine.
Some over at numpy were also unaware of memmove()s' existence in the new
releases, and seemed interested.
Thanks again,
Ray
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X