numpy-discussion — Discussion list for all users of Numerical Python

[Numpy-discussion] array operation

[Dave B. <ba...@co...>]

I am trying to see if I can use where() or choose() to do this.  I can't
really figure it out.

I have a 2-d array data where each row is an observation and each column is
an attribute of the observation:

data =
[[.3, .2, 2.3,...]    <- observation 1
 [.7, 1.2, .4...]     <- observation 2
...]]

I have another 1-d array that contains a code for the class of object:

class = [0,1,0,1,1,3,2,0,...]

where class[i] = the class of the ith object in the data array.  Thus,
observation 1 above is class 0, observation 2 is class 1, and so on.

I want to select all objects of a given class from data array.  I can do
this with a loop

for i in range(ndat):
    if class == 0:
        do something
   ....

Is there a way to use where() or choose() to do this?  Would it be more
efficient?

Thanks,

Dave

RE: [Numpy-discussion] RE: numarray/Numeric upkeep?

[Perry G. <pe...@st...>]

Michael Stone wrote:

> >No one seems to be doing bugfixes for Numeric or numarray.
> >Nothing seems to have happened for several months.  Lots of bugs 
> have been
...

It certainly isn't true that nothing has happened for several months
with numarray. On what do you base this belief? While not all bugs
have been fixed, the oldest listed in the numarray bug tracker is
from December. Is there a bug you feel needs urgent attention?

Work is continuing and new releases will be coming out.

As to Paul's comments regarding when numarray will be ready,
my guess is when the following are complete:

- Package reorganization (make numarray a package)
- Optimization for small arrays (making numarray'speed with small arrays
   more comparable with Numeric; this is probably the single largest
   remaining item)
- Porting some well known packages such as MA (which Paul is working on),
   scipy, pyopengl and such to work with numarray. Some of this has been
   started.

There are other smaller things to do as well. But I'm hoping that
we can be done with these in a few months.

Perry

[Numpy-discussion] RE: numarray/Numeric upkeep?

[<pa...@pf...>]

Here are some of the factors leading to the slow rate of change of Numeri=
c
lately.
a. I changed to a new project and have had a lot of startup learning to
do. My new project uses Numeric but not in as central a way as my old one=
.
b. I mistakenly thought numarray would be ready sooner so that I was tryi=
ng
to let it slide.
c. I announced last year, in view of (a), that I was needing to be replac=
ed
as HeadNummie. It would be logical to turn this over to the Numarray peop=
le,
but they aren't ready to do it until Numarray is ready, so nothing happen=
ed.
d. Except for Travis, most of the other listed Numeric developers aren't
in fact doing patches, releases, etc.
e. Not all patches that are submitted are correct or desirable, historica=
lly.
I'm not saying anything about any patches you may have submitted, just po=
inting
out that applying them requires real work, not just mechanical patching.
In fact the rate of error in patches is quite high and I've learned to be=

cautious.
f. Some patches interfere with each other; for example, a patch for makin=
g
64 bit machines work right and a patch for some specific bug collided.

I've started to work on the MA for Numarray but I'm not able to do much
work on Numeric right now. This is a place where someone else has to help=
.


>-- Original Message --
>To: du...@us...
>Subject: numarray/Numeric upkeep?
>From: Michael Stone <mbr...@us...>
>Cc: <mbr...@us...>
>Date: Tue, 21 Jan 2003 11:32:03 -0800
>
>
>
>No one seems to be doing bugfixes for Numeric or numarray.
>Nothing seems to have happened for several months.  Lots of bugs have be=
en
>posted for Numeric, some easily fixable (I submitted one with a patch).
>
>Any idea if either project will become active again anytime soon?

Re: [Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Francesc A. <fa...@op...>]

A Dimarts 21 Gener 2003 19:41, Sebastian Haase va escriure:
> Hi,
> I think this is actually quite related to my post from Friday:
> [Numpy-discussion] make C array accessible to python without copy
>
> -> So, to reformulate: Who hold actually the array data in memory? Or:
> where gets the memory allocated and where/how many pointers to that exi=
st?=20
>   I understood the answer that Todd Miller gave, that there is such a t=
hing
> as a "buffer object" that does all the work, so then: one would just ha=
ve
> to take that and build a "new" numarray or Numeric structure around it=20
> (referring to the Subject of this email)   or  (in the case of my
> Friday-email)  just have that "buffer object" point to a different memo=
ry
> space (that got already allocated by the C-program) .
>
> Agree ? (Did I get it right?)

Well, so so. I think the buffer object is a property of numarray objects,
not Numeric objects. So, in the numarray =3D=3D> Numeric conversion proce=
ss you
may need to access the internals of the buffer (for example by using the
high level numarray C-API) and manage to obtain a data buffer (in the C
sense, not an object) that can be used to build the Numeric object (with =
the
help of the numarray object metadata). The opposite way needs something
similar but with inverted roles. See my previous message for a more in-de=
pth
explanation.

I think the conversion (without copying) is not a difficult process, but =
no
so-easy like that.

Well, I'm just a newcomer to numarray and my opinions about that may
perfectly be completely wrong, of course. Take them with caution!.

--=20
Francesc Alted

Re: [Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Francesc A. <fa...@op...>]

A Dimarts 21 Gener 2003 17:26, v=E0reu escriure:
> Francesc Alted wrote:
> >Anybody is aware of any function (either in C or Python or a mixture o=
f
> >both) to easily convert Numerical Python arrays from/to numarray array=
s?
>
> I think you should look at numarray.fromlist() and NumArray.tolist().  =
I
> think fromlist() will work on a nested sequence object,  and hence a
> Numeric array.

Yeah, I knew that, but I was looking for something more optimal.

>
> >I mean, I would like to use such a funtion that, without having to cop=
y
> >element by element all the data, be able to copy the data buffer (or e=
ven
> >use the same if possible at all) from one object to the other.
>
> I have not looked at this yet;   it's a very good question.  Note that
> going from numarray to Numeric there are issues with making the buffer
> well-behaved.

I think this should be not too difficult to achieve and I'll try to expla=
in
why.

When going from numarray to Numeric, numarray already have NA_InputArray
C-API function that returns a well-behaved array. But strictly speaking, =
we
don't even need a well-behaved array (this is a too restrictive condition=
)
as both Numeric and numarray support discontiguous data. Even the byteord=
er
should be not a problem, because, as Numeric itself has no such a propert=
y,
we can create a Numeric array that is in native order as the result and
byteswap the numarray object (if needed) before doing the conversion.

So, non-alignment remains as the only issue that may cause a buffer copy
during numarray =3D=3D> Numeric conversion. Is that correct?. If yes, it =
is
possible to do a workaround about that, i.e. we can still get a Numeric f=
rom
a numarray without copying the data in case of numarray misaligned object=
s?.

Regarding to going in the other sense (ie. Numeric =3D=3D> numarray), as
numarray supports discontiguity, misalignment and byteswapped data, this
conversion should not imply a data buffer copy at all.=20

Once we have a pointer to the data buffer, it is only a matter of
wrapping a Numeric or numarray object around it getting this info from th=
e
original object, and returning the new object as a result.

All in all, this conversion *seems* to be not a too difficult task.

Making such a conversion functions (in C, but also having Python
counterparts) available might represent to open the door to a co-existenc=
e
of Numeric and numarray objects in the same program, and that would easy =
the
numarray deployment in existing Numeric software.

Comments?

--=20
Francesc Alted

Re: [Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Sebastian H. <ha...@ms...>]

Hi,
I think this is actually quite related to my post from Friday:
[Numpy-discussion] make C array accessible to python without copy

-> So, to reformulate: Who hold actually the array data in memory? Or: where
gets the memory allocated and where/how many pointers to that exist?    I
understood the answer that Todd Miller gave, that there is such a thing as a
"buffer object" that does all the work, so then: one would just have to take
that and build a "new" numarray or Numeric structure around it  (referring
to the Subject of this email)   or  (in the case of my Friday-email)  just
have that "buffer object" point to a different memory space (that got
already allocated by the C-program) .

Agree ? (Did I get it right?)

Sebastian Haase

----- Original Message -----

From: "Francesc Alted" <fa...@op...>
To: <num...@li...>
Sent: Tuesday, January 21, 2003 8:44 AM
Subject: [Numpy-discussion] Conversion functions between Numeric and
numarray arrays?


Hi,

Anybody is aware of any function (either in C or Python or a mixture of
both) to easily convert Numerical Python arrays from/to numarray arrays?

I mean, I would like to use such a funtion that, without having to copy
element by element all the data, be able to copy the data buffer (or even
use the same if possible at all) from one object to the other.

Thanks,

[Numpy-discussion] Conversion functions between Numeric and numarray arrays?

[Francesc A. <fa...@op...>]

Hi,

Anybody is aware of any function (either in C or Python or a mixture of
both) to easily convert Numerical Python arrays from/to numarray arrays?

I mean, I would like to use such a funtion that, without having to copy
element by element all the data, be able to copy the data buffer (or even
use the same if possible at all) from one object to the other.

Thanks,

--=20
Francesc Alted

[Numpy-discussion] recarray2 re-visited

[Francesc A. <fa...@op...>]

Hi,

As I needed a byteswap() method for recarray, after a bit of hacking I've
made one myself. This is based on my own version of recarray to take
advantage of the _fields cache so as to both speed-up and simplify the ne=
w
code.

Basically, the new method takes a recarray, checking which columns are
numarray arrays and invoking their byteswap() method if needed. Easy, but
effective. Moreover, a _byteswap() and togglebyteorder() are provided to =
be
compatible with existing methods in NumArray objects.

As a plus, the recarray __str__ has been modified in order to allow a
printing having in mind the byteorder of the recarray, and improving the
speed of printing by a factor of 30, that can be handy in some situations=
=2E

Do with it whatever you want,

--=20
Francesc Alted

Re: [Numpy-discussion] Accessing a Numeric 'c' array from numarray]

[Francesc A. <fa...@op...>]

A Dissabte 18 Gener 2003 21:12, Todd Miller va escriure:
> >By the way, is it safe to assume that CharArray objects are contiguous=
? or
> >RawCharArray?.
>
> Mostly no.   Each fixed length element is stored as a contiguous
> sequence of bytes.  Anything goes for the rest,  so you need to look at
> the strides arrays and byteoffset.
>
> >The same question goes for RecArray objects.
>
> No.  It's possible to select every 10th record, for instance, in a
> slice.  I believe the resulting decimated array would be a discontiguou=
s
> view of the original.
>
> >Or it is always
> >convenient to check with iscontiguous() method if they are or not?.
>
> I'm not even certain the method works correctly for chararray and
> recarray.

Well, during my tests with numarray 0.4, iscontiguous() seems to work wel=
l,
both for chararrays and recarrays.

> In both cases,  I think I'd just forget about
> contiguity and use the strides arrays.

Yeah, but I still want to use iscontiguous() method just to speed-up a bi=
t
the code.

> You might argue that  records contain
> byteswapped and misaligned fields.   I don't have an immediate answer t=
o
> that.

Exactly, I am pondering how to deal with HDF5 objects coming from machine=
s
with a different endianess (misalignment is not a problem in my case) tha=
n
the local machine. But I think I can manage that by creating recarrays
buffers with the byteorder parameter set appropriately during the HDF5 ta=
ble
reads. Then, all the data can be read correctly because numarray will
byteswap the data whenever this recarray will be accessed.

Moreover, if this object is to be used frequently, I can speed-up the acc=
ess
to this recarray by byteswapping the columns (as arrays) using their
byteswap() method. In the future it would be nice to provide a generica
byteswap method for recarrays.

Thanks,

--=20
Francesc Alted

[Numpy-discussion] numarray bug?

[<ve...@em...>]

Hi, 
 
The following gives an error: 
 
>>> print numarray.Int8 == numarray.Any 
Traceback (most recent call last): 
  File "<stdin>", line 1, in ? 
  File "/usr/local/lib/python2.2/site-packages/numarray/numerictypes.py", line 
102, in __cmp__ 
    return genericTypeRank.index(self.name) - 
genericTypeRank.index(other.name) 
ValueError: list.index(x): x not in list 
 
A bug? 
 
Cheers, Peter 
 
-- 
Dr. Peter J. Verveer 
Cell Biology and Cell Biophysics Programme 
EMBL 
Meyerhofstrasse 1 
D-69117 Heidelberg 
Germany 
Tel. : +49 6221 387245 
Fax  : +49 6221 387242 
Email: ve...@em... 
 
 

Re: [Numpy-discussion] Accessing a Numeric 'c' array from numarray]

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>A Dissabte 18 Gener 2003 17:41, Todd Miller va escriure:
>  
>
>>>I just want to access the buffer data, and the shape of this object from C
>>>(well, I'm actually using Pyrex, but I think this is not important). Is
>>>that possible by only using numarray C calls?
>>>      
>>>
>>Look at Lib/chararray.py and Src/_chararraymodule.c.
>>
>>If you can handle using a CharArray or RawCharArray, try:
>>
>>1. call NA_updateDataPtr( array ) to refresh the data buffer pointer in
>>the PyArrayObject.  Even _chararraymodule.c doesn't do this right yet.
>>
>>2. call NA_OFFSETDATA(array) to add the byteoffset to the pointer.
>>
>>3. shape, strides, and itemsize should be directly accessible from the
>>PyArrayObject.
>>    
>>
>
>Ok. I'll try to do that.
>
>  
>
>>CharArray has some extra stripping and padding semantics; these are lazy
>>and hence absent without extra care in C.  RawCharArray has none.
>>
>>    
>>
>
>By the way, is it safe to assume that CharArray objects are contiguous? or
>RawCharArray?.
>
Mostly no.   Each fixed length element is stored as a contiguous 
sequence of bytes.  Anything goes for the rest,  so you need to look at 
the strides arrays and byteoffset.

>The same question goes for RecArray objects. 
>
No.  It's possible to select every 10th record, for instance, in a 
slice.  I believe the resulting decimated array would be a discontiguous 
view of the original.  

>Or it is always
>convenient to check with iscontiguous() method if they are or not?.
>
I'm not even certain the method works correctly for chararray and 
recarray.  

I think the portion of chararray that has been written in C considers 
array strides.
recarray is pure python.  In both cases,  I think I'd just forget about 
contiguity and use the strides arrays.

> In case
>these objects can be non-contiguous, I guess there's still not a function
>like NA_InputArray that works with CharArray or RecArray objects in order to
>obtain well-behaved objects. Is that true?
>
True.  But neither recarray nor chararray really has behavedness 
problems like misalignment,
byteswapping, or type conversion.  I think contiguity is the only issue, 
and that is solved
just by calling .copy().  You might argue that  records contain 
byteswapped and misaligned fields.   I don't have an immediate answer to 
that.

My preference is to use strides and forget about contiguity,  but you 
could also make contiguous copies simply.  Noone I'm aware of has yet 
tried access to misbehaved records in C.

>
>I think it would be possible to me to include support for numarray objects
>in next release of PyTables. 
>
Great!

>Thanks!,
>  
>

Re: [Numpy-discussion] Accessing a Numeric 'c' array from numarray]

[Francesc A. <fa...@op...>]

A Dissabte 18 Gener 2003 17:41, Todd Miller va escriure:
> >I just want to access the buffer data, and the shape of this object fr=
om C
> >(well, I'm actually using Pyrex, but I think this is not important). I=
s
> > that possible by only using numarray C calls?
>
> Look at Lib/chararray.py and Src/_chararraymodule.c.
>
> If you can handle using a CharArray or RawCharArray, try:
>
> 1. call NA_updateDataPtr( array ) to refresh the data buffer pointer in
> the PyArrayObject.  Even _chararraymodule.c doesn't do this right yet.
>
> 2. call NA_OFFSETDATA(array) to add the byteoffset to the pointer.
>
> 3. shape, strides, and itemsize should be directly accessible from the
> PyArrayObject.

Ok. I'll try to do that.

>
> CharArray has some extra stripping and padding semantics; these are laz=
y
> and hence absent without extra care in C.  RawCharArray has none.
>

By the way, is it safe to assume that CharArray objects are contiguous? o=
r
RawCharArray?. The same question goes for RecArray objects. Or it is alwa=
ys
convenient to check with iscontiguous() method if they are or not?. In ca=
se
these objects can be non-contiguous, I guess there's still not a function
like NA_InputArray that works with CharArray or RecArray objects in order=
 to
obtain well-behaved objects. Is that true?

I think it would be possible to me to include support for numarray object=
s
in next release of PyTables. Thanks!,

--=20
Francesc Alted

Re: [Numpy-discussion] Accessing a Numeric 'c' array from numarray]

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>Hi,
>
>I'm trying to make a C array from a Numeric "c" (Character) typecode array
>using the high level call:
>
>NA_InputArray(PyObject *numarray, NumarrayType t, int requires)
>
Unified handling of character arrays and numeric arrays doesn't exist 
yet in numarray.  There is no C-API for the chararray module because we 
haven't needed one.  But CharArrays are NDArrays and have attributes 
stored in PyArrayObjects just like numarrays.

>with no success.
>
>As I have been able to access all the other types (i.e.
>'1','b','s','i','l','f','d') successfully, perhaps character type is not
>supported?
>
>In the NumarrayType enum, there is no tChar, but I've tried tUInt8 and tAny
>as the value for NumarrayType parameter, but both choices issues the same
>error:
>
>Traceback (most recent call last):
>  File "table-tree2.py", line 77, in ?
>    h5file.createArray('/columns', 'name', array(names), "Name column")
>  File "/home/falted/PyTables/pytables-0.3/tables/File.py", line 400, in 
>createArray
>    setattr(group, name, object)
>  File "/home/falted/PyTables/pytables-0.3/tables/Group.py", line 355, in 
>__setattr__
>    value._f_putObjectInTree(name, self)
>  File "/home/falted/PyTables/pytables-0.3/tables/Leaf.py", line 71, in 
>_f_putObjectInTree
>    self.create()
>  File "/home/falted/PyTables/pytables-0.3/tables/Array.py", line 83, in 
>create
>    self.createArray(self.object, self.title)
>  File "/home/falted/PyTables/pytables-0.3/src/hdf5Extension.pyx", line 913, 
>in createArray
>    array = NA_InputArray(arr, numfmt2[arr.typecode()], C_ARRAY)
>libnumarray.error: getShape: sequence object nested more than MAXDIM deep.
>
NA_InputArray was intended to accept non-numeric sequences.  It could 
report this better...

>although I was passing only a Numeric 'c' with a rather small shape (10,16).
>
>I just want to access the buffer data, and the shape of this object from C
>(well, I'm actually using Pyrex, but I think this is not important). Is that
>possible by only using numarray C calls?
>
Look at Lib/chararray.py and Src/_chararraymodule.c.

If you can handle using a CharArray or RawCharArray, try:

1. call NA_updateDataPtr( array ) to refresh the data buffer pointer in 
the PyArrayObject.  Even _chararraymodule.c doesn't do this right yet.

2. call NA_OFFSETDATA(array) to add the byteoffset to the pointer.

3. shape, strides, and itemsize should be directly accessible from the 
PyArrayObject.

CharArray has some extra stripping and padding semantics; these are lazy 
and hence absent without extra care in C.  RawCharArray has none.

CharArrays are really arrays of fixed length strings of bytes.  The 
string length is defined by the array itemsize.

>Thanks,
>
>  
>
Todd

[Numpy-discussion] Accessing a Numeric 'c' array from numarray

[Francesc A. <fa...@op...>]

Hi,

I'm trying to make a C array from a Numeric "c" (Character) typecode arra=
y
using the high level call:

NA_InputArray(PyObject *numarray, NumarrayType t, int requires)

with no success.

As I have been able to access all the other types (i.e.
'1','b','s','i','l','f','d') successfully, perhaps character type is not
supported?

In the NumarrayType enum, there is no tChar, but I've tried tUInt8 and tA=
ny
as the value for NumarrayType parameter, but both choices issues the same
error:

Traceback (most recent call last):
  File "table-tree2.py", line 77, in ?
    h5file.createArray('/columns', 'name', array(names), "Name column")
  File "/home/falted/PyTables/pytables-0.3/tables/File.py", line 400, in=20
createArray
    setattr(group, name, object)
  File "/home/falted/PyTables/pytables-0.3/tables/Group.py", line 355, in=
=20
__setattr__
    value._f_putObjectInTree(name, self)
  File "/home/falted/PyTables/pytables-0.3/tables/Leaf.py", line 71, in=20
_f_putObjectInTree
    self.create()
  File "/home/falted/PyTables/pytables-0.3/tables/Array.py", line 83, in=20
create
    self.createArray(self.object, self.title)
  File "/home/falted/PyTables/pytables-0.3/src/hdf5Extension.pyx", line 9=
13,=20
in createArray
    array =3D NA_InputArray(arr, numfmt2[arr.typecode()], C_ARRAY)
libnumarray.error: getShape: sequence object nested more than MAXDIM deep=
=2E

although I was passing only a Numeric 'c' with a rather small shape (10,1=
6).

I just want to access the buffer data, and the shape of this object from =
C
(well, I'm actually using Pyrex, but I think this is not important). Is t=
hat
possible by only using numarray C calls?

Thanks,

--=20
Francesc Alted

Re: [Numpy-discussion] make C array accessible to python without copy

[Todd M. <jm...@st...>]

Sebastian Haase wrote:

>Hi,
>What is the C API to make an array that got allocated,
>let's say, by  a = new short[512*512],
>accessible to python as numarray.
>
What you want to do is not currently supported well in C.  The way to do 
what you want is:

1.  Create a buffer object from your C++ array.  The buffer object can 
be built such that it refers to the original copy of the data.

2.  Call  back into Python (numarray.NumArray) with your buffer object 
as the buffer parameter.

You can scavenge the code in NA_newAll (Src/newarray.ch) for most of the 
callback.

>I tried NA_New - but that seems to make a copy.
>I would need it to use the original memory space
>so that I can "observe" the array from Python WHILE
>the underlying C array changes (it's actually a camera image)
>
That sounds cool!

>
>Thanks,
>Sebastian Haase
>
>
>
>
>-------------------------------------------------------
>This SF.NET email is sponsored by: Thawte.com - A 128-bit supercerts will
>allow you to extend the highest allowed 128 bit encryption to all your 
>clients even if they use browsers that are limited to 40 bit encryption. 
>Get a guide here:http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0030en
>_______________________________________________
>Numpy-discussion mailing list
>Num...@li...
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>  
>

[Numpy-discussion] make C array accessible to python without copy

[Sebastian H. <ha...@ms...>]

Hi,
What is the C API to make an array that got allocated,
let's say, by  a = new short[512*512],
accessible to python as numarray.

I tried NA_New - but that seems to make a copy.
I would need it to use the original memory space
so that I can "observe" the array from Python WHILE
the underlying C array changes (it's actually a camera image)

Thanks,
Sebastian Haase

Re: [Numpy-discussion] Exporting Numpy C functionality to other extension modules

[Todd M. <jm...@st...>]

Take a look at the attached extension module "testlite" which 
demonstrates the technique I evolved from this discussion. As we 
discussed,  this usage pattern enables the construction of an extension 
which will take advantage of numarray if it is there,  but will continue 
to work if the user has not installed numarray.  Here's how it works:

1. I created a new API function,  PyArray_isArray() which is safe to 
call in all contexts.  I defined it as:

 #define PyArray_isArray(o) (PyArray_API && NA_isNumArray(o))

I added NA_isNumArray(o) to the numarray C-API because it was the easy 
way  to do it.

2. Ordinary API functions are safe to call once an object has been 
identified to be a numarray because it implies (locally) that the 
PyArray_API pointer has been initialized.

3. I tried out the standard import_array() code and added some cleanup 
for the case where numarray is not installed.  

The only caveat I see at this point is that you are required to include 
numarray headers in order to use this.  In numarray's case,  this might 
necessitate header updates and/or function call modifications.  The 
numarray C-API should stabilize pretty soon,  but I don't think its 
quite there yet.

The same approach should apply to Numeric.

This stuff is in numarray CVS now and should be in the next numarray 
release.

Todd

Re: [Numpy-discussion] Exporting Numpy C functionality to other extension modules

[Todd M. <jm...@st...>]

Chris Barker wrote:

>Jack Jansen wrote:
>
>  
>
>>An extension module that knows about this protocol and gets passed an
>>object that it think might be a Numeric array checks whether the object
>>has an __Numeric_C_interface attribute. If so it retrieves it, checks
>>that it is a Cobject, gets the descriptor and tests it for
>>compatibility and if it is compatible gets the cobject pointer and
>>happily calls all the Numeric routines it needs.
>>    
>>
>
>Wow Jack! are single handely going to impliment all my pet projects that
>I'm too stupid to know how to do my self ? (the other one was Universal
>text file support)
>
>I can only barely follow what you're suggesting, but I still have a
>question about it. It seems while this would provide a way ro an
>extension module to identify whether an object was a Numeric array, and
>then get a pointer to it, how would it know the API for dealing with the
>arrays, without the Numeric header file? Or would you have to include
>the header file when compiling, but not need the library at runtime
>unless it was actually used, which seems a reasonable compromise.
>
>If this would work, I think it's a great idea. Short of including
>NumArray with the standard library (which I imagine is a least a couple
>of Python releases away), it would be a great solution for folks that
>are writing extensions that they want to be able take advantage of
>Numeric when it's there, but not require it.
>
>Do any of the primary Numarray developers think this is a good and
>doable idea?
>  
>
Roll out the time machine...  it's already done.

As long as you don't define the macros PY_ARRAY_UNIQUE_SYMBOL or 
NO_IMPORT_ARRAY,  any file that includes arrayobject.h gets a static 
copy of PyArray_API.

If the module executes import_array() at an appropriate time,  normally 
module initialization, but not necessarily,  the static PyArray_API gets 
filled in and becomes usable.    The import_array() call is critical; 
 without it,  API calls through the static PyArray_API are calls to NULL 
and segfault.

I think that if Numeric is not present,  and you call import_array(),   
it will fail quietly but leave the Python error status set.   So it 
might make sense to call PyErr_Clear() after doing import_array().  

>-Chris
>
So it sounds like your whole "weak linkage" scheme is plausible now with 
Numeric (maybe even numarray!), as would be a minimal API module.

1.  We discussed yesterday how to determine if an object is a Numeric 
array w/o even compiling with arrayobject.h.   The important idea there 
was that if Numeric is not present,  the "isarray" (or whatever) 
function will return false rather than segfaulting because the API 
pointer isn't filled in.

2. Call API functions in contexts where you know you're looking at 
Numeric arrays, i.e.,  right after isarray().  This creates a guard 
which prevents you from calling API functions when Numeric is not present.

3.  Call import_array() at some time before using the API functions, 
 possibly at module init time, failing quietly and clearing the error in 
installations where Numeric is not installed.


Todd

Re: [Numpy-discussion] Optionally using Numeric in another compiledextension package.

[<pet...@no...>]

On Thu, 16 Jan 2003, Chris Barker wrote:

> pet...@no... wrote:
> 
> > You can synchronise the output buffer using XSync(3) and then do the
> > timing.

Oops, that should be XSynchronize(3).

[...]

> I suppose I could try including Xlib, then calling XSync, but I need to
> pass a reference to a disply. I have not idea how to get that. 
> 
> Any hints?

wxGetDisplayName() gives the Display name but not a pointer to the display 
structure. So this is not much help.

In gtk+, any program can be called with --sync to aid debugging. I'd guess 
wxWindows may allow you to do the same.

Peter

Re: [Numpy-discussion] Exporting Numpy C functionality to other extension modules

[Chris B. <Chr...@no...>]

Jack Jansen wrote:

> An extension module that knows about this protocol and gets passed an
> object that it think might be a Numeric array checks whether the object
> has an __Numeric_C_interface attribute. If so it retrieves it, checks
> that it is a Cobject, gets the descriptor and tests it for
> compatibility and if it is compatible gets the cobject pointer and
> happily calls all the Numeric routines it needs.

Wow Jack! are single handely going to impliment all my pet projects that
I'm too stupid to know how to do my self ? (the other one was Universal
text file support)

I can only barely follow what you're suggesting, but I still have a
question about it. It seems while this would provide a way ro an
extension module to identify whether an object was a Numeric array, and
then get a pointer to it, how would it know the API for dealing with the
arrays, without the Numeric header file? Or would you have to include
the header file when compiling, but not need the library at runtime
unless it was actually used, which seems a reasonable compromise.

If this would work, I think it's a great idea. Short of including
NumArray with the standard library (which I imagine is a least a couple
of Python releases away), it would be a great solution for folks that
are writing extensions that they want to be able take advantage of
Numeric when it's there, but not require it.

Do any of the primary Numarray developers think this is a good and
doable idea?

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Optionally using Numeric in another compiledextension package.

[Chris B. <Chr...@no...>]

pet...@no... wrote:

> You can synchronise the output buffer using XSync(3) and then do the
> timing.

I'd love to try this, but I confess I have no idea how! I'm working with
the *.i files that tell swig what to add when creating wrappers around
wxWindows for Python. wxWindows is using wxGTK, which is using GTK,
which is using Xlib (I think, so I'm pretty far away from X, and I
barely know enough C/C++ to attempt this.

I suppose I could try including Xlib, then calling XSync, but I need to
pass a reference to a disply. I have not idea how to get that. 

Any hints?

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[<pet...@no...>]

On Wed, 15 Jan 2003, Chris Barker wrote:
[...]

> My idea is that I could make sure Numeric was around at compile time, so
> I could use all the Numeric API to access the array data, but it
> wouldn't have to be installed at runtime, as none of the Numeric calls
> would be executed if Numeric hadn't been imported. Would this work, or
> would the system try to load the .dll or .so or whatever even if the
> calls weren't executed?

One way is to import a dynamic library, explicitly, which has glue code to
handle the array objects when you need them.

[...]

> My timing indicated something different, but I'm using Linux/wxGTK/X11,
> and I think the drawing calls return after the message has been sent to
> X, but X may not have completed the actual drawing yet.

That's right. X's communication model between client and server is
asynchronous.

> This means that I'm not timing the whole process, and if I did, I might
> not see such a difference.

You can synchronise the output buffer using XSync(3) and then do the 
timing.

Peter

Re: [Numpy-discussion] Exporting Numpy C functionality to other extension modules

[Francesc A. <fa...@op...>]

A Dimecres 15 Gener 2003 23:33, Jack Jansen va escriure:
> Actually, wrt my previous message on cobjects for communicating between
> extension modules, we can do one better!
>
> This is an idea I've been toying with for the MacPython extension
> types, and I think it's applicable to Numeric too. It goes as follows.
>
> Each Numeric object has an attribute with a well-known name, lets call
> it "__Numeric_C_interface". This is a Cobject, and it is shared among
> all Numeric objects of the same type. The value of this C object is a
> pointer to a C structure with pointers to all the C routines you might
> want to call on the object, basically the PyArray_API structure (I
> think). The descr of the C object is a string with the version number
> of this particular PyArray_API structure.
>
> An extension module that knows about this protocol and gets passed an
> object that it think might be a Numeric array checks whether the object
> has an __Numeric_C_interface attribute. If so it retrieves it, checks
> that it is a Cobject, gets the descriptor and tests it for
> compatibility and if it is compatible gets the cobject pointer and
> happily calls all the Numeric routines it needs.

That's a nice idea. But I see two drawbacks:

- numarray needs to be reworked to include the Cobject descriptors, altho=
ugh
I don't know if this would be difficult or not.

- you still need to have Numeric or numarray installed on the client
machine. This could be the usual case, but what about extensions that wan=
t
to use Numeric internally (because a number of reasons, like better numbe=
r
representation, convenient interface to C, etc) without forcing the user =
to
install it?

However, designing a small library with a minimalist API (I'm thinking in
something similar to zlib) could be very handy in allowing extensions (bu=
t
also native python modules) to deal with numarray objects.=20

As I said before, this would require the user to install only this small
library, but it can also be included in the application or package. Howev=
er,
this second alternative can be tricky, as Chris Barker has signaled, beca=
use
the different numarray versions coming in the future. But IMO a series of
factors may alleviate this handicap:

- The numarray data structure should be very stable, as improvements are
normally made at the functionality level.

- The library should provide a minimalistic, high level API that, if it i=
s
well designed, should cope with small modifications in the numarray data
structures.=20

- Finally, when these differences has to be added, and that would break t=
he
current API, this version should be marked as a major release,
and existing extensions (or whatever software that is embedding the libra=
ry)
will know that they have to release new versions if they want to support =
the
newest objects. But, hopefully, that should happen quite unfrequently.

Of course, this small library should cope with both numarray and Numeric =
(at
least, the not too old versions of it) objects. But I think this shouldn'=
t
pose a big problem as the actual numarray API already can do that.

This logical separation between structure and functionality migth also le=
ad
to a better acceptation by numerical software cratftsmen, as they can be
more confident in that the API to deal with numarray objects will be quit=
e
stable throughout the time.

Well, this is just a thought. I must confess that I'm so interested on th=
at
issue because I really want to support numarray objects in my project, an=
d
I'm just wondering which is the best way to do that without creating too
much nuissance to the users. In fact, I'm pondering to build up such a
library myself, but that can be a waste of time if I've to redone it in
every numarray release.

Cheers,

--=20
Francesc Alted

[Numpy-discussion] Exporting Numpy C functionality to other extension modules

[Jack J. <Jac...@or...>]

Actually, wrt my previous message on cobjects for communicating between 
extension modules, we can do one better!

This is an idea I've been toying with for the MacPython extension 
types, and I think it's applicable to Numeric too. It goes as follows.

Each Numeric object has an attribute with a well-known name, lets call 
it "__Numeric_C_interface". This is a Cobject, and it is shared among 
all Numeric objects of the same type. The value of this C object is a 
pointer to a C structure with pointers to all the C routines you might 
want to call on the object, basically the PyArray_API structure (I 
think). The descr of the C object is a string with the version number 
of this particular PyArray_API structure.

An extension module that knows about this protocol and gets passed an 
object that it think might be a Numeric array checks whether the object 
has an __Numeric_C_interface attribute. If so it retrieves it, checks 
that it is a Cobject, gets the descriptor and tests it for 
compatibility and if it is compatible gets the cobject pointer and 
happily calls all the Numeric routines it needs.
--
- Jack Jansen        <Jac...@or...>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -

Re: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Jack J. <Jac...@or...>]

On woensdag, jan 15, 2003, at 19:01 Europe/Amsterdam, Chris Barker 
wrote:

> Paul F Dubois wrote:
>>
>> If you could do:
>> try:
>>     import Numeric
>>     haveNumeric = 1
>> except:
>>     haveNumeric = 0
>>
>> in some initialization routine, then you could use this flag.
>> Alternately you could test on the fly
>> 'Numeric' in [m.__name__ for m in sys.modules]
>
> Thanks, but I'm talking about doing this at the C++ level in an
> extension package, not at the Python level. This kind of thing is Soo
> much easier in Python, of course!

This can be done, but it is difficult, and you need the cooperation of 
both parties (Numeric and wxPython, in this case). The problem is that 
you need a way to pass C pointers from one extension module to the 
other. One of the pointers you want to pass is the PyTypeObject, so you 
can check that an object passed in from Python is of the correct type. 
Another is the address of some C routine that will get you a C pointer 
to the data. The first one may be visible from Python (so you can get 
at it through normal means) but the second one won't be.

The dirty way to do this (and you should probably avoid this) is to put 
these pointers into Python integers in the supplying module, and put 
them in the module namespace with a funny name 
(__ConvertToCPointerAddress). In wxPython you import Numeric, and if it 
succeeds you look up the funny name, convert the Python integer to a C 
pointer, cross your fingers, and call the address.

A cleaner way to do this is with cobject objects. These are in the 
core, in Objects/cobject.c. Numeric exports a cobject (again named 
__ConvertToCPointerAddress) with the address of the routine as the 
value. But, and this is the nice bit, cobjects can be passed along by 
Python code but can't be fiddled with. And cobject.c even provides a C 
function PyCObject_Import(char *modulename, char *attributename) which 
directly returns you the pointer you're looking for by importing the 
module, looking up the name, checking that it's a cobject and 
extracting the value.

And it even has support for "protocols": Cobjects have an extra field 
called the description, again only settable and readable from C. 
Modules that don't know about each others' existence could still decide 
on a common description that would signify that the pointer in the 
cobject has a specific meaning. We could decide here that if the 
description is the C string "this pointer is a function that you pass 
one Python object and that returns the data just as Numeric would store 
it" would fit that bill, and anyone in the world writing an extension 
module could follow the protocol.
--
- Jack Jansen        <Jac...@or...>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- Emma 
Goldman -