numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] Optionally using Numeric in another compiled

[<gve...@gr...>]

> Gerard Vermeulen wrote:
> > I just want to point out that PyQwt plots NumPy arrays. I have played
> > a little bit with the Scipy-wxWindows interface, but it is no match
> > for PyQwt (I display x-y data with 16000 points).
> 
> Thanks for the tip, I'll check it out. I think what you have there is
> that the plotting is all done at the C++ level, expecting some kind of
> sequence of data points. That's exactly what I want to adress with
> wxPython: being able to pass in a whole sequence and have the looping
> done at the C++ level.
>
Yes, I am using PyArray_ContiguousFromObject() to convert any sequence
into a NumPy array before copying the data into Qwt's double arrays. 
>
> Have you ever tested whether it's fster or slower to plot data passed in
> as a list vs. a NumPy array?
>
I did not test it, but there is certainly more overhead if you pass
a list or a tuple into PyArray_ContiguousFromObject() than a NumPy array
> 
> How do you access the data in the passed in sequence? Do you use:
> PySequence_GetItem ?
> 
No, see above. The code looks like (in "sip" language, sip is a sort of
swig, but more specialized to C++ and Qt):

    void setData(double *, double *, int);
%MemberCode
    PyObject *xSeq, *ySeq;
    $C *ptr;
    if (sipParseArgs(&sipArgsParsed, sipArgs, "mOO",
                     sipThisObj, sipClass_$C, &ptr, &xSeq, &ySeq)) {
        PyArrayObject *x = (PyArrayObject *)
            PyArray_ContiguousFromObject(xSeq, PyArray_DOUBLE, 1, 0);
        if (!(x))
            return 0;
        PyArrayObject *y = (PyArrayObject *)
            PyArray_ContiguousFromObject(ySeq, PyArray_DOUBLE, 1, 0);
        if (!(y))
            return 0;
        int size;
        Py_BEGIN_ALLOW_THREADS
        size = (x->dimensions[0] < y->dimensions[0]) ?
            x->dimensions[0] : y->dimensions[0];
        ptr->setData((double*)(x->data), (double*)(y->data), size);
        Py_END_ALLOW_THREADS

        Py_DECREF(x);
        Py_DECREF(y);

        Py_INCREF(Py_None);
        return Py_None;
    }
%End

The setData calls copy the data.
>
> thanks for the tip. Qwt (and PyQwt) look very nice, I may have to
> reconsider using PyQT!
> 

Gerard

>
> -Chris
> 
> 
> 
>  
> > Take a look at http://gerard.vermeulen.free.fr
> > 
> > PyQwt is an addon for PyQt (a Python wrapper for Qt) that knows nothing
> > about NumPy
> > 
> > Maybe it is possible to make a NumPy plot add-on for wxWindows, too.
> > 
> > Gerard
> > 
> > On Wed, Jan 15, 2003 at 09:22:20AM -0800, Chris Barker wrote:
> > > Hi folks,
> > >
> > > I use Numeric an wxPython together a lot (of course I do, I use Numeric
> > > for everything!).
> > >
> > > Unfortunately, since wxPython is not Numeric aware, you lose some real
> > > potential performance advantages. For example, I'm now working on
> > > expanding the extensions to graphics device contexts (DCs) so that you
> > > can draw a whole bunch of objects with a single Python call. The idea is
> > > that the looping can be done in C++, rather than Python, saving a lot of
> > > overhead of the loop itself, as well as the Python-wxWindows translation
> > > step.
> > >
> > > For drawing thousands of points, the speed-up is substantial. It's less
> > > substantial on more complex objects (rectangles give a factor of two
> > > improvement for ~1000 objects), due to the longer time it takes to draw
> > > the object itself, rather than make the call.
> > >
> > > Anyway, at the moment, Robin Dunn has the wrappers set up so that you
> > > can pass in a NumPy array (or, indeed, and sequence) rather than a list
> > > or tuple of coordinates, but it is faster to use a list than a NumPy
> > > array, because for arrays, it uses the generic PySequence_GetItem call.
> > > If we used the NumPy API directly, it should be faster than using a
> > > list, not slower! THis is how a representative section of the code looks
> > > now:
> > >
> > >
> > > bool      isFastSeq  = PyList_Check(pyPoints) ||
> > > PyTuple_Check(pyPoints);
> > > .
> > > .
> > > .
> > >                 // Get the point coordinants
> > >                 if (isFastSeq) {
> > >                     obj = PySequence_Fast_GET_ITEM(pyPoints, i);
> > >                 }
> > >                 else {
> > >                     obj = PySequence_GetItem(pyPoints, i);
> > >                 }
> > >
> > > .
> > > .
> > > .
> > >
> > > So you can see that if a NumPy array is passed in, PySequence_GetItem
> > > will be used.
> > >
> > > What I would like to do is have an isNumPyArray check, and then access
> > > the NumPy array directly in that case.
> > >
> > > The tricky part is that Robin does not want to have wxPython require
> > > Numeric. (Oh how I dream of the day that NumArray becomes part of the
> > > standard library!)
> > > How can I check if an Object is a NumPy array (and then use it as such),
> > > without including Numeric during compilation?
> > >
> > > I know one option is to have condition compilation, with a NumPy and
> > > non-Numpy version, but Robin is managing a whole lot of different
> > > version as it is, and I don't think he wants to deal with twice as many!
> > >
> > > Anyone have any ideas?
> > >
> > > By the way, you can substitute NumArray for NumPy in this, as it is the
> > > wave of the future, and particularly if it would be easier.
> > >
> > > -Chris
> > >
> > >
> > > --
> > > Christopher Barker, Ph.D.
> > > Oceanographer
> > >
> > > NOAA/OR&R/HAZMAT         (206) 526-6959   voice
> > > 7600 Sand Point Way NE   (206) 526-6329   fax
> > > Seattle, WA  98115       (206) 526-6317   main reception
> > >
> > > Chr...@no...
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.NET email is sponsored by: A Thawte Code Signing Certificate
> > > is essential in establishing user confidence by providing assurance of
> > > authenticity and code integrity. Download our Free Code Signing guide:
> > > http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0028en
> > > _______________________________________________
> > > Numpy-discussion mailing list
> > > Num...@li...
> > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 
> -- 
> Christopher Barker, Ph.D.
> Oceanographer
>                                     		
> NOAA/OR&R/HAZMAT         (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chr...@no...
> 


-------------------------------------------------------------
This message was sent using HTTPS service from CNRS Grenoble.
         --->   https://grenoble.cnrs.fr   <---         

Re: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Chris B. <Chr...@no...>]

Francesc Alted wrote:

> that having a small core with the "glue"
> functionality between numarray objects and 3rd party extensions in C (or
> SWIG, Pyrex or whatever) can be a good thing (until numarray is in the
> Standard Library).
> 
> That way, people interested in supporting numarray objects in their
> extensions has only to install this small core (or even include it as part
> of the extension).

I think that's a fabulous idea, but I have no idea how hard it would be.
There would still be the problem of keeping versions in-sync. If I
distributed my package with the glue code, it would only work on
installations using the same version of Numeric (or NumArray, I suppose)


Thanks to all who have commented on my post. These are some ideas I now
have based on your comments:

> > Use the Python C-API and string literals as the basis for the
> > interface. I think the steps are something like this:
> >
> > 1.  Import "Numeric". (PyImport_ImportModule)
> >
> > 2.  Get the module dictionary.    (PyModule_GetDict)
> >
> > 3.  Get "array" out of the dictionary.   (PyDict_GetItemString)
> >
> > 4.  Call "isinstance" on Numeric.array and the object.
> > (PyObject_IsInstance)

OK, so now I can know, at runtime, whether Numeric has been imported.

> But...  It's not clear to me that knowing an object is an array will
> help since getting data elements still has to be done fast,  and that
> seems hard to do without knowing the arrayobject struct.

Exactly. that's my whole problem. However, I have an idea about this. If
I do the above test, I can now put all the Numeric specific code into a
conditional, so it would only get called in Numeric were imported. My
idea is that I could make sure Numeric was around at compile time, so I
could use all the Numeric API to access the array data, but it wouldn't
have to be installed at runtime, as none of the Numeric calls would be
executed if Numeric hadn't been imported. Would this work, or would the
system try to load the .dll or .so or whatever even if the calls weren't
executed?

All that being said, Tim Hochberg has mentioned that when he first made
wxPython DCs work with Numeric Arrays,( sorry I didn't give him credit
before, I had forgotten who did that, thanks Tim ) he did some timing
and discovered that the the overhead of the drawing calls was
substantially larger than the overhead of the indexing anyway, so
speedin up that process couldn't make much difference. 

My timing indicated something different, but I'm using Linux/wxGTK/X11,
and I think the drawing calls return after the message has been sent to
X, but X may not have completed the actual drawing yet. This means that
I'm not timing the whole process, and if I did, I might not see such a
difference. I did some tests with 100,000 points, and found that I could
see the difference with a List and Array, and the List was about twice
as fast. Drawing rectangles, however, I can't see the difference.

So, I think I'll probably shelve this for the moment, and concentrate on
getting all the drawing shapes supported by DrawXXXList methods.

Thanks for all your input.

-Chris











-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Francesc A. <fa...@op...>]

A Dimecres 15 Gener 2003 21:54, Todd Miller va escriure:
> >I think you can use the numarray High-Level C API to overcome these
> >dificulties.
>
> But doesn't using the numarray  C-API require a level of coupling
> (direct knowledge of numarray during compilation) that Chris is trying
> to avoid?
>

Ooops!, you are right.

Perhaps this kind of scenario (accessing Numeric and numarray arrays from=
 C)
would be more and more common as people is getting more aware of the
numarray capabilities and want to integrate it in their extensions. That
reinforces me in the belief that having a small core with the "glue"
functionality between numarray objects and 3rd party extensions in C (or
SWIG, Pyrex or whatever) can be a good thing (until numarray is in the
Standard Library).

That way, people interested in supporting numarray objects in their
extensions has only to install this small core (or even include it as par=
t
of the extension).

Well, speaking as non-interested and impartial person ;-)

--=20
Francesc Alted

Re: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>A Dimecres 15 Gener 2003 21:16, Todd Miller va escriure:
>  
>
>>But...  It's not clear to me that knowing an object is an array will
>>help since getting data elements still has to be done fast,  and that
>>seems hard to do without knowing the arrayobject struct.   Keep in mind
>>that Numeric and numarray arrays are strided and possibly discontiguous,
>> so there's more to data access than owning a base pointer, as would be
>>the case in C.
>>    
>>
>
>I think you can use the numarray High-Level C API to overcome these
>dificulties. 
>
<snip>

But doesn't using the numarray  C-API require a level of coupling 
(direct knowledge of numarray during compilation) that Chris is trying 
to avoid?

>
>  
>

Todd

Re: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Francesc A. <fa...@op...>]

A Dimecres 15 Gener 2003 21:16, Todd Miller va escriure:
>
> My idea to couple these was "not good".  They're not compatible at that
> level anyway.
>
> Since numarray and Numeric are only source level compatible,  C-code ca=
n
> be compiled to work with one or the other,  but not both at the same
> time.  It probably makes more sense to just implement for Numeric.  If
> you do want to implement for both,  treat them as seperate cases with
> seperate recognizer functions and element access code.
>
> But...  It's not clear to me that knowing an object is an array will
> help since getting data elements still has to be done fast,  and that
> seems hard to do without knowing the arrayobject struct.   Keep in mind
> that Numeric and numarray arrays are strided and possibly discontiguous=
,
>  so there's more to data access than owning a base pointer, as would be
> the case in C.

I think you can use the numarray High-Level C API to overcome these
dificulties. For example, by using the calls:

PyArrayObject* NA InputArray(PyObject *numarray, NumarrayType t, int requ=
ires)
PyArrayObject* NA OutputArray(PyObject *numarray, NumarrayType t, int=20
requires)
PyArrayObject* NA IoArray(PyObject *numarray, NumarrayType t, int require=
s)

as documented in the User's Guide, you can get well-behaved (i.e.
contiguous and well-aligned) C arrays (copying them, if needed) from both
numarray or Numeric arrays if you pass C_ARRAY as the value for requires
parameter.

In fact, I'm using the InputArray in PyTables to manage both numarray and
Numeric arrays with good results.

--=20
Francesc Alted

Re: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Todd M. <jm...@st...>]

Todd Miller wrote:

> Chris Barker wrote:
>
>> How can I check if an Object is a NumPy array (and then use it as such),
>> without including Numeric during compilation?
>>
>> I know one option is to have condition compilation, with a NumPy and
>> non-Numpy version, but Robin is managing a whole lot of different
>> version as it is, and I don't think he wants to deal with twice as many!
>>
>> Anyone have any ideas?
>>
> Use the Python C-API and string literals as the basis for the 
> interface. I think the steps are something like this:
>
> 1.  Import "Numeric". (PyImport_ImportModule)
>
> 2.  Get the module dictionary.    (PyModule_GetDict)
>
> 3.  Get "array" out of the dictionary.   (PyDict_GetItemString)
>
> 4.  Call "isinstance" on Numeric.array and the object.   
> (PyObject_IsInstance)
>
> Similarly:
>
> 1. Import "numarray".
>
> 2. Get the module dictionary.
>
> 3. Get "NumArray" out of the dictionary
>
> 4. Call the C-API equivalent of "isinstance" on numarray.NumArray and 
> the object.
>
> The first 3 steps of both cases can be initialized once, I think, and 
> stored in C static variables to avoid repeated fetches. 

On second thought,  just do two functions,  one for Numeric,  one for 
numarray.  

If any of the first 3 steps fail, return False.  Otherwise, return the 
result of the isinstance call.

>
> If it's not a Numeric array,  check to see if it's a numarray. 

My idea to couple these was "not good".  They're not compatible at that 
level anyway.

Since numarray and Numeric are only source level compatible,  C-code can 
be compiled to work with one or the other,  but not both at the same 
time.  It probably makes more sense to just implement for Numeric.  If 
you do want to implement for both,  treat them as seperate cases with 
seperate recognizer functions and element access code.

But...  It's not clear to me that knowing an object is an array will 
help since getting data elements still has to be done fast,  and that 
seems hard to do without knowing the arrayobject struct.   Keep in mind 
that Numeric and numarray arrays are strided and possibly discontiguous, 
 so there's more to data access than owning a base pointer, as would be 
the case in C.

Todd

Re: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Chris B. <Chr...@no...>]

Paul F Dubois wrote:
> 
> If you could do:
> try:
>     import Numeric
>     haveNumeric = 1
> except:
>     haveNumeric = 0
> 
> in some initialization routine, then you could use this flag.
> Alternately you could test on the fly
> 'Numeric' in [m.__name__ for m in sys.modules]

Thanks, but I'm talking about doing this at the C++ level in an
extension package, not at the Python level. This kind of thing is Soo
much easier in Python, of course!

-Chris





-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Todd M. <jm...@st...>]

Chris Barker wrote:

>Hi folks,
>
>I use Numeric an wxPython together a lot (of course I do, I use Numeric
>for everything!).
>
>Unfortunately, since wxPython is not Numeric aware, you lose some real
>potential performance advantages. For example, I'm now working on
>expanding the extensions to graphics device contexts (DCs) so that you
>can draw a whole bunch of objects with a single Python call. The idea is
>that the looping can be done in C++, rather than Python, saving a lot of
>overhead of the loop itself, as well as the Python-wxWindows translation
>step.
>
>For drawing thousands of points, the speed-up is substantial. It's less
>substantial on more complex objects (rectangles give a factor of two
>improvement for ~1000 objects), due to the longer time it takes to draw
>the object itself, rather than make the call. 
>
>Anyway, at the moment, Robin Dunn has the wrappers set up so that you
>can pass in a NumPy array (or, indeed, and sequence) rather than a list
>or tuple of coordinates, but it is faster to use a list than a NumPy
>array, because for arrays, it uses the generic PySequence_GetItem call.
>If we used the NumPy API directly, it should be faster than using a
>list, not slower! THis is how a representative section of the code looks
>now:
>
>
>bool      isFastSeq  = PyList_Check(pyPoints) ||
>PyTuple_Check(pyPoints);
>.
>.
>.
>                // Get the point coordinants
>                if (isFastSeq) {
>                    obj = PySequence_Fast_GET_ITEM(pyPoints, i);
>                }
>                else {
>                    obj = PySequence_GetItem(pyPoints, i);
>                }
>
>.
>.
>.
>
>So you can see that if a NumPy array is passed in, PySequence_GetItem
>will be used.
>
>What I would like to do is have an isNumPyArray check, and then access
>the NumPy array directly in that case.
>
>The tricky part is that Robin does not want to have wxPython require
>Numeric. (Oh how I dream of the day that NumArray becomes part of the
>standard library!)
>How can I check if an Object is a NumPy array (and then use it as such),
>without including Numeric during compilation?
>
>I know one option is to have condition compilation, with a NumPy and
>non-Numpy version, but Robin is managing a whole lot of different
>version as it is, and I don't think he wants to deal with twice as many!
>
>Anyone have any ideas?
>
Use the Python C-API and string literals as the basis for the interface. 
 I think the steps are something like this:

1.  Import "Numeric". (PyImport_ImportModule)

2.  Get the module dictionary.    (PyModule_GetDict)

3.  Get "array" out of the dictionary.   (PyDict_GetItemString)

4.  Call "isinstance" on Numeric.array and the object.   
(PyObject_IsInstance)

Similarly:

1. Import "numarray".

2. Get the module dictionary.

3. Get "NumArray" out of the dictionary

4. Call the C-API equivalent of "isinstance" on numarray.NumArray and 
the object.

The first 3 steps of both cases can be initialized once, I think, and 
stored in C static variables to avoid repeated fetches.
If any of the first 3 steps fail, then consider that case failed and 
returning False.
If it's not a Numeric array,  check to see if it's a numarray.

>
>By the way, you can substitute NumArray for NumPy in this, as it is the
>wave of the future, and particularly if it would be easier.
>
>-Chris
>  
>
Todd

RE: [Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Paul F D. <pa...@pf...>]

If you could do:
try:
    import Numeric
    haveNumeric = 1
except:
    haveNumeric = 0

in some initialization routine, then you could use this flag.
Alternately you could test on the fly
'Numeric' in [m.__name__ for m in sys.modules]

> -----Original Message-----
> From: num...@li... 
> [mailto:num...@li...] On 
> Behalf Of Chris Barker
> Sent: Wednesday, January 15, 2003 9:22 AM
> Cc: Numpy-discussion
> Subject: [Numpy-discussion] Optionally using Numeric in 
> another compiled extension package.
> 
> 
> Hi folks,
> 
> I use Numeric an wxPython together a lot (of course I do, I 
> use Numeric for everything!).
> 
> Unfortunately, since wxPython is not Numeric aware, you lose 
> some real potential performance advantages. For example, I'm 
> now working on expanding the extensions to graphics device 
> contexts (DCs) so that you can draw a whole bunch of objects 
> with a single Python call. The idea is that the looping can 
> be done in C++, rather than Python, saving a lot of overhead 
> of the loop itself, as well as the Python-wxWindows translation step.
> 
> For drawing thousands of points, the speed-up is substantial. 
> It's less substantial on more complex objects (rectangles 
> give a factor of two improvement for ~1000 objects), due to 
> the longer time it takes to draw the object itself, rather 
> than make the call. 
> 
> Anyway, at the moment, Robin Dunn has the wrappers set up so 
> that you can pass in a NumPy array (or, indeed, and sequence) 
> rather than a list or tuple of coordinates, but it is faster 
> to use a list than a NumPy array, because for arrays, it uses 
> the generic PySequence_GetItem call. If we used the NumPy API 
> directly, it should be faster than using a list, not slower! 
> THis is how a representative section of the code looks
> now:
> 
> 
> bool      isFastSeq  = PyList_Check(pyPoints) ||
> PyTuple_Check(pyPoints);
> .
> .
> .
>                 // Get the point coordinants
>                 if (isFastSeq) {
>                     obj = PySequence_Fast_GET_ITEM(pyPoints, i);
>                 }
>                 else {
>                     obj = PySequence_GetItem(pyPoints, i);
>                 }
> 
> .
> .
> .
> 
> So you can see that if a NumPy array is passed in, 
> PySequence_GetItem will be used.
> 
> What I would like to do is have an isNumPyArray check, and 
> then access the NumPy array directly in that case.
> 
> The tricky part is that Robin does not want to have wxPython 
> require Numeric. (Oh how I dream of the day that NumArray 
> becomes part of the standard library!) How can I check if an 
> Object is a NumPy array (and then use it as such), without 
> including Numeric during compilation?
> 
> I know one option is to have condition compilation, with a 
> NumPy and non-Numpy version, but Robin is managing a whole 
> lot of different version as it is, and I don't think he wants 
> to deal with twice as many!
> 
> Anyone have any ideas?
> 
> By the way, you can substitute NumArray for NumPy in this, as 
> it is the wave of the future, and particularly if it would be easier.
> 
> -Chris
> 
> 
> -- 
> Christopher Barker, Ph.D.
> Oceanographer
>                                     		
> NOAA/OR&R/HAZMAT         (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chr...@no...
> 
> 
> -------------------------------------------------------
> This SF.NET email is sponsored by: A Thawte Code Signing Certificate 
> is essential in establishing user confidence by providing 
> assurance of 
> authenticity and code integrity. Download our Free Code 
> Signing guide: 
> http://ads.sourceforge.net/cgi-> bin/redirect.pl?thaw0028en
> 
> 
> _______________________________________________
> Numpy-discussion mailing list Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 

[Numpy-discussion] Optionally using Numeric in another compiled extension package.

[Chris B. <Chr...@no...>]

Hi folks,

I use Numeric an wxPython together a lot (of course I do, I use Numeric
for everything!).

Unfortunately, since wxPython is not Numeric aware, you lose some real
potential performance advantages. For example, I'm now working on
expanding the extensions to graphics device contexts (DCs) so that you
can draw a whole bunch of objects with a single Python call. The idea is
that the looping can be done in C++, rather than Python, saving a lot of
overhead of the loop itself, as well as the Python-wxWindows translation
step.

For drawing thousands of points, the speed-up is substantial. It's less
substantial on more complex objects (rectangles give a factor of two
improvement for ~1000 objects), due to the longer time it takes to draw
the object itself, rather than make the call. 

Anyway, at the moment, Robin Dunn has the wrappers set up so that you
can pass in a NumPy array (or, indeed, and sequence) rather than a list
or tuple of coordinates, but it is faster to use a list than a NumPy
array, because for arrays, it uses the generic PySequence_GetItem call.
If we used the NumPy API directly, it should be faster than using a
list, not slower! THis is how a representative section of the code looks
now:


bool      isFastSeq  = PyList_Check(pyPoints) ||
PyTuple_Check(pyPoints);
.
.
.
                // Get the point coordinants
                if (isFastSeq) {
                    obj = PySequence_Fast_GET_ITEM(pyPoints, i);
                }
                else {
                    obj = PySequence_GetItem(pyPoints, i);
                }

.
.
.

So you can see that if a NumPy array is passed in, PySequence_GetItem
will be used.

What I would like to do is have an isNumPyArray check, and then access
the NumPy array directly in that case.

The tricky part is that Robin does not want to have wxPython require
Numeric. (Oh how I dream of the day that NumArray becomes part of the
standard library!)
How can I check if an Object is a NumPy array (and then use it as such),
without including Numeric during compilation?

I know one option is to have condition compilation, with a NumPy and
non-Numpy version, but Robin is managing a whole lot of different
version as it is, and I don't think he wants to deal with twice as many!

Anyone have any ideas?

By the way, you can substitute NumArray for NumPy in this, as it is the
wave of the future, and particularly if it would be easier.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Sparse Arrays in NumPy?

[Travis O. <oli...@ee...>]

> Hello all,
>
> I have been trying to find a package/addon that will provide a sparse array
> class to NumPy, or will at least trick NumPy to use a sparse array as a
> regular array, to no avail.
>

Sparse arrays are not a common object.  Sparse matrices have many, many
implementations of which I'm sure you're aware.

What you want is a general purpose N-D array that uses some kind of sparse
storage.  I'm not aware of such an object in any other language.  Most of
the time people remap their particular problem so that any sparse arrays
become sparse matrices.  All of the effort is then focused in manipulating
certain classes of sparse matrices.

-Travis

[Numpy-discussion] questions about array types

[eric h. <eha...@co...>]

Rereading the numeric docs I see the reference to types Float, Float32, Float64 -- which make sense, however I am curious to understand the usefulness of types Float0, Float8 and Float16 which all seem synonyms for Float32.  Was there some thinking that there would be a converter written for 8bit floats?



>>> from Numeric import *
>>> a = array([1,2,3,4],Float32)
>>> fromstring(a.tostring(),Float32)
array([ 1.,  2.,  3.,  4.],'f')
>>> fromstring(a.tostring(),Float)
array([   2.00000047,  512.00012255])  # corrupt, as would be expected
>>> fromstring(a.tostring(),Float0) #seems to convert back as if Float0 == Float32
array([ 1.,  2.,  3.,  4.],'f')
>>> fromstring(a.tostring(),Float8)
array([ 1.,  2.,  3.,  4.],'f')
>>> fromstring(a.tostring(),Float16)
array([ 1.,  2.,  3.,  4.],'f')
>>>

[Numpy-discussion] Sparse Arrays in NumPy?

[Costas M. <co...@ma...>]

Hello all,

I have been trying to find a package/addon that will provide a sparse array
class to NumPy, or will at least trick NumPy to use a sparse array as a
regular array, to no avail.

By sparse array here, I donot mean a sparse matrix equation solver, but an
array class that accepts a "default value".  In other words, I would like to
instantiate a 1000x1000x1000 (1e9) array that will have at most 5-10%
populated (i.e. non-zero) elements.  The current NumPy will instantiate the
entire 1e9 array, which is a non-starter if you would like to calculate an
expression with say 4-5 arrays.  Instead, I'd like a class that will only
store the populated cells, and return the default value for the others
(ideally, but doing some smart disk I/O to preserve memory).

I've tried SciPy, Scientific Python, and a few other modules floating
around; none seem to do the trick, yet I can't help but wonder that this is
not un uncommon setup for a lot of problem domains.  Is there a package out
there?  If there isn't, where should I start looking to create one? From
their description I think SparseLib++ at least would be a good starting
point as a base library.

As a secondary issue, is anyone aware of a package that can handle storage
of such arrays?  netCDF and HDF do not seem to fit the bill; a B-Tree
library seems a more natural fit...

Thanks in advance --any and all input appreciated,

Costas

RE: [Numpy-discussion] Some datatypes missing in numarray recarray?

[Perry G. <pe...@st...>]

> Hi,
>
> I think there are some data types missing in the recarray module. I can
> create recarrays using the fromarrays function with no problems
> except if I
> use UInt16, UInt32 and UInt64.
>
> As these types are well supported by numarray, is there any
> reason why they
> don't appear on numfmt and revfmt mappings in recarray module?. Is it safe
> to add them by hand in the source?
>
> Thanks,
>
> --
> Francesc Alted
>
Good point. We were using this for an I/O library that didn't use
these types so that's why they didn't get in there originally.
But you are right, they should be. Do you want to make the changes?

Thanks, PErry

[Numpy-discussion] Some datatypes missing in numarray recarray?

[Francesc A. <fa...@op...>]

Hi,

I think there are some data types missing in the recarray module. I can
create recarrays using the fromarrays function with no problems except if=
 I
use UInt16, UInt32 and UInt64.

As these types are well supported by numarray, is there any reason why th=
ey
don't appear on numfmt and revfmt mappings in recarray module?. Is it saf=
e
to add them by hand in the source?

Thanks,

--=20
Francesc Alted

[Numpy-discussion] some recarray rework

[Francesc A. <fa...@op...>]

Hi,

In the context of optimizing the PyTables support for numarray and recarray
objects I have been playing with recarray module, and ended with a
somewhat improved version of it. Roughly, the modifications done are:

- Addition of a cache to quickly access the columns (numarrays) in
  recarrays. This object is a map (dictionary) where keys are the name
  fields and values are the pointers to columns regarded as numarrays
  entities. This dictionary is accessible through the new attribute
  "_fields".

- Addition of an attribute for recarray objects named "_record" which
  points to a special object ("Record2" class) and that it is aware of
  the "_fields" cache. It that can be used to access the different
  rows in recarray objects in an efficient way.

- The "_record" object is callable (it defines the "__call__" method)
  so as to select the recarray row that is active during access to the
  different fields.

Advantages

- Access to rows and columns (fields) in recarray objects are one
  order of magnitude faster (!).

- The new "_fields" and "_record" attributes provides convenient and
  intuitive ways to access the information in recarrays.

- The "_record" attribute suports the "__getattr__" and "__setattr__"
  methods that are very convenient to access fields in a row.

Drawbacks

- "_record" attribute points always to the same object and you must
  pass it the row over which you want to operate. So, if you want to
  have two different objects pointing to different rows, you can't use
  the "_record" attribute to get them (but you can still use the
  existing Record class through by calling the "__getitem__" method
  of a recarray object).

- Two new attributes are added to the already large number of recarray
  variables. However, this new variables has no special space
  requirements as "_record" object has only three scalar variables
  and "_fields" is a dictionary with many entries as fields in
  recarray, which should be not a large amount.

I'm attaching this modified version as well as a testbed program in order to
test their new access methods and improved performance. The output of this
program ran in a pentium4@2GHz machine is also included.

Feel free to play with it and/or take/adapt the parts you consider better
suited to recarray module.

-- 
Francesc Alted                            PGP KeyID:      0x61C8C11F

Re: [Numpy-discussion] package vs module

[Tim H. <tim...@ie...>]

Pearu Peterson wrote:

>On Mon, 6 Jan 2003, Perry Greenfield wrote:
>
>  
>
>>The main disadvantages I see so far are:
>>
>>1) One will either have to change import statements in old code
>>   to match the new style (a pain, but generally changing imports
>>   is not terribly difficult since they are easy to identify) or
>>   explicitly add the path to each 3rd party module to Python
>>   Path (or some equivalent).
>>2) If numarray were accepted into the Python Standard Library, it
>>   would be the first case (as far as I can tell) of a standard
>>   library package where we would expect to add sub modules to
>>   it (e.g., FFT)). Normally these would not be distributed with
>>   the standard library, so some general mechanism will be needed
>>   to allow numarray to find 3rd party packages outside of the
>>   Python directory structure. For example, I don't think we can
>>   require having people install FFT in the Standard Library 
>>   directory structure after Python is installed. Rather, we would
>>   probably have numarray look for extension modules in a standard
>>   named site-packages directory (or site-numarray?) or otherwise
>>   check a numarraypath environmental variable so that
>>   import numarray.FFT works properly. Perhaps others have ideas
>>   about how to best handle this.
>>
>>Any other issues being overlooked?
>>    
>>
>
>There is one, though not so critical at this point but I will raise
>it anyway. In summary, I am +1 for making numarray a package.
>
>The issue is releated to import time and memory usage: more extension
>modules in a package increase both of them, even if users have no
>indention to use these modules. On slower machines this may cause
>inconvinieces, especially in applications that call Python multiple times
>for short tasks containing numarray operation.
>  
>
That's not right, is it? I'm pretty certain that submodules in a package 
are not loaded until explicitly imported. I'm not sure why SciPy is 
slow, maybe the __init__ imports everything? I don't have a copy here so 
I can't check right now.

In any event I'm +1 for putting it in a package unless it interferes 
with it getting into the core. As Paul mentioned keeping it in a zip 
archive would be even cooler once that's an option.

-tim

Re: [Numpy-discussion] package vs module

[<pa...@pf...>]

1. I favor the package approach.

2. I don't care if FFT is numarray.FFT or numpy.FFT (i.e., in a separate
place). However, see (3).

3. Extensions built with one version of Python/numarray may not work with=

a different version. This means the safer approach is to have all addons
inside the same directory, so that you can blow away just one directory
and be sure that no 'old' packages remain. 

Some new stuff being put into Python also envisions being able to add var=
ious
zipped files to the Python path as places to be searched. Perhaps this re=
presents
a packaging opportunity. I haven't paid enough attention to be sure.

While we are on the subject of packaging, the current distribution places=

all sorts of extraneous test and installation-related files in the Lib di=
rectory.
This makes it harder to work with the source when you are new to it.

Re: [Numpy-discussion] package vs module

[Konrad H. <hi...@cn...>]

Perry Greenfield <pe...@st...> writes:

> Back in December the issue of whether numarray should be a package
> potentially reduces name collision issues in general. Most feel
> it is a cleaner way to organize the software (at least based on
> the feedback so far).

I agree. We have discussed converting NumPy into a package a few times
in the past, the major argument against it was compatibility issues.
Numarray will require some changes to import statements anyway, so
this seems the right time to make the change.

> 2) If numarray were accepted into the Python Standard Library, it
>    would be the first case (as far as I can tell) of a standard
>    library package where we would expect to add sub modules to
>    it (e.g., FFT)). Normally these would not be distributed with
>    the standard library, so some general mechanism will be needed
>    to allow numarray to find 3rd party packages outside of the
>    Python directory structure. For example, I don't think we can

If you plan to unbundle FFT etc. from numarray, then I would prefer a
different naming scheme: numarray being just numarray, and some other
package name grouping together the other modules. That is not only a
question of installation, but also of general maintenance and of
clarity for users. I see the Python package system as a tree:
everything inside a package belongs together, is distributed together
and is maintained by the same people.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] package vs module

[Francesc A. <fa...@op...>]

On Mon, Jan 06, 2003 at 07:29:15PM -0500, Perry Greenfield wrote:
> The main disadvantages I see so far are:
> 
> 1) One will either have to change import statements in old code
>    to match the new style (a pain, but generally changing imports
>    is not terribly difficult since they are easy to identify) or
>    explicitly add the path to each 3rd party module to Python
>    Path (or some equivalent).

I think this should be regarded as a minor annoyance compared with the
advantages of making numarray a package. In addition, the introduction of
numarray as substitute of Numeric can justify some re-code on existing
applications.

> 2) If numarray were accepted into the Python Standard Library, it
>    would be the first case (as far as I can tell) of a standard
>    library package where we would expect to add sub modules to
>    it (e.g., FFT)). Normally these would not be distributed with
>    the standard library, so some general mechanism will be needed
>    to allow numarray to find 3rd party packages outside of the
>    Python directory structure. For example, I don't think we can
>    require having people install FFT in the Standard Library 
>    directory structure after Python is installed. Rather, we would
>    probably have numarray look for extension modules in a standard
>    named site-packages directory (or site-numarray?) or otherwise
>    check a numarraypath environmental variable so that
>    import numarray.FFT works properly. Perhaps others have ideas
>    about how to best handle this.
> 

Great. I would be glad to see a package containing numarray kernel in order
to allow aplications to use their core features, and have a mechanism to add
3rd party packages. In particular, having something similar to site-numarray
to install these packages can be quite neat. In fact, I was pondering to
include a subset of numarray in the PyTables package (it only needs the
numarray core functionality), but if this reorganization takes place, I
would not need to do that anymore.

> Any other issues being overlooked?

Yeah. In case you decide to break numarray in several modules, which would
be the granularity of the separation. My opinion goes to have a reduced core
with basic functionality (to maximize the chances to be included in the
Pyhton Standard Library, but also to allow an easy entry for people who may
wish to use this functionality) and then different, small, 3rd party
packages, but perhaps this is also the most laborious solution.

-- 
Francesc Alted                            PGP KeyID:      0x61C8C11F

Re: [Numpy-discussion] package vs module

[Pearu P. <pe...@ce...>]

On Mon, 6 Jan 2003, Perry Greenfield wrote:

> The main disadvantages I see so far are:
> 
> 1) One will either have to change import statements in old code
>    to match the new style (a pain, but generally changing imports
>    is not terribly difficult since they are easy to identify) or
>    explicitly add the path to each 3rd party module to Python
>    Path (or some equivalent).
> 2) If numarray were accepted into the Python Standard Library, it
>    would be the first case (as far as I can tell) of a standard
>    library package where we would expect to add sub modules to
>    it (e.g., FFT)). Normally these would not be distributed with
>    the standard library, so some general mechanism will be needed
>    to allow numarray to find 3rd party packages outside of the
>    Python directory structure. For example, I don't think we can
>    require having people install FFT in the Standard Library 
>    directory structure after Python is installed. Rather, we would
>    probably have numarray look for extension modules in a standard
>    named site-packages directory (or site-numarray?) or otherwise
>    check a numarraypath environmental variable so that
>    import numarray.FFT works properly. Perhaps others have ideas
>    about how to best handle this.
> 
> Any other issues being overlooked?

There is one, though not so critical at this point but I will raise
it anyway. In summary, I am +1 for making numarray a package.

The issue is releated to import time and memory usage: more extension
modules in a package increase both of them, even if users have no
indention to use these modules. On slower machines this may cause
inconvinieces, especially in applications that call Python multiple times
for short tasks containing numarray operation.

Let me repeat, currently this is not a problem neither with Numeric
(because it never imports its extension modules) or numarray until
numarray will contain a number of extension modules that
presumably are not small.

For a realistic example of this issue consider Scipy (as a sort of upper
bound what numarray may become one day). Scipy contains a linalg module
that is an (almost complete) wrapper to ATLAS/BLAS/LAPACK libraries and
therefore importing the corresponding extension modules can be both time
and memory consuming.  For example, importing scipy to Python may take 2-5
seconds on PII 400MHz, mainly because of loading the linalg extension
modules. This time may be annoying for small but frequent tasks.

I wish Python import mechanism would be a bit smarter or lazier in
loading extension modules that are never used...

Pearu

Re: [Numpy-discussion] package vs module

[Magnus L. H. <ma...@he...>]

Perry Greenfield <pe...@st...>:
>
> Back in December the issue of whether numarray should be a package
> or set of modules came up. When I asked about the possibility
> of making numarray a package (on the scipy mailing list but I
> can't seem to find the thread where it was discussed), I got
> only positive comments. The issue needs to be raised here also.
> 
> Is there any objection to making numarray package based?

I think this seems like a very good and natural thing to do. (Maybe
names like RandomArray2 etc. can be changed too, now... :)

-- 
Magnus Lie Hetland
http://hetland.org

[Numpy-discussion] package vs module

[Perry G. <pe...@st...>]

Back in December the issue of whether numarray should be a package
or set of modules came up. When I asked about the possibility
of making numarray a package (on the scipy mailing list but I
can't seem to find the thread where it was discussed), I got
only positive comments. The issue needs to be raised here also.

Is there any objection to making numarray package based?
The implications are that 3rd party modules (e.g. FFT)
will be imported as part of the package structure, i.e.,

  import numarray.FFT 

or

  from numarray.FFT import *

instead of 

  import FFT

As usual there are advantages and disadvantages. The advantages
are that we will not have name collisions with existing Numeric
modules (currently we name FFT as FFT2 for this reason). It also
potentially reduces name collision issues in general. Most feel
it is a cleaner way to organize the software (at least based on
the feedback so far).

The main disadvantages I see so far are:

1) One will either have to change import statements in old code
   to match the new style (a pain, but generally changing imports
   is not terribly difficult since they are easy to identify) or
   explicitly add the path to each 3rd party module to Python
   Path (or some equivalent).
2) If numarray were accepted into the Python Standard Library, it
   would be the first case (as far as I can tell) of a standard
   library package where we would expect to add sub modules to
   it (e.g., FFT)). Normally these would not be distributed with
   the standard library, so some general mechanism will be needed
   to allow numarray to find 3rd party packages outside of the
   Python directory structure. For example, I don't think we can
   require having people install FFT in the Standard Library 
   directory structure after Python is installed. Rather, we would
   probably have numarray look for extension modules in a standard
   named site-packages directory (or site-numarray?) or otherwise
   check a numarraypath environmental variable so that
   import numarray.FFT works properly. Perhaps others have ideas
   about how to best handle this.

Any other issues being overlooked?

Feedback?

Thanks, Perry

Re: [Numpy-discussion] Grepping the source

[Magnus L. H. <ma...@he...>]

Edward C. Jones <edc...@er...>:
[snip]
>     lines = f.read().splitlines()

You could use f.readlines() here... Or you could just use

  for line in open(...):

later, if you're using Python 2.2+

-- 
Magnus Lie Hetland
http://hetland.org

[Numpy-discussion] Grepping the source

[Edward C. J. <edc...@er...>]

Here is a short program I find useful.

#! /usr/bin/env python

import os, sys, tempfile

"""Greps the numarray source code"""

command = \
"""grep -n "%s" \
   /usr/local/src/numarray-0.4/Include/numarray/arrayobject.h \
    ...
   /usr/local/src/numarray-0.4/Lib/_ufunc.py \
   ...
   /usr/local/src/numarray-0.4/Src/libnumarraymodule.c \
 > %s
"""

if len(sys.argv) != 2:
     raise Exception, 'program requires exactly one argument'

temp = tempfile.mktemp()
try:
     os.system(command % (sys.argv[1], temp))
     f = file(temp, 'r')
     lines = f.read().splitlines()
     f.close()
finally:
     if os.path.exists(temp):
         os.remove(temp)

common = len('/usr/local/src/numarray-0.4/')
d = {}
names = []
for line in lines:
     line = line[common:]
     colonloc = line.index(':')
     name = line[:colonloc]
     text = line[colonloc+1:]
     if not d.has_key(name):
         d[name] = []
         names.append(name)
     d[name].append(text)

for name in names:
     if len(d[name]) == 0:
         continue
     print '%s:' % name
     for text in d[name]:
         print '   %s' % text
     print