numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] transpose and putmask questions.

[Chris B. <chr...@ho...>]

"Paul F. Dubois" wrote:

>> 1) why does the array passed as the first argument to putmask have to be
>> contiguous? this can be a substantial limitation, I end up making copies
>> when I have no other reason to.
> -----> because nobody has implemented the general case; feel free to
> volunteer.

Fair enough. I have yet to open up the NumPy code myself, but I have
written some custom extensions with it, and have a number of small
improvements in mind that I would like to make. Hopefully I will get to
this some day.

> b) No one has gotten around to making the code smart enough to do this
> when it is unambigous.?
> ------> Numpy's attitude toward extra dimensions of length 1 is somewhat
> schizophrenic.
>         If you know you want one-d you can put a ravel() around it.
> 
>         In general the answer to your question is (b) but it is very
> complicated so that is why people don't work on it.

I suspected as much. I'm starting to get used to it. I think my problem
is that I come from MATLAB, where everything is a 2-d array.

Thanks for your comments.

-Chris


-- 
Christopher Barker,
Ph.D.                                                           
Chr...@ho...                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------

[Numpy-discussion] transpose and putmask questions.

[Chris B. <chr...@ho...>]

Hi all,

I recently tried to take a slice out of an array, transpose it, and then
change it with the putmask function.

It didn't work, because it ends up not being contiguous. This brings up
two points:

1) why does the array passed as the first argument to putmask have to be
contiguous? this can be a substantial limitation, I end up making copies
when I have no other reason to.

2) the docs say that transpose "returns a new array..." This is
confusing. I had assumed that "new array" meant a copy, when, in fact it
is another array, but the data is a reference, much like a slice. We
might want to make it a little more clear in the docs what that means.

Another question I have is about how to match array shapes. I find I am
often getting error when I try to do something (like putmask) with two
arrays, one of shape (n,1) and one of shape (n,). It seems to me that
what I want to do is unambiguous, so I should be able to do it. Is the
reason I can't do this:

a) The NumPy interface is designed to be unambiguous, and so it is
important that the code never silently translate a rank one array into a
rank two array where one of the dimensions is one (and vice-versa)

or

b) No one has gotten around to making the code smart enough to do this
when it is unambigous.?


-Thanks,

-Chris



-- 
Christopher Barker,
Ph.D.                                                           
Chr...@ho...                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------

[Numpy-discussion] creation of array of objects

[kk <kir...@al...>]

hi all..
   I want to create an array of class objects of size say mXn. What 
would be the easiest way to do this?
eg.
class test:
   pass
i want 5x10 array of test() instances.
Then only way i know of doing this ,is by creating
a=array(test())
resize(a,[5,10])
but this creates only references to the initial objects.Is there some 
way of creating fresh objects ? Also, if the __init__() call of the 
class takes some parameters, i should be able to enter values..

thanx in anticipation
ciao

Re: [Numpy-discussion] Thoughts on masked arrays

[Michael H. <mh...@al...>]

"Paul F. Dubois" <pa...@pf...> writes:
> The method __array__(self, typecode=None) is a special (existing)
> hook for conversion to a Numeric array. Many operations in Numeric,
> when presented with an object x to be operated upon, such as
> Numeric.sqrt(x), will call x.__array__ as a final act of desperation
> in an attempt to convert their argument to a Numeric
> array. Heretofore it was essentially returning x.filled(). This
> bothered me, because it was a silent conversion that replaced masked
> values with the fill value.

This bothered me too.

> Solution:
> 
> a. Add a method 'unmask()' which will replace the mask by None if
> possible.  It will not fail.
> 
> b. Change MaskedArray.__array__ to work as follows:
>    a. self.unmask(), and then
>    b. Return the raw data if the mask is now None.
>       Otherwise, throw an MAError.

Perfect.  Great solution!

Michael

-- 
Michael Haggerty
mh...@al...

[Numpy-discussion] Re: New version of compress for you

[Michael H. <mh...@al...>]

Paul F. Dubois writes:
> def compress (condition, x, dimension=-1):  
>     """Select those parts of x for which condition is true.
>        Masked values in condition are considered false.
>     """
>     c = filled(condition, 0)
>     m = getmask(x)
>     if m is not None:
>         m=Numeric.compress(c, m, dimension)
>     d = Numeric.compress(c, filled(x), dimension)
>     return masked_array(d, m)
> 
> 
> I did want a treatment of masked conditions. Consider:
> 
> compress( x > 10, x)

I see your point, but this doesn't generalize to x having more than
one dimension.  And with my semantics, for your example (assuming
1-d), you could substitute

    compress((x > 10).filled(0), x)

which isn't very obscure.  Moreover, sometimes it is interesting to
compress an array in such a way that masked values are carried along
as masked values; this cannot be done with the existing compress().

Michael

--
Michael Haggerty
mh...@al...

Re: [Numpy-discussion] Numpy bug?

[Karl B. <Kar...@um...>]

How can we correct this error?

Otherwise, Numpy can only be initialized once per application/thread?

Here is what I do, a user loads an image into my application, then runs
a Python script. I do a Py_Initialize(), initialize Numpy, initialize my
module, run the script and finally Py_Finalize(). Each image can do
this. My module only knows about its own image and can run a python
script multiple times. 

So, if I have to make it more global, I would have to run
Py_Intialize/Py_Finalize once per application. Redesign my module so
that a user must pick the specific image to work on. Redesign the
scripts so that a user must transverse all the images to the correct
image. Hey, I would rather fix numpy so it can be initialized multiple
times. Do you remember why the person wanted to do that? Or, if you let
me know where you did that, I can fix it in my local source instead.


"Paul F. Dubois" wrote:
> 
> The import_array is not proper, but I have no idea if that is the problem or
> not.
> import array only goes in the initialization routine of modules that need
> the Numpy C-API.
> 
> Somebody convinced me to decref something in one of the numpy C module
> initializers a while back, so maybe that was an error. (?)
> I just do what they tell me...
> 
> -----Original Message-----
> From: num...@li...
> [mailto:num...@li...]On Behalf Of Karl
> Bellve
> Sent: Thursday, May 10, 2001 5:57 AM
> To: Numpy
> Subject: [Numpy-discussion] Numpy bug?
> 
> Since I didn't get a response, I will post again with a more appropiate
> subject heading. If you run the following code, you get an error. It
> appears that Numpy doesn't like to be restarted. If you take out the
> lines "import_array();" and "PyRun_SimpleString("from Numeric import
> *\n");" the code loops fine.
> 
> for (int x = 0; x < 100; x++)
> {
>    Py_Initialize();
>    import_array();
>    PyRun_SimpleString("from Numeric import *\n");
>    TRACE("%d\n",x);
>    Py_Finalize();
> }
> 
> Here is the output:
> 0
> 1
> Fatal Python error: UNREF invalid object
> 
> --
> Cheers,
> 
> Karl Bellve
> 

-- 
Cheers,



Karl Bellve

RE: [Numpy-discussion] Thoughts on masked arrays

[Paul F. D. <pa...@pf...>]

Thank you for provoking me to think about these issues in MA. Here is the
conclusion I have reached. Please let me know what you think of it.

Background:

Michael wanted a way to use a masked array as a Numeric array but with
assurance that in fact no element was masked,
without obscure tests such as count(x) == product(x.shape).

The method __array__(self, typecode=None) is a special (existing) hook for
conversion to a Numeric array. Many operations in Numeric, when presented
with an object x to be operated upon, such as Numeric.sqrt(x), will call
x.__array__ as a final act of desperation in an attempt to convert their
argument to a Numeric array. Heretofore it was essentially returning
x.filled(). This bothered me, because it was a silent conversion that
replaced masked values with the fill value.

Solution:

a. Add a method 'unmask()' which will replace the mask by None if possible.
It will not fail.

b. Change MaskedArray.__array__ to work as follows:
   a. self.unmask(), and then
   b. Return the raw data if the mask is now None.
      Otherwise, throw an MAError.

Example usage:
>>> from MA import *
>>> x=arange(10)
>>> Numeric.array(x)
[0,1,2,3,4,5,6,7,8,9,]
>>> x[3]=masked
>>> Numeric.array(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/pcmdi/dubois/linux/lib/python2.1/site-packages/MA/MA.py", line 578,
in __array__
    raise MAError, \
MA.MA.MAError: Cannot convert masked array to Numeric because data
               is masked in one or more locations.

Merits of this solution:
   a. It reads like what it is -- there is no doubt you are converting to a
Numeric array when you see Numeric.array.

   b. It gives you the full range of options in Numeric.array, such as
messing with the typecode.

   c. It allows Numeric operations for speed on masked arrays that you know
to be masked in name only. No copy of data occurs here unless the typecode
needs to be changed.

   d. It removes the possibility of a 'dishonest' conversion.

   e. No new method or function is required, other than the otherwise-useful
unmask().

   f. Successive conversions are optimized because once the mask is None,
unmask is cheap.

Deficiency: __array__ becomes a query with an internal, albeit safe,
side-effect. Mitigating this is that __array__ is not a "public" method and
would not normally be used in assertions.

Re: [Numpy-discussion] Thoughts on masked arrays

[Michael H. <mh...@al...>]

"Paul F. Dubois" <pa...@pf...> writes:

> -----Original Message-----
> Michael Haggerty wrote
> 1. I couldn't find a simple way to tell if all of the cells of a
>    masked array are unmasked.
> ======
> So your test could be if count(x) < product(x.shape): error...
> 
> So your test could be
>    if make_mask(m.mask(),flag=1) is not None:
>        error...
> 
> You could also consider if not Numeric.allclose(m.filled(0), m.filled(1))
> or
> m.mask() is not None and not Numeric.alltrue(Numeric.ravel(m.mask())):

Shouldn't that be

    m.mask() is not None and Numeric.sometrue(Numeric.ravel(m.mask()))

?  ("Proof" that these expressions are nonintuitive.)

> Is that enough ways to do it? (TM) (:->

Frankly, it's too many ways to do it, none of them obvious to the
writer or the reader.  This is a simple and useful concept and it
should have one obvious implementation.

> I'm not against is_unmasked but I'm not sure how much it would get
> used and I don't like the name.  I hate query methods with side
> effects (if you use them in an assert you change the program).

In this case the side effect is to change the internal representation
of the object without changing its semantics, so I don't find it too
objectionable.  But omit this optimization if you prefer; the query
method would be just as useful even without the side effect.

Because of the relationship with filled(), maybe this query function
should be called m.isfull().  There should probably also be an
isfull(m) function for the same reason that there is a mask(m)
function.

> A method that replaces the mask with None if possible might make
> sense.  m.unmask()? m.demask()? m.debride() ?

Of these names, I like m.unmask() the best.  I assume that it would
set m.__mask=None if possible and throw an exception if not.

On the other hand, while it would be desirable to have a function
equivalent (i.e., unmask(m)), this would be awkward because a function
should usually not change its argument.

Therefore, I suggest adding a safe analogue of raw_data() that throws
an exception if the array has a nontrivial mask and otherwise returns
self.__data.  E.g. [untested]:

class MaskedArray:
    [...]
    def data(self):
        """If no values are masked, return self.__data().  Otherwise
           raise an exception.
        """
        d = self.__data
        m = self.__mask
        if m is not None and Numeric.sometrue(Numeric.ravel(m)):
	    raise MAError, "MaskedArray cannot be converted to array"
	elif d.iscontiguous():
	    return d
        else:
	    return Numeric.array(d, typecode=d.typecode(), copy=1,
                                 savespace = d.spacesaver())


def data(a):
    if isinstance(a, MaskedArray):
        return m.data()
    elif isinstance(a, Numeric.ArrayType) and a.iscontiguous():
        return a
    else:
        return Numeric.array(a)

A more obscure name should be chosen since you seem to encourage "from
MA import *".

> 3. I found the semantics of MA.compress(condition,a,axis=0) to be
>    inconvenient and inconsistent with those of Numeric.compress.
> ======
> It has been an interesting project in that there are hundreds of these
> individual little design questions.
> Can you propose the semantics you would like in a precise way? Include the
> case where the condition has masked values.
> ======

In the simple case where the condition has no masked values, I think
compress() should simply pick slices out according to condition,
without regard to which cells of x are masked.  When condition is
masked, I don't think that there is a sensible interpretation for
compress() because a "masked" value in condition means you don't know
whether that slice of x should be included or not.  Since you can't
have an output array of indeterminate shape, I would throw an
exception if condition is masked.  Here is my attempt [untested]:

def compress(condition, x, dimension=-1):
    # data function is defined above (throws exception if condition is masked):
    c = data(condition)
    if mask(x) is None:
        mask = None
    else:
        mask=Numeric.compress(condition, mask(x), dimension)
    return array(Numeric.compress(condition, filled(x), dimension), mask=mask)


Yours,
Michael

-- 
Michael Haggerty
mh...@al...

[Numpy-discussion] 20.1.0a3 -- for those who like pickles

[Paul F. D. <pa...@pf...>]

I put a source tarball up for 20.1.0a3, also in CVS as of now. This
contains:

DL_EXPORT changes

Konrad's fix to import_array

A new MA that pickles

A new package "Properties", that replaces MA.activeattr. Properties allows
you to have properties such as MA/Numeric's shape attribute, that look like
attributes but actually invoke functions on get, set, and/or delete. It can
also be used to make attributes unwriteable or undeleteable. It doesn't have
anything to do with Numeric but I use it in MA and kinds and don't have much
else of an idea where to put it.

RE: [Numpy-discussion] Numpy bug?

[Paul F. D. <pa...@pf...>]

The import_array is not proper, but I have no idea if that is the problem or
not.
import array only goes in the initialization routine of modules that need
the Numpy C-API.

Somebody convinced me to decref something in one of the numpy C module
initializers a while back, so maybe that was an error. (?)
I just do what they tell me...

-----Original Message-----
From: num...@li...
[mailto:num...@li...]On Behalf Of Karl
Bellve
Sent: Thursday, May 10, 2001 5:57 AM
To: Numpy
Subject: [Numpy-discussion] Numpy bug?




Since I didn't get a response, I will post again with a more appropiate
subject heading. If you run the following code, you get an error. It
appears that Numpy doesn't like to be restarted. If you take out the
lines "import_array();" and "PyRun_SimpleString("from Numeric import
*\n");" the code loops fine.



for (int x = 0; x < 100; x++)
{
   Py_Initialize();
   import_array();
   PyRun_SimpleString("from Numeric import *\n");
   TRACE("%d\n",x);
   Py_Finalize();
}


Here is the output:
0
1
Fatal Python error: UNREF invalid object


--
Cheers,



Karl Bellve

_______________________________________________
Numpy-discussion mailing list
Num...@li...
http://lists.sourceforge.net/lists/listinfo/numpy-discussion

[Numpy-discussion] Numpy bug?

[Karl B. <Kar...@um...>]

Since I didn't get a response, I will post again with a more appropiate
subject heading. If you run the following code, you get an error. It
appears that Numpy doesn't like to be restarted. If you take out the
lines "import_array();" and "PyRun_SimpleString("from Numeric import
*\n");" the code loops fine.



for (int x = 0; x < 100; x++)
{
   Py_Initialize();
   import_array();
   PyRun_SimpleString("from Numeric import *\n");
   TRACE("%d\n",x);
   Py_Finalize();
}


Here is the output:
0
1
Fatal Python error: UNREF invalid object


-- 
Cheers,



Karl Bellve

RE: [Numpy-discussion] Modified header files

[Paul F. D. <pa...@pf...>]

I have added these files to CVS. Thanks!
--Paul


-----Original Message-----
From: num...@li...
[mailto:num...@li...]On Behalf Of Konrad
Hinsen
Sent: Wednesday, May 09, 2001 9:12 AM
To: num...@li...
Subject: [Numpy-discussion] Modified header files


Recently we had a discussion about how to use NumPy arrays from
extension modules with multiple source files, on various platforms.
The modified header files that are attached to this message provide a
(hopefully!) general solution. In fact, I propose to make them part of
the official distribution, unless there are objections.

If used like before, these header files give exactly the same result
as the ones in NumPy 20.0.0. However, they permit to define the name
of the C API pointer array and make it globally visible. Under the
condition that the chosen name is unique, this should not create
problems under any platform, no matter if static or dynamic linking is
used.

To use NumPy features in multiple source file extension modules, you have
to write

   #define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX
   #include "Numeric/arrayobject.h"

in the main source file (the one that contains the init function) and

   #define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX
   #define NO_IMPORT_ARRAY
   #include "Numeric/arrayobject.h"

in all the others. The symbol you choose instead of PyArrayXXX should
contain both the name of the imported module (array) and the name of
the importing module (whatever your module is called) in order to be
unique with a reasonably high probability.

The same applies to the Ufunc module, just replace "array" by
"ufunc" in the example. I have also applied the "static" correction
to the Ufunc header file, there is no reason not to do it.

Konrad.
--
----------------------------------------------------------------------------
---
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
----------------------------------------------------------------------------
---

RE: [Numpy-discussion] Thoughts on masked arrays

[Paul F. D. <pa...@pf...>]

-----Original Message-----
From: num...@li...
[mailto:num...@li...]On Behalf Of
Michael Haggerty wrote
1. I couldn't find a simple way to tell if all of the cells of a
   masked array are unmasked.  There are times when you fill an array
   incrementally and you want to convert it to a Numeric array but
   first make sure that all of the elements have been set.
   "m.filled()" is a bit dangerous (in my opinion) because it silently
   fills.  The shortest idiom I could think of is

    >>> assert not logical_or.reduce(ravel(MA.getmaskarray(m)))

   which isn't very short :-) and is also awkward because it creates a
   mask array even if m.mask() is None.  How about a m.is_unmasked()
   method, or even giving a special meaning to "m.filled(masked)",
   namely that it raises an exception if any cells are still masked.
   (As an optimization, this method could set m.__mask = None to speed
   up future checks.)
======
>>> from MA import *
>>> x=array([[1,2],[3,4]],mask=[[0,0],[0,0]])
>>> count(x)
4
>>> product(x.shape)
4
>>>
So your test could be if count(x) < product(x.shape): error...

make_mask(m, flag=1) will make a mask and have it be None if possible. It
also accepts an argument of None correctly.
So your test could be
   if make_mask(m.mask(),flag=1) is not None:
       error...

You could also consider if not Numeric.allclose(m.filled(0), m.filled(1))
or
m.mask() is not None and not Numeric.alltrue(Numeric.ravel(m.mask())):

Is that enough ways to do it? (TM) (:->

I don't recommend using assert if the test is data-driven, since it won't
get executed with python -O. Instead use if...: raise ....

I'm not against is_unmasked but I'm not sure how much it would get used and
I don't like the name. I hate query methods with side effects (if you use
them in an assert you change the program).

A method that replaces the mask with None if possible might make sense.
m.unmask()? m.demask()? m.debride() ?


=============

2. I can't reproduce this problem now, but I could swear that the
   MaskedArray.__str__() method sometimes printed "typecode='O'" if
   masked.enabled() is true.  This would be a byproduct of using
   Numeric's __str__() method to print the array, at least under the
   unknown circumstances in which Numeric.__str__() prints the
   typecode.  This confused me for a while.

=========
Short of writing my own print routine, I basically have to create something
filled with '--', which is of course of type object. That's why you can
disable it. (:->
=========

3. I found the semantics of MA.compress(condition,a,axis=0) to be
   inconvenient and inconsistent with those of Numeric.compress.
   MA.compress() squeezes out not only those elements for which
   condition is false, but also those elements that are masked.  This
   differs from the behavior of Numeric.compress, which always returns
   an array with the "axis" dimension equal to the number of nonzero
   elements of "condition".  The real problem, though, is that
   MA.compress can't be used on a multidimensional array with a
   nontrivial mask, because squeezing out the masked values is highly
   unlikely to result in a rectangular matrix.  It is nice to be able
   to squeeze masked values out of a 1-d array, but not at the price
   of not being able to use compress on a multidimensional array.  I
   suggest giving MA.compress() semantics closer to those of
   Numeric.compress(), and adding an optional argument or a separate
   method to cause masked elements to be omitted.
======
It has been an interesting project in that there are hundreds of these
individual little design questions.
Can you propose the semantics you would like in a precise way? Include the
case where the condition has masked values.
======

Thanks for a great package!

Yours,
Michael

===
I appreciate the encouragement. -- Paul

[Numpy-discussion] Thoughts on masked arrays

[Michael H. <mh...@al...>]

Hi,

I've spent several days using the masked arrays that have been added
to NumPy recently.  They're a great feature and they were just what I
needed for the little project I was working on (aside from a few bugs
that I found).

However, there were a few things about MA that I found inconvenient
and/or counterintuitive, so I thought I'd post them to the list while
they're fresh in my mind.  I'm using Numeric-20.0.0b2.

1. I couldn't find a simple way to tell if all of the cells of a
   masked array are unmasked.  There are times when you fill an array
   incrementally and you want to convert it to a Numeric array but
   first make sure that all of the elements have been set.
   "m.filled()" is a bit dangerous (in my opinion) because it silently
   fills.  The shortest idiom I could think of is

    >>> assert not logical_or.reduce(ravel(MA.getmaskarray(m)))

   which isn't very short :-) and is also awkward because it creates a
   mask array even if m.mask() is None.  How about a m.is_unmasked()
   method, or even giving a special meaning to "m.filled(masked)",
   namely that it raises an exception if any cells are still masked.
   (As an optimization, this method could set m.__mask = None to speed
   up future checks.)

2. I can't reproduce this problem now, but I could swear that the
   MaskedArray.__str__() method sometimes printed "typecode='O'" if
   masked.enabled() is true.  This would be a byproduct of using
   Numeric's __str__() method to print the array, at least under the
   unknown circumstances in which Numeric.__str__() prints the
   typecode.  This confused me for a while.

3. I found the semantics of MA.compress(condition,a,axis=0) to be
   inconvenient and inconsistent with those of Numeric.compress.
   MA.compress() squeezes out not only those elements for which
   condition is false, but also those elements that are masked.  This
   differs from the behavior of Numeric.compress, which always returns
   an array with the "axis" dimension equal to the number of nonzero
   elements of "condition".  The real problem, though, is that
   MA.compress can't be used on a multidimensional array with a
   nontrivial mask, because squeezing out the masked values is highly
   unlikely to result in a rectangular matrix.  It is nice to be able
   to squeeze masked values out of a 1-d array, but not at the price
   of not being able to use compress on a multidimensional array.  I
   suggest giving MA.compress() semantics closer to those of
   Numeric.compress(), and adding an optional argument or a separate
   method to cause masked elements to be omitted.

Thanks for a great package!

Yours,
Michael   

--
Michael Haggerty
mh...@al...

Re: [Numpy-discussion] UNREF

[Karl B. <Kar...@um...>]

More info:

I can reproduce this error with the following loop:

for (int x = 0; x < 100; x++)
{
   Py_Initialize();
   import_array();
   PyRun_SimpleString("from Numeric import *\n");
   TRACE("%d\n",x);
   Py_Finalize();
}


Here is the output:
0
1
Fatal Python error: UNREF invalid object


-- 
Cheers,



Karl Bellve

[Numpy-discussion] Modified header files

[Konrad H. <hi...@cn...>]

Recently we had a discussion about how to use NumPy arrays from
extension modules with multiple source files, on various platforms.
The modified header files that are attached to this message provide a
(hopefully!) general solution. In fact, I propose to make them part of
the official distribution, unless there are objections.

If used like before, these header files give exactly the same result
as the ones in NumPy 20.0.0. However, they permit to define the name
of the C API pointer array and make it globally visible. Under the
condition that the chosen name is unique, this should not create
problems under any platform, no matter if static or dynamic linking is
used.

To use NumPy features in multiple source file extension modules, you have
to write

   #define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX
   #include "Numeric/arrayobject.h"

in the main source file (the one that contains the init function) and

   #define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX
   #define NO_IMPORT_ARRAY
   #include "Numeric/arrayobject.h"

in all the others. The symbol you choose instead of PyArrayXXX should
contain both the name of the imported module (array) and the name of
the importing module (whatever your module is called) in order to be
unique with a reasonably high probability.

The same applies to the Ufunc module, just replace "array" by
"ufunc" in the example. I have also applied the "static" correction
to the Ufunc header file, there is no reason not to do it.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

[Numpy-discussion] UNREF

[Karl B. <Kar...@um...>]

Occasionally, if I reinitialize Python and rerun a script, I get the
following error:

FATAL Python error: UNREF invalid object

from my code I do the following:
	/* alot of other crap and then */
	PyRun_SimpleString("From Numeric import *\n");

eventually in multiarraymodule.c
	array_set_string_function()
		Py_INCREF(Py_None)

which eventually calls (according to my stack trace) where the error
occurs 
	PyArray_SetStringFunction()
 		Py_XDECREF(PyArray_ReprFunction)



Is this a Numpy problem, Python problem, or my problem?

I can run the same python script several times, and then I get the
error. Something isn't being cleaned up properly around Py_Initialize()
Py_Finalize() pair.

I am using Python 2.1b1 and Numeric 18_1 with Boost



-- 
Cheers,



Karl Bellve

Re: [Numpy-discussion] Can this be done efficiently with array functions?

[Chris B. <chr...@ho...>]

Janko and Paul,

Thanks for your suggestions. I had come up with something like Janko's
solution, but not as cleanly. It seems that Paul's suggestion is
essentially the same thing. Maybe I'll do some tests to see which is
faster.

By the way, I tried Janko's code, and got an error on "a == 0" (I'm
running Python 2.0). I looked at it again, and then looked at the
release notes for Python 2.1, and saw that rich comparisons are
working..Glory be! I'll be upgrading real soon, just for that!

thanks again,
-Chris

PS: anyone have a response for my Int32 question? Why isn't it a "long"
on Linux or windows on Intel?


Janko Hauser wrote:
> truely_max=100
> 
> a=array([[10, 0, 5, 0,],
>          [ 0, 0, 0, 0,],
>          [ 0, 5,15, 0,],
>          [ 0, 0, 0, 0,],
>          [ 0, 3, 1,25,],
>          [ 0, 0, 0, 0,],
>          [ 4, 7, 2,12,],
>          [ 0, 0, 0, 0,]])
> 
> am=minimum.reduce(where(a==0,truely_max,a),1)
> am=where(am==truely_max,0,am)
> print am
> [5 0 5 0 1 0 2 0]

> "Paul F. Dubois" wrote:

> but there doesn't seem to be
> a minimum.reduce() in MA.
> 
> --> I'll add this to my list.

That will be great if you get a chance.

>     Meanwhile you could do something like this,
>     where x is your array:

>     y = MA.masked_equal(x,0,copy=0)
>     Numeric.minimum.reduce(y.filled(100000), 1)
>     If x is floating point use masked_value rather than masked_equal.
> ---

> --> There are two problems. There is a hit, even in the case of no mask.
>     It is hard to quantify because it depends on what operations you do.
>     Much of that could be eliminated if MA was in C but it is Python.
>     It would have driven me mad to write it in C, however.

WOW, I'm impressed, I had assumed that some C was involved. Nice job.
 
>     More seriously, there are a few semantic issues where it isn't clear
>     what you mean if a mask is present.

I'm sure these could be resolved. It would be nice if we could have in
mind to move in that direction. I'm not going ot be able to do any of
the coding, so that's the last you'll here from me about it.

-Chris






-- 
Christopher Barker,
Ph.D.                                                           
Chr...@ho...                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------

RE: [Numpy-discussion] Can this be done efficiently with array functions?

[Paul F. D. <pa...@pf...>]

-----Original Message-----
...snip

what I want is minimum.reduce(a,1), except that the zeros would be
ignored. At first glace, Masked arrays seemed the perfect option (as
that's what the zero means, no real data), but there doesn't seem to be
a minimum.reduce() in MA.

--> I'll add this to my list. Meanwhile you could do something like this,
    where x is your array:
    y = MA.masked_equal(x,0,copy=0)
    Numeric.minimum.reduce(y.filled(100000), 1)
    If x is floating point use masked_value rather than masked_equal.
---
By the way, what is the downside to Masked Arrays? they seem to be a
really powerful option. Is there any performance hit to using a masked
array over a regular onw if there is no mask? If not, would it make
sense to move toward just using Masked arrays for everything?

thanks,

-Chris

--> There are two problems. There is a hit, even in the case of no mask.
    It is hard to quantify because it depends on what operations you do.
    Much of that could be eliminated if MA was in C but it is Python.
    It would have driven me mad to write it in C, however.

    More seriously, there are a few semantic issues where it isn't clear
    what you mean if a mask is present. 

[Numpy-discussion] Can this be done efficiently with array functions?

[Chris B. <chr...@ho...>]

I have a problem that I can't figure otu a non-kludgey way to solve:

Given a (MXN) array, a, I want to compute the minimum value in each row,
excluding the values that are zero. example:

for this array, a:

[[10, 0, 5, 0,]
 [ 0, 0, 0, 0,]
 [ 0, 5,15, 0,]
 [ 0, 0, 0, 0,]
 [ 0, 3, 1,25,]
 [ 0, 0, 0, 0,]
 [ 4, 7, 2,12,]
 [ 0, 0, 0, 0,]]

I want:

[[5,]
 [0,]
 [5,]
 [0,]
 [1,]
 [0,]
 [2,]
 [0,]]

what I want is minimum.reduce(a,1), except that the zeros would be
ignored. At first glace, Masked arrays seemed the perfect option (as
that's what the zero means, no real data), but there doesn't seem to be
a minimum.reduce() in MA.

By the way, what is the downside to Masked Arrays? they seem to be a
really powerful option. Is there any performance hit to using a masked
array over a regular onw if there is no mask? If not, would it make
sense to move toward just using Masked arrays for everything?

thanks,

-Chris





-- 
Christopher Barker,
Ph.D.                                                           
Chr...@ho...                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------

[Numpy-discussion] Just what is an "Int32" ?

[Chris B. <chr...@ho...>]

Hi all,

I'm writing an extension that uses a NumPy array (among other things).
To make sure that I have passed in what I'm looking for, I do a:

if (Array->descr->type_num != PyArray_LONG) ...
	PyErr_SetString (PyExc_ValueError,"Array is the wrong type");
	return NULL;
  }
(I want Array to be an array of longs)

When I call my extension with an array built using array([...],Int32),
this error results (this is on both Windows and Linux on Intel
hardware). If I build the array with "Int" it works fine. This isn't
really a problem, but I am curious, what is an Int32 on Intel hardware?,
and why wouldn't it be a long?

-Chris





-- 
Christopher Barker,
Ph.D.                                                           
Chr...@ho...                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------

Re: [Numpy-discussion] static pointer

[Konrad H. <hi...@cn...>]

> > That seems to me the preferred solution.
> 
> But this will still fail on OSX, I guess.

Not necessarily. The API pointer just has to be passed between the
source code modules in a global variable whose name is highly unlikely
to be used somewhere else.

> That sounds very good. I don't mind providing a symbol name for the pointer 
> myself.

Good to hear. Is there anyone who would oppose this?

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] static pointer

[Peter V. <Pet...@em...>]

On Wednesday 02 May 2001 19:39, Konrad Hinsen wrote:
> > If I understand correctly, the addition of the static keyword means that
> > the import_array() function can now only be used by extensions that are
> > defined in a single C source file. I can easily work around this problem,
> > by
>
> True. That's exactly what I pointed out in a previous message in this
> list.
>
> > defining my own api pointer, and using a modified import_array() in my
> > code,
>
> That seems to me the preferred solution.

But this will still fail on OSX, I guess.

> > built ok, and we were formerly not following the instructions in the
> > Python documentation and now we are. And, OS-X now worked. So far so
> > good. If I
>
> And those instructions mention the static keyword for a good reason:
> without it there are problems not only with portability, but also with
> static linking of extension modules.
>
> > Is there a way of institutionalizing your work-around so that others can
> > use it?
>
> I thought about providing a general solution right in the header
> files. The problem is the generation of a unique symbol by
> preprocessor tricks, perhaps somehow based on the module name.
> I don't see how this can be done.
>
> On the other hand, if we accept that the extension module programmer
> has to supply a unique symbol, then it is even easy. This would only
> be necessary for multi-file extension modules. Does that seem like a
> good solution?

That sounds very good. I don't mind providing a symbol name for the pointer 
myself.

> > Wait. I have an idea. Konrad was the one who put all this in. He should
> > be punished for his good deed...(:->
>
> I am even thinking of punishing myself by imposing extra work on me! ;-)
>
> Konrad.

-- 
Dr. Peter J. Verveer
Bastiaens Group
Cell Biology and Cell Biophysics Programme
EMBL
Meyerhofstrasse 1
D-69117 Heidelberg
Germany
Tel. : +49 6221 387245
Fax  : +49 6221 387242
Email: Pet...@em...

[Numpy-discussion] MA pickle

[Paul F. D. <du...@ll...>]

I have a version of MA that pickles. I am so frightened of it that I don'=
t even
want to chuck it into CVS yet .  I just wanted those who expressed concer=
n about
this to know there is hope. If someone is desperate please email me and I=
'll
let you try it privately.

Re: [Numpy-discussion] static pointer

[Konrad H. <hi...@cn...>]

> If I understand correctly, the addition of the static keyword means that the
> import_array() function can now only be used by extensions that are defined
> in a single C source file. I can easily work around this problem, by

True. That's exactly what I pointed out in a previous message in this
list.

> defining my own api pointer, and using a modified import_array() in my code,

That seems to me the preferred solution.

> built ok, and we were formerly not following the instructions in the Python
> documentation and now we are. And, OS-X now worked. So far so good. If I

And those instructions mention the static keyword for a good reason:
without it there are problems not only with portability, but also with
static linking of extension modules.

> Is there a way of institutionalizing your work-around so that others can use
> it?

I thought about providing a general solution right in the header
files. The problem is the generation of a unique symbol by
preprocessor tricks, perhaps somehow based on the module name.
I don't see how this can be done.

On the other hand, if we accept that the extension module programmer
has to supply a unique symbol, then it is even easy. This would only
be necessary for multi-file extension modules. Does that seem like a
good solution?

> Wait. I have an idea. Konrad was the one who put all this in. He should be
> punished for his good deed...(:->

I am even thinking of punishing myself by imposing extra work on me! ;-)

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------