numpy-discussion

[Numpy-discussion] adding an attribute to an nd-array

[Stefan v. d. W. <st...@su...>]

A quick question on extending numpy arrays: is it possible to easily
add an attribute to an ndarray?

With Python-defined classes one can do

class X(object):
    pass

x =3D X()
x.foo =3D 'bar'

but with ndarrays you get

x =3D N.array([1,2,3])
x.foo =3D 'bar'

AttributeError: 'numpy.ndarray' object has no attribute 'foo'

Is there an easy way around this (without writing a C extension)?

Thanks for any advice.
St=E9fan

Re: [Numpy-discussion] adding an attribute to an nd-array

[Pierre GM <pgm...@gm...>]

On Wednesday 18 October 2006 20:29, Stefan van der Walt wrote:
> A quick question on extending numpy arrays: is it possible to easily
> add an attribute to an ndarray?

It might be easier to create a subclass: pleasehave a look here:
http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/test_subclasses.py
That's a tiny example of subclassing ndarrays, with some extra attributes.
(BTW, I'm not sure that's the most obvious place where to look: if it turns 
out to be useful, I'll put it on the scipy wiki)

Re: [Numpy-discussion] adding an attribute to an nd-array

[Stefan v. d. W. <st...@su...>]

On Wed, Oct 18, 2006 at 09:17:49PM -0400, Pierre GM wrote:
> On Wednesday 18 October 2006 20:29, Stefan van der Walt wrote:
> > A quick question on extending numpy arrays: is it possible to easily
> > add an attribute to an ndarray?
>=20
> It might be easier to create a subclass: pleasehave a look here:
> http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/test_=
subclasses.py
> That's a tiny example of subclassing ndarrays, with some extra attribut=
es.
> (BTW, I'm not sure that's the most obvious place where to look: if it t=
urns=20
> out to be useful, I'll put it on the scipy wiki)

Thanks very much, Pierre.

If I understand correctly, the following should work:

import numpy as N

class InfoArray(N.ndarray):
    def __new__(info_arr_cls,arr,info=3D{}):
        info_arr_cls.info =3D info
        return N.array(arr).view(info_arr_cls)

When does __array_finalize__ get called, and is it always necessary to
specify it?

Cheers
St=E9fan

Re: [Numpy-discussion] adding an attribute to an nd-array

[Travis O. <oli...@ie...>]

Stefan van der Walt wrote:
> On Wed, Oct 18, 2006 at 09:17:49PM -0400, Pierre GM wrote:
>   
>> On Wednesday 18 October 2006 20:29, Stefan van der Walt wrote:
>>     
>>> A quick question on extending numpy arrays: is it possible to easily
>>> add an attribute to an ndarray?
>>>       
>> It might be easier to create a subclass: pleasehave a look here:
>> http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/test_subclasses.py
>> That's a tiny example of subclassing ndarrays, with some extra attributes.
>> (BTW, I'm not sure that's the most obvious place where to look: if it turns 
>> out to be useful, I'll put it on the scipy wiki)
>>     
>
> Thanks very much, Pierre.
>
> If I understand correctly, the following should work:
>
> import numpy as N
>
> class InfoArray(N.ndarray):
>     def __new__(info_arr_cls,arr,info={}):
>         info_arr_cls.info = info
>         return N.array(arr).view(info_arr_cls)
>
> When does __array_finalize__ get called, and is it always necessary to
> specify it?
>   
It gets called whenever a new array is created.  No, it is not necessary 
to specify it.

-Travis

Re: [Numpy-discussion] adding an attribute to an nd-array

[Travis O. <oli...@ee...>]

Stefan van der Walt wrote:

>On Wed, Oct 18, 2006 at 09:17:49PM -0400, Pierre GM wrote:
>  
>
>>On Wednesday 18 October 2006 20:29, Stefan van der Walt wrote:
>>    
>>
>>>A quick question on extending numpy arrays: is it possible to easily
>>>add an attribute to an ndarray?
>>>      
>>>
>>It might be easier to create a subclass: pleasehave a look here:
>>http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/test_subclasses.py
>>That's a tiny example of subclassing ndarrays, with some extra attributes.
>>(BTW, I'm not sure that's the most obvious place where to look: if it turns 
>>out to be useful, I'll put it on the scipy wiki)
>>    
>>
>
>Thanks very much, Pierre.
>
>If I understand correctly, the following should work:
>
>import numpy as N
>
>class InfoArray(N.ndarray):
>    def __new__(info_arr_cls,arr,info={}):
>        info_arr_cls.info = info
>        return N.array(arr).view(info_arr_cls)
>
>When does __array_finalize__ get called, and is it always necessary to
>specify it?
>  
>

Actually something as simple as

class InfoArray(N.ndarray):
         pass

will allow you to add attributes to InfoArray.

I just learned about how to allow built-ins to have attributes assigned 
to their instances.   It's actually pretty easy because of Python 
support for it --- but it comes at a cost.  You have to add a dictionary 
to the PyArrayObject structure, create that dictionary when the ndarray 
is allocated, and set the tp_dictoffset in the TypeObject structure to 
its location in PyArrayObject.   It takes 4 lines of code with the cost of
creating a new dictionary for every ndarray.



I don't think the extra bytes for every ndarray object are worth it, 
given how easy it is to sub-class and create your own ndarray that can 
have attributes attached.    What are others opinions.

-Travis

P.S.  Here is the patch that adds it:


Index: numpy/core/include/numpy/ndarrayobject.h
===================================================================
--- numpy/core/include/numpy/ndarrayobject.h    (revision 3366)
+++ numpy/core/include/numpy/ndarrayobject.h    (working copy)
@@ -1172,6 +1172,7 @@
         PyArray_Descr *descr;   /* Pointer to type structure */
         int flags;              /* Flags describing array -- see below*/
         PyObject *weakreflist;  /* For weakreferences */
+        PyObject *instancedict; /* For instance attributes */
 } PyArrayObject;

 #define NPY_AO PyArrayObject
Index: numpy/core/src/arrayobject.c
===================================================================
--- numpy/core/src/arrayobject.c        (revision 3366)
+++ numpy/core/src/arrayobject.c        (working copy)
@@ -1906,6 +1906,8 @@
         if (self->weakreflist != NULL)
                 PyObject_ClearWeakRefs((PyObject *)self);

+        Py_DECREF(self->instancedict);
+
         if(self->base) {
                 /* UPDATEIFCOPY means that base points to an
                    array that should be updated with the contents
@@ -5305,6 +5307,7 @@
         self->descr = descr;
         self->base = (PyObject *)NULL;
         self->weakreflist = (PyObject *)NULL;
+        self->instancedict = PyDict_New();

         if (nd > 0) {
                 self->dimensions = PyDimMem_NEW(2*nd);
@@ -6689,7 +6692,7 @@
         0,                                        /* tp_dict */
         0,                                        /* tp_descr_get */
         0,                                        /* tp_descr_set */
-        0,                                        /* tp_dictoffset */
+        offsetof(PyArrayObject, instancedict),    /* tp_dictoffset */
         (initproc)0,                              /* tp_init */
         array_alloc,                              /* tp_alloc */
         (newfunc)array_new,                       /* tp_new */

Re: [Numpy-discussion] adding an attribute to an nd-array

[Charles R H. <cha...@gm...>]

On 10/19/06, Travis Oliphant <oli...@ee...> wrote:
>
> Stefan van der Walt wrote:
>
> >On Wed, Oct 18, 2006 at 09:17:49PM -0400, Pierre GM wrote:
> >
> >
> >>On Wednesday 18 October 2006 20:29, Stefan van der Walt wrote:
> >>


<snip>

I don't think the extra bytes for every ndarray object are worth it,
> given how easy it is to sub-class and create your own ndarray that can
> have attributes attached.    What are others opinions.


I'd be more inclined to worry about speed. One of the drawbacks of numarray
was the time it took to create arrays. Since I  often use many small arrays,
numpy was a big improvement in that area.

Chuck

Re: [Numpy-discussion] adding an attribute to an nd-array

[Stefan v. d. W. <st...@su...>]

On Thu, Oct 19, 2006 at 09:45:02AM -0600, Travis Oliphant wrote:
> Stefan van der Walt wrote:
>=20
> >If I understand correctly, the following should work:
> >
> >import numpy as N
> >
> >class InfoArray(N.ndarray):
> >    def __new__(info_arr_cls,arr,info=3D{}):
> >        info_arr_cls.info =3D info
> >        return N.array(arr).view(info_arr_cls)

One has to be careful of this approach.  It ads *the same* information
to all arrays, i.e.

In [2]: a =3D ImageInfo(N.array([1,2,3]),{1:1})

In [3]: b =3D ImageInfo(N.array([1,2,3]),{1:2})

In [4]: a
Out[4]: ImageInfo([1, 2, 3])

In [5]: b
Out[5]: ImageInfo([1, 2, 3])

In [6]: a.info
Out[6]: {1: 2}

In [7]: b.info
Out[7]: {1: 2}

Regards
St=E9fan

Re: [Numpy-discussion] adding an attribute to an nd-array

[Pierre GM <pgm...@gm...>]

> > >class InfoArray(N.ndarray):
> > >    def __new__(info_arr_cls,arr,info={}):
> > >        info_arr_cls.info = info
> > >        return N.array(arr).view(info_arr_cls)
>
> One has to be careful of this approach.  It ads *the same* information
> to all arrays, i.e.

Indeed. That's basically why you have to edit your __array_finalize__ .

class InfoArray(N.ndarray):
    def __new__(info_arr_cls,arr,info={}):
        info_arr_cls._info = info
        return N.array(arr).view(info_arr_cls)
    def __array_finalize__(self, obj):
        if hasattr(obj,'info'):
            self.info = obj.info
        else:
            self.info = self._info
        return

OK, so you end up w/ two attributes 'info' and '_info', the latter having the 
info you want, the latter playing a temporary placeholder. That looks a bit 
overkill, but that works pretty nice.

a = InfoArray(N.array([1,2,3]),{1:1})
b = InfoArray(N.array([1,2,3]),{1:2})

assert a.info=={1:1}
assert b.info=={1:2}
assert (a+1).info==a.info 
assert (b-2).info==b.info

Re: [Numpy-discussion] adding an attribute to an nd-array

[Stefan v. d. W. <st...@su...>]

On Thu, Oct 19, 2006 at 09:03:57PM -0400, Pierre GM wrote:
> Indeed. That's basically why you have to edit your __array_finalize__ .
>=20
> class InfoArray(N.ndarray):
>     def __new__(info_arr_cls,arr,info=3D{}):
>         info_arr_cls._info =3D info
>         return N.array(arr).view(info_arr_cls)
>     def __array_finalize__(self, obj):
>         if hasattr(obj,'info'):
>             self.info =3D obj.info
>         else:
>             self.info =3D self._info
>         return
>=20
> OK, so you end up w/ two attributes 'info' and '_info', the latter havi=
ng the=20
> info you want, the latter playing a temporary placeholder. That looks a=
 bit=20
> overkill, but that works pretty nice.

Is there any reason why one can't simply do

class InfoArray(N.ndarray):
    def __new__(info_arr_cls,arr,info=3D{}):
        x =3D N.array(arr).view(info_arr_cls)
	x.info =3D info
        return x
    def __array_finalize__(self, obj):
        if hasattr(obj,'info'):
            self.info =3D obj.info
        return

Regards
St=E9fan

Re: [Numpy-discussion] adding an attribute to an nd-array

[Albert S. <fu...@gm...>]

Hello all

> -----Original Message-----
> From: num...@li... [mailto:numpy-
> dis...@li...] On Behalf Of Travis Oliphant
> Sent: Thursday, October 19, 2006 5:45 PM
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] adding an attribute to an nd-array
> 
> Stefan van der Walt wrote:
> 
> >On Wed, Oct 18, 2006 at 09:17:49PM -0400, Pierre GM wrote:
> >>On Wednesday 18 October 2006 20:29, Stefan van der Walt wrote:
> >>>A quick question on extending numpy arrays: is it possible to easily
> >>>add an attribute to an ndarray?
> >>It might be easier to create a subclass: pleasehave a look here:
> >>http://projects.scipy.org/scipy/numpy/attachment/wiki/MaskedArray/test_s
> ubclasses.py
> >>That's a tiny example of subclassing ndarrays, with some extra
> attributes.
> >>(BTW, I'm not sure that's the most obvious place where to look: if it
> turns
> >>out to be useful, I'll put it on the scipy wiki)
> >
> >Thanks very much, Pierre.
> >
> >If I understand correctly, the following should work:
> >
> >import numpy as N
> >
> >class InfoArray(N.ndarray):
> >    def __new__(info_arr_cls,arr,info={}):
> >        info_arr_cls.info = info
> >        return N.array(arr).view(info_arr_cls)
> >
> >When does __array_finalize__ get called, and is it always necessary to
> >specify it?
> 
> Actually something as simple as
> 
> class InfoArray(N.ndarray):
>          pass
> 
> will allow you to add attributes to InfoArray.

I would like to create an "info" array like this to attach some metadata,
mostly sampling frequency, to some of my arrays.

The first problem I'm running into is that this new InfoArray type needs
something like the array function so that I can create it from an existing
"normal" array. It would also be useful if metadata could survive through
type conversions like this:

y = array(x, 'f4')

One frequently wants to do this where you start with a 8-bit or 16-bit
samples of a speech signal and you then want to convert to a floating point
type for further processing.

I browsed through the NumPy book and looked at the __new__ and
__array_finalize__ stuff in Pierre GM's new MaskedArray, but I'm still
thoroughly confused about how to create a simple array+metadata subclass
that Just Works(TM).

Does anybody have some code that they would be willing to share?

Thanks.

Regards,

Albert

Re: [Numpy-discussion] adding an attribute to an nd-array

[Pierre GM <pgm...@gm...>]

On Friday 27 October 2006 18:37, Albert Strasheim wrote:
> Hello all
...
> I would like to create an "info" array like this to attach some metadata,
> mostly sampling frequency, to some of my arrays.
>
...
> I browsed through the NumPy book and looked at the __new__ and
> __array_finalize__ stuff in Pierre GM's new MaskedArray, but I'm still
> thoroughly confused about how to create a simple array+metadata subclass
> that Just Works(TM).
>
> Does anybody have some code that they would be willing to share?

I'm not sure about the JustWorks(R)(TM), but I just created a page on the Wiki 
describing a simple example of InfoArray class
http://www.scipy.org/Subclasses
That should do what you want, but let me know if you have some more specific 
requests.
I guess I should go more into the details of subclassing on that page, but I'm 
quite far from being a specialist...

Re: [Numpy-discussion] adding an attribute to an nd-array

[Robert K. <rob...@gm...>]

Travis Oliphant wrote:
> I just learned about how to allow built-ins to have attributes assigned 
> to their instances.   It's actually pretty easy because of Python 
> support for it --- but it comes at a cost.  You have to add a dictionary 
> to the PyArrayObject structure, create that dictionary when the ndarray 
> is allocated, and set the tp_dictoffset in the TypeObject structure to 
> its location in PyArrayObject.   It takes 4 lines of code with the cost of
> creating a new dictionary for every ndarray.
> 
> I don't think the extra bytes for every ndarray object are worth it, 
> given how easy it is to sub-class and create your own ndarray that can 
> have attributes attached.    What are others opinions.

I'd say leave it off. Many uses of that feature will require custom 
__array_finalize__ methods anyways.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Re: [Numpy-discussion] adding an attribute to an nd-array

[Lisandro D. <da...@gm...>]

On 10/19/06, Robert Kern <rob...@gm...> wrote:
> Travis Oliphant wrote:
> > I don't think the extra bytes for every ndarray object are worth it,
> > given how easy it is to sub-class and create your own ndarray that can
> > have attributes attached.    What are others opinions.
>
> I'd say leave it off. Many uses of that feature will require custom
> __array_finalize__ methods anyways.
>

I second Travis and Robert.

--=20
Lisandro Dalc=EDn
---------------
Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIMEC)
Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INTEC)
Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICET)
PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

Re: [Numpy-discussion] adding an attribute to an nd-array

[Christopher B. <Chr...@no...>]

Travis Oliphant wrote:
> Actually something as simple as
> 
> class InfoArray(N.ndarray):
>          pass
> 
> will allow you to add attributes to InfoArray.

Well, sure, but how the heck do you initialize it?

 >>> class InfoArray(N.ndarray):
...          pass
...
 >>> InfoArray((1,2,3))
InfoArray([[[  6.61855173e-306,               nan,   4.94708449e+173],
         [  3.32457336e-309,   1.08825491e+175,   2.12468326e-314]]])

InfoArray(N.array((1,2,3)))
InfoArray([[[  2.17854722e-305,   1.90979621e-313,   1.90979621e-313],
         [  1.90979621e-313,   1.90979621e-313,   1.03977794e-312]]])

I, for one, would like a very easy to subclass version of ndarray, maybe 
one that had a default constructor like numpy.array().

oh, and:


Charles R Harris wrote:
> I'd be more inclined to worry about speed. One of the drawbacks of 
> numarray was the time it took to create arrays. Since I  often use many 
> small arrays, numpy was a big improvement in that area.

+1

keep basic arrays simple and lean.

Wasn't there one a UserArray class or something?


-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] adding an attribute to an nd-array

[Tim H. <tim...@ie...>]

Christopher Barker wrote:
> Travis Oliphant wrote:
>   
>> Actually something as simple as
>>
>> class InfoArray(N.ndarray):
>>          pass
>>
>> will allow you to add attributes to InfoArray.
>>     
>
> Well, sure, but how the heck do you initialize it?
>
>  >>> class InfoArray(N.ndarray):
> ...          pass
> ...
>  >>> InfoArray((1,2,3))
> InfoArray([[[  6.61855173e-306,               nan,   4.94708449e+173],
>          [  3.32457336e-309,   1.08825491e+175,   2.12468326e-314]]])
>
> InfoArray(N.array((1,2,3)))
> InfoArray([[[  2.17854722e-305,   1.90979621e-313,   1.90979621e-313],
>          [  1.90979621e-313,   1.90979621e-313,   1.03977794e-312]]])
>
> I, for one, would like a very easy to subclass version of ndarray, maybe 
> one that had a default constructor like numpy.array().
>
>   
Back when I was working on basearray, I had an idea for this. Basically, 
it was to to move the ndarray constructor[1] over to basearray and make 
the ndarray constructor work just like array does now[2]. Thus 
array(args) and ndarray(args) would behave the same[2 again]. For the 
full power/complexity of the current ndarray constructor, you would 
instead use basearray(). I believe (and I did some experiments on this 
at one point), that this would allow straightforward inheritance from 
ndarray. I've been away from this for several months now, so sadly I 
forget many of the details.

-tim




[1] I know, I know, it's really __new__, but it gets used like a 
constructor so I'll call it one for the time being.
[2] Modulo the copy arg which would disappear if I had anything to with it.

Re: [Numpy-discussion] adding an attribute to an nd-array

[Travis O. <oli...@ee...>]

Christopher Barker wrote:

>Travis Oliphant wrote:
>  
>
>>Actually something as simple as
>>
>>class InfoArray(N.ndarray):
>>         pass
>>
>>will allow you to add attributes to InfoArray.
>>    
>>
>
>Well, sure, but how the heck do you initialize it?
>  
>
You use the same constructor as ndarray has. 

numpy.info(numpy.ndarray)

If you want an array-like function that produces the array, you do

array(obj).view(InfoArray)

or wrap that up in your own function.


There are many over-lapping ways to construct ndarray's.   You can use 
all of these ways to construct instances of your own type by getting the 
ndarray and using the .view() method.


-Travis

Re: [Numpy-discussion] adding an attribute to an nd-array

[Stefan v. d. W. <st...@su...>]

On Thu, Oct 19, 2006 at 01:59:49PM -0700, Christopher Barker wrote:
> Travis Oliphant wrote:
> > Actually something as simple as
> >=20
> > class InfoArray(N.ndarray):
> >          pass
> >=20
> > will allow you to add attributes to InfoArray.
>=20
> Well, sure, but how the heck do you initialize it?

Looks like

x =3D N.array([1,2,3])
x.view(InfoArray)

works.

Thanks to everyone for the useful feedback!

Cheers
St=E9fan