numpy-discussion — Discussion list for all users of Numerical Python

[Numpy-discussion] [H O T] Enjoy the same deep discounts offered to US residents mightn't besotting cachalot custard

[Randee E. <ma...@sh...>]

We believe ordering medication should be as simple as ordering anything else on the Internet: Private, secure, and easy.

Everything is done on-line and Safe..


Click The Link Below
mediente.com

Best Regards,
Randee Erdman
mediente.com customer service

oetewlcsqv ECrHgeggQKrYdOArPhPdegiINJLCEn

Josephus PVC attained gadwall edicts inferences gadwall encase dogmas Punic amounters discord grassland fanned burlesques canons chances amounters ducked fixating determines dim Samoa dusting battleship amazingly archiving colonies expertly airs contrivances

Re: [Numpy-discussion] adaptive thresholding: get adacent cells for each pixel

[Johannes L. <a.u...@gm...>]

Hi,

> I'm just starting with numpy (via scipy) and I'm wanting to perform
> adaptive thresholding
> (http://www.cee.hw.ac.uk/hipr/html/adpthrsh.html) on an image.
> Basically that means that I need to get a threshold for each pixel by
> examining the pixels around it. In numpy this translates to finding
> the adjacent cells for each cell (not including the value of the cell
> we are examining) and getting the mean, or median of those cells.
> 
> I've written something that works, but is terribly slow. How would
> someone with more experience get the adjacent cells for each cell
> minus the cell being examined?

regarding the mean value, you can take a look at
scipy.signal.convolve2d. If you convolve with an array like this:

[[0.125 0.125 0.125]
 [0.125 0.0   0.125]
 [0.125 0.125 0.125]]

you get the 3x3 mean value (btw why leave out the center pixel?).

For the median, I can not think of any good method right now.

Also another method springs to my mind (just substract the top row and
add a new bottom row to the averaging window), but I have no idea how to
do this in an efficient way.

Generally, always try to find a way to process the whole array as one.
If you perform anything on an array elementwise, it will be dead slow.


Best regards,
Johannes

Re: [Numpy-discussion] adaptive thresholding: get adacent cells for each pixel

[Filip W. <fi...@ft...>]

Hi,

> I'm just starting with numpy (via scipy) and I'm wanting to perform
> adaptive thresholding
> (http://www.cee.hw.ac.uk/hipr/html/adpthrsh.html) on an image.
> Basically that means that I need to get a threshold for each pixel by
> examining the pixels around it. In numpy this translates to finding
> the adjacent cells for each cell (not including the value of the cell
> we are examining) and getting the mean, or median of those cells.

> I've written something that works, but is terribly slow. How would
> someone with more experience get the adjacent cells for each cell
> minus the cell being examined?

You can get the mean value of surrounding cells by filtering.

import numpy
from scipy import signal
im = numpy.ones((10,10), dtype='d') * range(10)
fi = numpy.ones((3,3), dtype='d') / 8
fi[1,1]=0
print fi
#[[ 0.125  0.125  0.125]
# [ 0.125  0.     0.125]
# [ 0.125  0.125  0.125]]

signal.convolve2d(im, fi, mode='same', boundary='symm') # or correlate2d in this case


Also check help(signal.convolve2d) for information on various
parameters this function takes.


cheers,
fw

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Tim H. <tim...@co...>]

David M. Cooke wrote:

>On Fri, 09 Jun 2006 16:03:32 -0700
>Andrew Straw <str...@as...> wrote:
>
>  
>
>>Tim Hochberg wrote:
>>
>>    
>>
>>>Which of the following should we require for an object to be "supporting 
>>>the array interface"? Here a producer is something that supplies 
>>>array_struct or array_interface (where the latter is the Python level 
>>>version of the former as per recent messages). Consumers do something 
>>>with the results.
>>>
>>>  1. Producers can supply either array_struct (if implemented in C) or
>>>     array_interface (if implemented in Python). Consumers must accept
>>>     both.
>>>  2. Producers must supply both array_struct and array_interface.
>>>     Consumers may accept either.
>>>  3. Producers most supply both array_struct and array_interface.
>>>     Consumers must accept both as well.
>>> 
>>>
>>>      
>>>
>>I haven't been following as closely as I could, but is the following a 
>>possibility?
>>    4. Producers can supply either array_struct or array_interface. 
>>Consumers may accept either. The intermediate is a small, standalone 
>>(does not depend on NumPy) extension module that does automatic 
>>translation if necessary by provides 2 functions: as_array_struct() 
>>(which returns a CObject) and as_array_interface() (which returns a 
>>tuple/dict/whatever).
>>    
>>
>
>For something to go in the Python standard library this is certainly
>possible. Heck, if it's in the standard library we can have one attribute
>which is a special ArrayInterface object, which can be queried from both
>Python and C efficiently.
>
>For something like numpy (where we don't require a special object: the
>"producer" and "consumers" in Tim's terminology could be Numeric and
>numarray, for instance), we don't want a 3rd-party dependence. There's one
>case that I mentioned in another email:
>
>5. Producers must supply array_interface, and may supply array_struct.
>Consumers can use either.
>
>Requiring array_struct means that Python-only modules can't play along, so I
>think it should be optional (of course, if you're concerned about speed, you
>would provide it).
>
>Or maybe we should revisit the "no external dependencies". Perhaps one module
>would make everything easier, with helper functions and consistent handling
>of special cases. Packages wouldn't need it if they don't interact: you could
>conditionally import it when __array_interface__ is requested, and fail if
>you don't have it. It would just be required if you want to do sharing.
>  
>
Here's another idea: move array_struct *into* array_interface. That is, 
array_interface becomes a dictionary with the following items:

    shape : sequence specifying the shape
    typestr : the typestring
    descr: you get the idea
    strides: ...
    shape: ...
    mask: ...
    offset: ...
    data: A buffer object
    struct: the array_struct or None.

The downside is that you have to do two lookups to get the array_struct, 
and that should be the fast path. A partial solution is to instead have 
array_interface be a super_tuple similar to the result of os.stat. This 
should be faster since tuple is quite fast to index if you know what 
index you want.

An advantage of having one module that you need to import is that we 
could use something other than CObject, which would allow us to bullet 
proof the array interface at the python level. One nit with using a 
CObject is that I can pass an object that doesn't refer to a 
PyArrayInterface with unpleasant results.

-tim

Re: [Numpy-discussion] Building shared libraries with numpy.distutils

[Robert K. <rob...@gm...>]

Albert Strasheim wrote:
> Hello all
> 
> For my Summer of Code project, I'm adding Support Vector Machine code to
> SciPy. Underneath, I'm currently using libsvm. Thus far, I've been compiling
> libsvm as a shared library (DLL on Windows) using SCons and doing the
> wrapping with ctypes.
> 
> Now, I would like to integrate my code into the SciPy build. Unfortunately,
> it doesn't seem as if numpy.distutils or distutils proper knows about
> building shared libraries.
> 
> Building shared libraries across multiple platforms is tricky to say the
> least so I don't know if implementing this functionality again is something
> worth doing. The alternative -- never using shared libraries, doesn't seem
> very appealing either.
> 
> Is anybody building shared libraries? Any code or comments?

Ed Schofield worked out a way:

  http://www.scipy.net/pipermail/scipy-dev/2006-April/005708.html

You'll have some experimenting to do, but the basics are there.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco

[Numpy-discussion] adaptive thresholding: get adacent cells for each pixel

[stephen e. <ste...@gm...>]

I'm just starting with numpy (via scipy) and I'm wanting to perform
adaptive thresholding
(http://www.cee.hw.ac.uk/hipr/html/adpthrsh.html) on an image.
Basically that means that I need to get a threshold for each pixel by
examining the pixels around it. In numpy this translates to finding
the adjacent cells for each cell (not including the value of the cell
we are examining) and getting the mean, or median of those cells.

I've written something that works, but is terribly slow. How would
someone with more experience get the adjacent cells for each cell
minus the cell being examined?

Thanks
Stephen Emslie

Re: [Numpy-discussion] Any Numeric or numarray users on this list?

[Travis O. <oli...@ee...>]

Thanks for your response to the questionaire.

>>3) Please, explain your reason(s) for not making the switch. (if  
>>you answered No to #2)
>>    
>>
>
>Lack of time. Some of the changes from Numeric are subtle and require  
>a careful analysis of the code, and then careful testing. For big  
>applications, that's a lot of work. There are also modules (I am  
>thinking of RNG) that have been replaced by something completely  
>different that needs to be evaluated first.
>  
>

You may be interested to note that I just added the RNG interface to 
numpy for back-wards compatibility.  
It can be accessed and used by re-placing

import RNG

with

import numpy.random.oldrng as RNG


Best regards,

-Travis

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[David M. C. <co...@ph...>]

On Fri, 09 Jun 2006 16:03:32 -0700
Andrew Straw <str...@as...> wrote:

> Tim Hochberg wrote:
> 
> >Which of the following should we require for an object to be "supporting 
> >the array interface"? Here a producer is something that supplies 
> >array_struct or array_interface (where the latter is the Python level 
> >version of the former as per recent messages). Consumers do something 
> >with the results.
> >
> >   1. Producers can supply either array_struct (if implemented in C) or
> >      array_interface (if implemented in Python). Consumers must accept
> >      both.
> >   2. Producers must supply both array_struct and array_interface.
> >      Consumers may accept either.
> >   3. Producers most supply both array_struct and array_interface.
> >      Consumers must accept both as well.
> >  
> >
> I haven't been following as closely as I could, but is the following a 
> possibility?
>     4. Producers can supply either array_struct or array_interface. 
> Consumers may accept either. The intermediate is a small, standalone 
> (does not depend on NumPy) extension module that does automatic 
> translation if necessary by provides 2 functions: as_array_struct() 
> (which returns a CObject) and as_array_interface() (which returns a 
> tuple/dict/whatever).

For something to go in the Python standard library this is certainly
possible. Heck, if it's in the standard library we can have one attribute
which is a special ArrayInterface object, which can be queried from both
Python and C efficiently.

For something like numpy (where we don't require a special object: the
"producer" and "consumers" in Tim's terminology could be Numeric and
numarray, for instance), we don't want a 3rd-party dependence. There's one
case that I mentioned in another email:

5. Producers must supply array_interface, and may supply array_struct.
Consumers can use either.

Requiring array_struct means that Python-only modules can't play along, so I
think it should be optional (of course, if you're concerned about speed, you
would provide it).

Or maybe we should revisit the "no external dependencies". Perhaps one module
would make everything easier, with helper functions and consistent handling
of special cases. Packages wouldn't need it if they don't interact: you could
conditionally import it when __array_interface__ is requested, and fail if
you don't have it. It would just be required if you want to do sharing.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|co...@ph...

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Andrew S. <str...@as...>]

Tim Hochberg wrote:

>Which of the following should we require for an object to be "supporting 
>the array interface"? Here a producer is something that supplies 
>array_struct or array_interface (where the latter is the Python level 
>version of the former as per recent messages). Consumers do something 
>with the results.
>
>   1. Producers can supply either array_struct (if implemented in C) or
>      array_interface (if implemented in Python). Consumers must accept
>      both.
>   2. Producers must supply both array_struct and array_interface.
>      Consumers may accept either.
>   3. Producers most supply both array_struct and array_interface.
>      Consumers must accept both as well.
>  
>
I haven't been following as closely as I could, but is the following a 
possibility?
    4. Producers can supply either array_struct or array_interface. 
Consumers may accept either. The intermediate is a small, standalone 
(does not depend on NumPy) extension module that does automatic 
translation if necessary by provides 2 functions: as_array_struct() 
(which returns a CObject) and as_array_interface() (which returns a 
tuple/dict/whatever).

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Tim H. <tim...@co...>]

Which of the following should we require for an object to be "supporting 
the array interface"? Here a producer is something that supplies 
array_struct or array_interface (where the latter is the Python level 
version of the former as per recent messages). Consumers do something 
with the results.

   1. Producers can supply either array_struct (if implemented in C) or
      array_interface (if implemented in Python). Consumers must accept
      both.
   2. Producers must supply both array_struct and array_interface.
      Consumers may accept either.
   3. Producers most supply both array_struct and array_interface.
      Consumers must accept both as well.

A possibly related point, array_interface['data'] should be required to 
be a buffer object; a 2-tuple of address/read-only should not be allowed 
as that's a simple way to crash the interpreter.

I see some reasonable arguments for either 1 or 2. 3 seems like excess work.

-tim

Re: [Numpy-discussion] Suggestions for NumPy

[Christopher B. <Chr...@no...>]

Andrew Straw wrote:
> Christopher Barker wrote:
>> Joe Harrington wrote:
>>> My
>>> suggestion is that all the other pages be automatic redirects to the
>>> scipy.org page or subpages thereof.

>> if that means something like:
>>
>> www.numpy.scipy.org (or www.scipy.org/numpy )
>>   Then I'm all for it.
>>   
> I just made www.scipy.org/numpy redirect to the already-existing 
> www.scipy.org/NumPy
> 
> So, hopefully you're on-board now. BTW, this is the reason why we have a 
> wiki -- if you don't like something it says, how the site is organized, 
> or whatever, please just jump in and edit it.

Thanks for that, but I wasn't taking issue with capitalization.  Now 
that you've done, though, the easier it is to find, the better. As I 
understood it, Joe's suggestion about "all other pages" referred to 
pages that are NOT hosted at scipy.org. Those I can't change.

My comment referred to an earlier suggestion that other pages about 
Numpy be referred to www.scipy.org, and I was simply suggesting that any 
non-scipy page that refers to numpy should refer to a page specifically 
about numpy, like  www.scipy.org/NumPy, rather than the main scipy page.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Getting rid of annoying weave nag

[Fernando P. <fpe...@gm...>]

On 6/9/06, David M. Cooke <co...@ph...> wrote:

> > This difference is so dramatic that I think a message is justified
> > (absent a proper logging framework).  It's helpful to know that the
> > time is going into c++ compilation, and not your code hanging for 30
> > seconds.
>
> Ok, I'll give you that one :-) It's the other 1000 uses of print that I'm
> concerned about.
>
> inline_tools.compile_function takes a verbose flag, though, which eventually
> gets passed to build_tools.build_extension (which I believe does all the
> compiling for weave). It's probably more reasonable to have
> inline_tools.compile_function default to verbose=1 instead of 0, then
> build_extension will print 'Compiling code...' (that should be changed to
> mention weave).

I failed to mention that I agree with you: the proper solution is to
use logging for this.  For now I'll commit the strict-prototypes fix,
and if I find myself with a lot of spare time, I'll try to clean
things up a little bit to use logging (there's already a logger
instance running in there).

Cheers,

f

Re: [Numpy-discussion] Getting rid of annoying weave nag

[Tim H. <tim...@co...>]

David M. Cooke wrote:

>On Fri, 9 Jun 2006 15:19:14 -0600
>"Fernando Perez" <fpe...@gm...> wrote:
>
>  
>
>>On 6/9/06, David M. Cooke <co...@ph...> wrote:
>>    
>>
>>>On Thu, Jun 08, 2006 at 11:28:04PM -0600, Fernando Perez wrote:
>>>      
>>>
>>>>Anyone object to this patch against current numpy SVN to get rid of
>>>>this thing? (tracking where the hell that thing was coming from was
>>>>all kinds of fun)
>>>>        
>>>>
>>>Go ahead.
>>>
>>>I'm against random messages being printed out anyways -- I'd get
>>>rid of the '<weave: compiling>' too. There's a bunch of code in scipy
>>>with 'print' statements that I don't think belong in a library. (Now,
>>>if we defined a logging framework, that'd be ok with me!)
>>>      
>>>
>>Before I commit anything, let's decide on that one.  Weave used to
>>print 'None' whenever it compiled anything, I changed it a while ago
>>to the current 'weave:compiling'.  I'm also of the opinion that
>>libraries should operate quietly, but with weave I've always wanted
>>that message in there.  The reason is that when weave compiles (esp.
>>with blitz in the picture), the execution takes a long time.  The same
>>function goes from miliseconds to 30 seconds of run time depending on
>>whether compilation is happening or not.
>>
>>This difference is so dramatic that I think a message is justified
>>(absent a proper logging framework).  It's helpful to know that the
>>time is going into c++ compilation, and not your code hanging for 30
>>seconds.
>>    
>>
>
>Ok, I'll give you that one :-) It's the other 1000 uses of print that I'm
>concerned about.
>
>inline_tools.compile_function takes a verbose flag, though, which eventually
>gets passed to build_tools.build_extension (which I believe does all the
>compiling for weave). It's probably more reasonable to have
>inline_tools.compile_function default to verbose=1 instead of 0, then
>build_extension will print 'Compiling code...' (that should be changed to
>mention weave).
>  
>

Assuming inline_tools doesn't already use logging, might it be 
advantageous to have it use Python's logging module?

 >>> logging.getLogger("scipy.weave").warning("compiling -- this may 
take some time")
WARNING:scipy.weave:compiling -- this may take some time

[I think warning is the lowest level that gets displayed by default]

-tim

Re: [Numpy-discussion] Getting rid of annoying weave nag

[David M. C. <co...@ph...>]

On Fri, 9 Jun 2006 15:19:14 -0600
"Fernando Perez" <fpe...@gm...> wrote:

> On 6/9/06, David M. Cooke <co...@ph...> wrote:
> > On Thu, Jun 08, 2006 at 11:28:04PM -0600, Fernando Perez wrote:
> 
> > > Anyone object to this patch against current numpy SVN to get rid of
> > > this thing? (tracking where the hell that thing was coming from was
> > > all kinds of fun)
> >
> > Go ahead.
> >
> > I'm against random messages being printed out anyways -- I'd get
> > rid of the '<weave: compiling>' too. There's a bunch of code in scipy
> > with 'print' statements that I don't think belong in a library. (Now,
> > if we defined a logging framework, that'd be ok with me!)
> 
> Before I commit anything, let's decide on that one.  Weave used to
> print 'None' whenever it compiled anything, I changed it a while ago
> to the current 'weave:compiling'.  I'm also of the opinion that
> libraries should operate quietly, but with weave I've always wanted
> that message in there.  The reason is that when weave compiles (esp.
> with blitz in the picture), the execution takes a long time.  The same
> function goes from miliseconds to 30 seconds of run time depending on
> whether compilation is happening or not.
> 
> This difference is so dramatic that I think a message is justified
> (absent a proper logging framework).  It's helpful to know that the
> time is going into c++ compilation, and not your code hanging for 30
> seconds.

Ok, I'll give you that one :-) It's the other 1000 uses of print that I'm
concerned about.

inline_tools.compile_function takes a verbose flag, though, which eventually
gets passed to build_tools.build_extension (which I believe does all the
compiling for weave). It's probably more reasonable to have
inline_tools.compile_function default to verbose=1 instead of 0, then
build_extension will print 'Compiling code...' (that should be changed to
mention weave).

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|co...@ph...

Re: [Numpy-discussion] Getting rid of annoying weave nag

[Fernando P. <fpe...@gm...>]

On 6/9/06, David M. Cooke <co...@ph...> wrote:
> On Thu, Jun 08, 2006 at 11:28:04PM -0600, Fernando Perez wrote:

> > Anyone object to this patch against current numpy SVN to get rid of
> > this thing? (tracking where the hell that thing was coming from was
> > all kinds of fun)
>
> Go ahead.
>
> I'm against random messages being printed out anyways -- I'd get
> rid of the '<weave: compiling>' too. There's a bunch of code in scipy
> with 'print' statements that I don't think belong in a library. (Now,
> if we defined a logging framework, that'd be ok with me!)

Before I commit anything, let's decide on that one.  Weave used to
print 'None' whenever it compiled anything, I changed it a while ago
to the current 'weave:compiling'.  I'm also of the opinion that
libraries should operate quietly, but with weave I've always wanted
that message in there.  The reason is that when weave compiles (esp.
with blitz in the picture), the execution takes a long time.  The same
function goes from miliseconds to 30 seconds of run time depending on
whether compilation is happening or not.

This difference is so dramatic that I think a message is justified
(absent a proper logging framework).  It's helpful to know that the
time is going into c++ compilation, and not your code hanging for 30
seconds.

Opinions?

f

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Tim H. <tim...@co...>]

Travis Oliphant wrote:

>Tim Hochberg wrote:
>
>  
>
>>I was going to say that it may help to think of array_interface as 
>>returning a *view*, since that seems to be the semantics that could 
>>probably be implemented safely without too much trouble. However, it 
>>looks like that's not what happens. array_interface->shape and strides 
>>point to the raw shape and strides for the array. That looks like it's a 
>>problem. Isn't:
>>
>>    
>>
>>>>>ai = a.__array_interface__
>>>>>a.shape = newshape
>>>>>          
>>>>>
>>going to result in ai having a stale pointers to shape and strides that 
>>no longer exist? 
>>
>>    
>>
>This is an implementation detail.  I'm still trying to gather some kind 
>of consensus on what to actually do here. 
>
There were three things mixed together in my post:

1. The current implementation of  __array_struct__ looks buggy. Should I 
go ahead and file a bug report so that this behaviour doesn't get 
blindly copied over from __array_struct__ to whatever the final dohickey 
is called or is that going to be totally rewritten in any case.

2. Whether __array_struct__ or __array_interface__ or whatever it gets 
called returns something that's kind of like a view (has it's own copies 
of shape and strides mainly) versus an alias for the original array 
(somehow tries to track the original arrays shape and strides) is a 
semantic difference, not an implementation details.  I suspect that no 
one really cares that much about this and we'll end up doing what's 
easiest to get right; I'm pretty certain that is view semantics. It may 
be helpful to pronounce on that now, since it's possible the semantics 
might influence the name chosen, but I don't think it's critical.

3. The implementation details I provided were, uh, implentation details.

-tim


> There is no such 
>__array_interface__  attribute at this point.
>
>
>-Travis
>
>
>
>_______________________________________________
>Numpy-discussion mailing list
>Num...@li...
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>  
>

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[David M. C. <co...@ph...>]

On Fri, 09 Jun 2006 12:08:51 -0600
Travis Oliphant <oli...@ee...> wrote:

> Tim Hochberg wrote:
> 
> > Sasha wrote:
> >
> >> On 6/8/06, David M. Cooke <co...@ph...> wrote:
> 
> >>>   
> >>
> >> My problem with __array_struct__ returning either a tuple or a CObject
> >> is that array protocol sholuld really provide both.
> >
> This is a convincing argument.   Yes, the array protocol should provide 
> both.  Thus, we can't over-ride the usage of the same name unless that 
> name produces an object through which both interfaces can be obtained.

True, didn't think about that. +1.

> >>> We still need __array_descr__, as the C struct doesn't provide all 
> >>> the info
> >>> that this does.
> >>
> >> What do you have in mind?
> >>
> > Is there any prospect of merging this data into the C struct? It would 
> > be cleaner if all of the information could be embedded into the C 
> > struct, but I can see how that might be a backward compatibility 
> > nightmare.
> 
> I do think it should be merged into the C struct.   The simplest thing 
> to do is to have an additional PyObject * as part of the C struct which 
> could be NULL (or unassigned).  The backward compatibility is a concern 
> but when thinking about what Python 2.6 should support we should not be 
> too crippled by it.
> 
> Perhaps we should just keep __array_struct__ and compress all the other 
> array_interface methods into the __array_interface__ attribute which 
> returns a dictionary from which the Python-side interface can be produced.

+1. I'm ok with two attributes: __array_struct__ (for C), and
__array_interface__ (as a dict for Python). For __array_descr__, I
would require everything that provides an __array_struct__ must also
provide an __array_interface__, then __array_descr__ can become a
'descr' key in __array_interface__. Requiring that would also mean
that any array-like object can be introspected from Python or C.

I think that the array_descr is complicated enough that keeping it as
a Python object is ok: you don't have to reinvent routines to make
tuple-like objects, and handle memory for strings, etc. If you're
using the array interface, you've got Python available: use it. If you
*do* want a C-level version, I'd make it simple, and concatenate the
typestr descriptions of each field together, like '>i2>f8', and forget
the names (you can grab them out of __array_interface__['descr'] if
you need them). That's simple enough to be parseable with sscanf.

> Keep in mind there are two different (but related) issues at play here. 
> 
> 1) What goes in to NumPy 1.0
> 2) What we propose should go into Python 2.6
> 
> 
> I think for #1 we should compress the Python-side array protocol into a 
> single __array_interface__ attribute that returns a dictionary. We 
> should also expand the C-struct to contain what _array_descr_ currently 
> provides. 


-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|co...@ph...

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Andrew S. <str...@as...>]

Travis Oliphant wrote:

> Andrew Straw wrote:
>
>> On the one hand, I feel we should keep __array_struct__ behaving 
>> exactly as it is now. There's already lots of code that uses it, and 
>> it's tremendously useful despite (because of?) it's simplicity. For 
>> these of use cases, the __array_descr__ information has already 
>> proven unnecessary. I must say that I, and probably others, thought 
>> that __array_struct__ would be future-proof. Although the magnitude 
>> of the proposed change to add this information to the C-struct 
>> PyArrayInterface is minor, it still breaks code in the wild.
>>
> I don't see how it breaks any code in the wild to add an additional 
> member to the C-struct.   We could easily handle it in new code with a 
> flag setting (like Python uses).     The only possible problem is 
> looking for it when it is not there.

Ahh, thanks for clarifying. Let me paraphrase to make sure I got it 
right: given a C-struct "inter" of type PyArrayInterface, if and only if 
((inter.flags & HAS_ARRAY_DESCR) == HAS_ARRAY_DESCR) inter could safely 
be cast as PyArrayInterfaceWithArrayDescr and thus expose a new member. 
This does seem to avoid all the issues and maintain backwards 
compatibility. I guess the only potential complaint is that it's a 
little C trick which might be unpalatable to the core Python devs, but 
it doesn't seem egregious to me.

If I do understand this issue, I'm +1 for the above scheme provided the 
core Python devs don't mind.

Cheers!
Andrew

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Travis O. <oli...@ee...>]

Tim Hochberg wrote:

>I was going to say that it may help to think of array_interface as 
>returning a *view*, since that seems to be the semantics that could 
>probably be implemented safely without too much trouble. However, it 
>looks like that's not what happens. array_interface->shape and strides 
>point to the raw shape and strides for the array. That looks like it's a 
>problem. Isn't:
>
> >>> ai = a.__array_interface__
> >>> a.shape = newshape
>
>going to result in ai having a stale pointers to shape and strides that 
>no longer exist? 
>
This is an implementation detail.  I'm still trying to gather some kind 
of consensus on what to actually do here.  There is no such 
__array_interface__  attribute at this point.


-Travis

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Tim H. <tim...@co...>]

Sasha wrote:

>On 6/9/06, Travis Oliphant <oli...@ee...> wrote:
>  
>
>>...   In NumPy this is not quite the rule followed.
>>Bascially attributes are used when getting or setting intrinsinc
>>"properties" of the array.  Attributes are used for properties that are
>>important in defining what an array *is*.   The flags attribute, for
>>example, is an important intrinsinc property of the array but it returns
>>an flags object when it is accessed.   The flat attribute also returns a
>>new object (it is arguable whether it should have been a method or an
>>attribute but it is enough of an intrinsic property --- setting the flat
>>attribute sets elements of the array -- that with historical precedence
>>it was left as an attribute).
>>
>>By this meausure,  the array interface should be an attribute.
>>
>>    
>>
>
>Array interface is not an intrinsic property of the array, but rather
>an alternative  representation of the array itself.
>  
>
I was going to say that it may help to think of array_interface as 
returning a *view*, since that seems to be the semantics that could 
probably be implemented safely without too much trouble. However, it 
looks like that's not what happens. array_interface->shape and strides 
point to the raw shape and strides for the array. That looks like it's a 
problem. Isn't:

 >>> ai = a.__array_interface__
 >>> a.shape = newshape

going to result in ai having a stale pointers to shape and strides that 
no longer exist? Potentially resulting in a segfault? It seems the safe 
approach is to give array_interface it's own shape and strides data. An 
implementation shortcut could be to actually generate a new view in 
array_struct_get and then pass that to PyCObject_FromVoidPtrAndDesc. 
Thus the CObject would have the only handle to the new view and it 
couldn't be corrupted.

[SNIP]

-tim

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Andrew S. <str...@as...>]

On the one hand, I feel we should keep __array_struct__ behaving exactly 
as it is now. There's already lots of code that uses it, and it's 
tremendously useful despite (because of?) it's simplicity. For these of 
use cases, the __array_descr__ information has already proven 
unnecessary. I must say that I, and probably others, thought that 
__array_struct__ would be future-proof. Although the magnitude of the 
proposed change to add this information to the C-struct PyArrayInterface 
is minor, it still breaks code in the wild.

On the other hand, I'm only beginning to grasp the power of the 
__array_descr__ information. So perhaps bumping the 
PyArrayInterface.version to 3 (2 is the current, and as far as I can 
tell, original version) and going forward would be justified. Perhaps 
there's a way towards backwards-compatibility -- the various array 
consumers could presumably support _reading_ both v2 and version 3 
nearly forever, but could spit out warnings when reading v2. It seems v3 
would be a simple superset of v2, so implementation of this wouldn't be 
hard. The challenge will be when a implementor returns a v3 
__array_struct__ to something that reads only v2. For this reason, maybe 
it's better to break backwards compatibility now before even more code 
is written to read v2.

Is it clear what would need to be done to provide a C-struct giving the 
_array_descr_ information?

What's the problem with keeping __array_descr__ access available only at 
the Python level? Your original email suggested limiting the number of 
attributes, which I agree with, but I don't think we need to go to the 
logical extreme. Does simply keeping __array_descr__ as part of the 
Python array interface avoid these issues? At what cost?

Cheers!
Andrew

Travis Oliphant wrote:

>Keep in mind there are two different (but related) issues at play here. 
>
>1) What goes in to NumPy 1.0
>2) What we propose should go into Python 2.6
>
>
>I think for #1 we should compress the Python-side array protocol into a 
>single __array_interface__ attribute that returns a dictionary. We 
>should also expand the C-struct to contain what _array_descr_ currently 
>provides. 
>
>
>-Travis
>
>
>
>_______________________________________________
>Numpy-discussion mailing list
>Num...@li...
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>  
>

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Sasha <nd...@ma...>]

On 6/9/06, Travis Oliphant <oli...@ee...> wrote:
> ...   In NumPy this is not quite the rule followed.
> Bascially attributes are used when getting or setting intrinsinc
> "properties" of the array.  Attributes are used for properties that are
> important in defining what an array *is*.   The flags attribute, for
> example, is an important intrinsinc property of the array but it returns
> an flags object when it is accessed.   The flat attribute also returns a
> new object (it is arguable whether it should have been a method or an
> attribute but it is enough of an intrinsic property --- setting the flat
> attribute sets elements of the array -- that with historical precedence
> it was left as an attribute).
>
> By this meausure,  the array interface should be an attribute.
>

Array interface is not an intrinsic property of the array, but rather
an alternative  representation of the array itself.

Flags are properly an attribute because they are settable.  Something like

>>> x.flags()['WRITEABLE'] = False

although technically possible, would be quite ugly.

Similarly, shape attribute,  although fails my rule of thumb by
creating a new object,

>>> x.shape is x.shape
False

is justifiably an attribute because otherwise two methods: get_shape
and set_shape would be required.

I don't think "flat" should be an attribute, however.   I could not
find the reference, but I remember a discussion of why __iter__ should
not be an attribute and IIRC the answer was because an iterator has a
mutable state that is not reflected in the underlying object:

>>> x = arange(5)
>>> i = x.flat
>>> list(i)
[0, 1, 2, 3, 4]
>>> list(i)
[]
>>> list(x.flat)
[0, 1, 2, 3, 4]

> >> My problem with __array_struct__ returning either a tuple or a CObject
> >> is that array protocol sholuld really provide both.
> >
> This is a convincing argument.   Yes, the array protocol should provide
> both.  Thus, we can't over-ride the usage of the same name unless that
> name produces an object through which both interfaces can be obtained.
>
> Is that Sasha's suggestion?
>
It was, but I quckly retracted it in favor of a mechanism to unpack the CObject.
FWIW, I am also now -0 on the name change from __array_struct__ to
__array_interface__ if what it provides is just a struct wrapped in a
CObject.

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Alexander B. <ale...@gm...>]

On 6/9/06, Travis Oliphant <oli...@ee...> wrote:
> ...   In NumPy this is not quite the rule followed.
> Bascially attributes are used when getting or setting intrinsinc
> "properties" of the array.  Attributes are used for properties that are
> important in defining what an array *is*.   The flags attribute, for
> example, is an important intrinsinc property of the array but it returns
> an flags object when it is accessed.   The flat attribute also returns a
> new object (it is arguable whether it should have been a method or an
> attribute but it is enough of an intrinsic property --- setting the flat
> attribute sets elements of the array -- that with historical precedence
> it was left as an attribute).
>
> By this meausure,  the array interface should be an attribute.
>

Array interface is not an intrinsic property of the array, but rather
an alternative  representation of the array itself.

Flags are properly an attribute because they are settable.  Something like

>>> x.flags()['WRITEABLE'] = False

although technically possible, would be quite ugly.

Similarly, shape attribute,  although fails my rule of thumb by
creating a new object,

>>> x.shape is x.shape
False

is justifiably an attribute because otherwise two methods: get_shape
and set_shape would be required.

I don't think "flat" should be an attribute, however.   I could not
find the reference, but I remember a discussion of why __iter__ should
not be an attribute and IIRC the answer was because an iterator has a
mutable state that is not reflected in the underlying object:

>>> x = arange(5)
>>> i = x.flat
>>> list(i)
[0, 1, 2, 3, 4]
>>> list(i)
[]
>>> list(x.flat)
[0, 1, 2, 3, 4]

> >> My problem with __array_struct__ returning either a tuple or a CObject
> >> is that array protocol sholuld really provide both.
> >
> This is a convincing argument.   Yes, the array protocol should provide
> both.  Thus, we can't over-ride the usage of the same name unless that
> name produces an object through which both interfaces can be obtained.
>
> Is that Sasha's suggestion?
>
It was, but I quckly retracted it in favor of a mechanism to unpack the CObject.
FWIW, I am also now -0 on the name change from __array_struct__ to
__array_interface__ if what it provides is just a struct wrapped in a
CObject.

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Travis O. <oli...@ee...>]

Tim Hochberg wrote:

> Sasha wrote:
>
>> On 6/8/06, David M. Cooke <co...@ph...> wrote:
>>  
>>
>>> ...
>>> +0 for name change; I'm happy with it as an attribute.
>>>
>>>   
>>
>> My rule of thumb for choosing between an attribute and a method is
>> that attribute access should not create new objects.
>>
Interesting rule.    In NumPy this is not quite the rule followed.  
Bascially attributes are used when getting or setting intrinsinc 
"properties" of the array.  Attributes are used for properties that are 
important in defining what an array *is*.   The flags attribute, for 
example, is an important intrinsinc property of the array but it returns 
an flags object when it is accessed.   The flat attribute also returns a 
new object (it is arguable whether it should have been a method or an 
attribute but it is enough of an intrinsic property --- setting the flat 
attribute sets elements of the array -- that with historical precedence 
it was left as an attribute). 

By this meausure,  the array interface should be an attribute.


>>>   
>>
>> My problem with __array_struct__ returning either a tuple or a CObject
>> is that array protocol sholuld really provide both.
>
This is a convincing argument.   Yes, the array protocol should provide 
both.  Thus, we can't over-ride the usage of the same name unless that 
name produces an object through which both interfaces can be obtained.

Is that Sasha's suggestion?

>
> A single attribute seems pretty appealing to me, I'm don't see much 
> use for anything else.
>
>
>>> We still need __array_descr__, as the C struct doesn't provide all 
>>> the info
>>> that this does.
>>>
>>>   
>>
>> What do you have in mind?
>>  
>>
> Is there any prospect of merging this data into the C struct? It would 
> be cleaner if all of the information could be embedded into the C 
> struct, but I can see how that might be a backward compatibility 
> nightmare.

I do think it should be merged into the C struct.   The simplest thing 
to do is to have an additional PyObject * as part of the C struct which 
could be NULL (or unassigned).  The backward compatibility is a concern 
but when thinking about what Python 2.6 should support we should not be 
too crippled by it.

Perhaps we should just keep __array_struct__ and compress all the other 
array_interface methods into the __array_interface__ attribute which 
returns a dictionary from which the Python-side interface can be produced.

Keep in mind there are two different (but related) issues at play here. 

1) What goes in to NumPy 1.0
2) What we propose should go into Python 2.6


I think for #1 we should compress the Python-side array protocol into a 
single __array_interface__ attribute that returns a dictionary. We 
should also expand the C-struct to contain what _array_descr_ currently 
provides. 


-Travis

Re: [Numpy-discussion] Array Protocol change for Python 2.6

[Tim H. <tim...@co...>]

Sasha wrote:

>On 6/9/06, Tim Hochberg <tim...@co...> wrote:
>  
>
>>Sasha wrote:
>>...
>>    
>>
>>>My rule of thumb for choosing between an attribute and a method is
>>>that attribute access should not create new objects.
>>>
>>>      
>>>
>>Conceptually at least, couldn't there be a single __array_interface__
>>object associated with a given array? In that sense, it doesn't really
>>feel like creating a new object.
>>
>>    
>>
>In my view, conceptually, __array_interface__ creates a adaptor to the
>array-like object.  What are the advantages of it being an attribute?
>It is never settable, so the most common advantage of packing get/set
>methods in a single attribute can be rulled out.  Saving typing of
>'()' cannot be taken seriousely when the name contains a pair of
>double underscores :-).
>
>There was a similar issue discussed on the python-3000 mailing list
>with respect to __hash__ method
><http://mail.python.org/pipermail/python-3000/2006-April/000362.html>.
>  
>
Isn't __array_interface__ always O(1)? By the criteria in that thread, 
that would make is good candidate for being an attribute.

[Stare at __array_interface__ spec...think..stare...]

OK, I think I'm coming around to making it a function. Presumably, in:

 >>> a = arange(6)
 >>> ai1 = a.__array_interface__()
 >>> a.shape = [3, 2]
 >>> ai2 = a.__array_interface__()

ai1 and ai2 will be different objects with different objects, pointing 
to structs with different shape and stride attributes. So, in that sense 
it's not conceptually constant and should be a function.

What happens if I then delete or resize a? Hmmm. It looks like that's 
probably OK since CObject grabs a reference to a.

FWIW, at this point, I marginally prefer array_struct to array_interface.

>  
>
>>....
>>    
>>
>>>My problem with __array_struct__ returning either a tuple or a CObject
>>>is that array protocol sholuld really provide both.  CObject is
>>>useless for interoperability at python level and a tuple (or dict) is
>>>inefficient at the C level.  Thus a good array-like object should
>>>really provide both __array_struct__ for use by C modules and
>>>__array_tuple__ (or whatever) for use by python modules.  On the other
>>>hand, making both required attributes/methods will put an extra burden
>>>on package writers.  Moreover, a pure python implementation of an
>>>array-like object will not be able to provide __array_struct__ at all.
>>>One possible solution would be an array protocol metaclass that adds
>>>__array_struct__ to a class with __array_tuple__ and __array_tuple__
>>>to a class with __array_struct__  (yet another argument to make both
>>>methods).
>>>
>>>
>>>      
>>>
>>I don't understand this. I'm don't see  how bringing in metaclass is
>>going to help a pure python type provide a sensible __array_struct__.
>>That seems like a hopeless task. Shouldn't pure python implementations
>>just provide __array__?
>>
>>    
>>
>
>My metaclass idea is very similar to your unpack_interface suggestion.
> A metaclass can autonatically add
>
>def __array_tuple__(self):
>    return unpack_interface(self.__array_interface__())
>
>
>or
>
>def __array_interface__(self):
>    return pack_interface(self.__array_tuple__())
>
>to a class that only implements only one of the two required methods.
>  
>
It seems like 99% of the people will never care about this at the Python 
level, so adding an extra attribute is mostly clutter. For those few who 
do care a function seems preferable. To be honest, I don't actually see 
a need for anything other than the basic __array_struct__.


>>A single attribute seems pretty appealing to me, I'm don't see much use
>>for anything else.
>>    
>>
>
>I don't mind just having __array_struct__ that must return a CObject.
>My main objection was against a method/attribute that may return
>either CObject or something else.  That felt like shifting the burden
>from package writer to the package user.
>  
>
I concur.


>
>_______________________________________________
>Numpy-discussion mailing list
>Num...@li...
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>  
>