Menu
▾
▴
numpy-discussion
|
From: Perry G. <pe...@st...> - 2002-06-13 20:39:39
|
> I guess the best reason to bite the bullet and carry around state > information > would be if there were significant other cases where one also > would want to > optimize operations under the hood. If there isn't much else in > this direction > then the effort involved might not be justified. One thing that bugs me in > Numeric (and that might already have been solved in numarray) is that > e.g. ``ravel`` (and I think also ``transpose``) creates > unnecessary copies, > whereas ``.flat`` doesn't, but won't work in all cases (viz. when > the array is > non-contiguous), so I can either have ugly or inefficient code. > I guess that depends on what you mean by unnecessary copies. If the array is non-contiguous what would you have it do? > > a feature one wants even though they are not the default, it turns > > out that it isn't all that simple to obtain views without sacrificing > > ordinary slicing syntax to obtain a view. It is simple to obtain > > copies of view slices though. > > I'm not sure I understand the above. What is the problem with > ``a.view[1:3]`` > (or``a.view()[1:3])? > I didn't mean to imply it wasn't possible, but that it was not quite as clean. The thing I don't like about this approach (or Paul's suggestion of a.sub) is the creation of an odd object that has as its only purpose being sliced. (Even worse, in my opinion, is making it a different kind of array where slicing behaves differently. That will lead to the problem we have discussed for other kinds of array behavior, namely, how do you keep from being confused about a particular array's slicing behavior). That could lead to confusion as well. Many may be under the impression that x = a.view makes x refer to an array when it doesn't. Users would need to know that a.view without a '[' is usually an error. Sure it's not hard to implement. But I don't view it as that clean a solution. On the other hand, a[1:3].copy() (or alternatively, a[1:3].copy) is another array just like any other. > Perry |
|
From: Alexander S. <a.s...@gm...> - 2002-06-14 00:35:44
|
"Perry Greenfield" <pe...@st...> writes:
> > I guess the best reason to bite the bullet and carry around state
> > information
> > would be if there were significant other cases where one also
> > would want to
> > optimize operations under the hood. If there isn't much else in
> > this direction
> > then the effort involved might not be justified. One thing that bugs me in
> > Numeric (and that might already have been solved in numarray) is that
> > e.g. ``ravel`` (and I think also ``transpose``) creates
> > unnecessary copies,
> > whereas ``.flat`` doesn't, but won't work in all cases (viz. when
> > the array is
> > non-contiguous), so I can either have ugly or inefficient code.
> >
> I guess that depends on what you mean by unnecessary copies.
In most cases the array of which I desire a flattened representation is
contiguous (plus, I usually don't intend to modify it). Consequently, in most
cases I don't want to any copies of it to be created (especially not if it is
really large -- which is not seldom the case).
The fact that you can never really be sure whether you can actually use
``.flat``, without checking beforehand if the array is in fact contiguous (I
don't think there are many guarantees about something being contiguous, or are
there?) and that ravel will always work but has a huge overhead, suggests to
me that something is not quite right.
> If the array is non-contiguous what would you have it do?
Simple -- in that case 'lazy ravel' would do the same as 'ravel' currently
does, create a copy (or alternatively rearrange the memory representation to
make it non-contiguous and then create a lazy copy, but I don't know whether
this would be a good or even feasible idea).
A lazy version of ravel would have the same semantics as ravel but only create
an actual copy if necessary-- which means as long as no modification takes
place and the array is non-contiguous, it will be sufficient to return the
``.flat`` (for starters). If it is contiguous than the copying can't be
helped, but these cases are rare and currently you either have to test for
them explicitly or slow everything down and waste memory by just always using
``ravel()``.
For example, if bar is contiguous ``foo = ravel(bar)`` would be
computationally equivalent to ``bar.flat``, as long as neither of them is
modified, but semantically equivalent to the current ``foo = ravel(bar)`` in
all cases.
Thus you could now write:
>>> a = ravel(a)[20:]
wherever you've written this boiler-plate code before:
>>> if a.iscontiguous():
>>> a = a.flat[20:]
>>> else:
>>> a = ravel(a)[20:]
without any loss of performance.
>
> > > a feature one wants even though they are not the default, it turns
> > > out that it isn't all that simple to obtain views without sacrificing
> > > ordinary slicing syntax to obtain a view. It is simple to obtain
> > > copies of view slices though.
> >
> > I'm not sure I understand the above. What is the problem with
> > ``a.view[1:3]``
> > (or``a.view()[1:3])?
> >
> I didn't mean to imply it wasn't possible, but that it was not
> quite as clean. The thing I don't like about this approach (or
> Paul's suggestion of a.sub) is the creation of an odd object
> that has as its only purpose being sliced. (Even worse, in my
I personally don't find it messy. And please keep in mind that the ``view``
construct would only very seldomly be used if copy-on-demand is the default
-- as I said, I've only needed the aliasing behavior once -- no doubt it was
really handy then, but the fact that e.g. matlab doesn't have anything along
those lines (AFAIK) suggests that many people will never need it.
So even if ``.view`` is messy, I'd rather have something messy that is almost
never used, in exchange for (what I perceive as) significantly nicer and
cleaner semantics for something that is used all the time (array slicing;
alias slicing is messy in at least the respect that it breaks standard usage
and generic sequence code as well as causing potentially devious
bugs. Unexpected behaviors like phantom buffers kept alive in their entirety
by partial views etc. or what ``A = A[::-1]`` does are not exactly pretty
either).
> opinion, is making it a different kind of array where slicing
> behaves differently. That will lead to the problem we have
> discussed for other kinds of array behavior, namely, how do
> you keep from being confused about a particular array's slicing
> behavior). That could lead to confusion as well. Many may be
I don't see that problem, frankly. The view is *not* an array. It doesn't need
(and shouldn't have) anything except a method to access slices (__getitem__).
As mentioned before, I also regard it as highly desirable that ``b =
a.view[3:10]`` sticks out immediately. This signals "warning -- potentially
tricky code ahead". Nothing in ``b = a[3:10]`` tells you that someone intends
to modify a and b depedently (because in more than 9 out of 10 cases he won't)
-- now *this* is confusing.
> under the impression that x = a.view makes x refer to an array
> when it doesn't. Users would need to know that a.view without
> a '[' is usually an error.
Since the ``.view`` shouldn't allow anything except slicing, they'll soon find
out ("Error: you can't multiply me, I'm a view and not an array"). And I can't
see why that would be harder to figure out (or look up in the docu) than that
a[1:3] creates an alias and *not* a copy contrary to *everything* else you've
ever heard or read about python sequences (especially since in most cases it
will work as intended).
Also what exactly is the confused person's notion of the purpose of ``x =
a.view`` supposed to be? That ``x = a`` is what ``x = a.copy()`` really does
and that to create aliases an alias to ``a`` they would have to use
``x = a.view``? In that case they'd better read the python tutorial before they do
any more python programming, because they are in for all kinds of unpleasant
surprises (``a = []; b = a; b[1] = 3; print a`` -- oops).
alex
--
Alexander Schmolck Postgraduate Research Student
Department of Computer Science
University of Exeter
A.S...@gm... http://www.dcs.ex.ac.uk/people/aschmolc/
|
|
From: Perry G. <pe...@st...> - 2002-06-14 15:01:37
|
<Alexander Schmolck writes>:
<Perry Greenfield writes>:
> > I guess that depends on what you mean by unnecessary copies.
>
> In most cases the array of which I desire a flattened representation is
> contiguous (plus, I usually don't intend to modify it).
> Consequently, in most
> cases I don't want to any copies of it to be created (especially
> not if it is
> really large -- which is not seldom the case).
>
Numarray already returns a view of the array if it is contiguous.
Copies are only produced if it is non-contiguous. I assume that
is the behavior you are asking for?
> The fact that you can never really be sure whether you can actually use
> ``.flat``, without checking beforehand if the array is in fact
> contiguous (I
> don't think there are many guarantees about something being
> contiguous, or are
> there?) and that ravel will always work but has a huge overhead,
> suggests to
> me that something is not quite right.
>
Not for numarray, at least in this context.
> > If the array is non-contiguous what would you have it do?
>
> Simple -- in that case 'lazy ravel' would do the same as 'ravel' currently
> does, create a copy (or alternatively rearrange the memory
> representation to
> make it non-contiguous and then create a lazy copy, but I don't
> know whether
> this would be a good or even feasible idea).
>
> A lazy version of ravel would have the same semantics as ravel
> but only create
> an actual copy if necessary-- which means as long as no modification takes
> place and the array is non-contiguous, it will be sufficient to return the
> ``.flat`` (for starters). If it is contiguous than the copying can't be
> helped, but these cases are rare and currently you either have to test for
> them explicitly or slow everything down and waste memory by just
> always using
> ``ravel()``.
>
Currently for numarray .flat will fail if it isn't contiguous. It isn't
clear if this should change. If .flat is meant to be a view always, then
it should always fail it the array is not contiguous. Ravel is not
guaranteed to be a view.
This is a problematic issue if we decide to switch from view to copy
semantics. If slices produce copies, then does .flat? If so, then
how does one produce a flattened view? x.view.flat?
> For example, if bar is contiguous ``foo = ravel(bar)`` would be
> computationally equivalent to ``bar.flat``, as long as neither of them is
> modified, but semantically equivalent to the current ``foo =
> ravel(bar)`` in
> all cases.
>
> Thus you could now write:
>
> >>> a = ravel(a)[20:]
>
> wherever you've written this boiler-plate code before:
>
> >>> if a.iscontiguous():
> >>> a = a.flat[20:]
> >>> else:
> >>> a = ravel(a)[20:]
>
> without any loss of performance.
>
I believe this is already true in numarray.
>
> I personally don't find it messy. And please keep in mind that
> the ``view``
> construct would only very seldomly be used if copy-on-demand is
> the default
> -- as I said, I've only needed the aliasing behavior once -- no
> doubt it was
> really handy then, but the fact that e.g. matlab doesn't have
> anything along
> those lines (AFAIK) suggests that many people will never need it.
>
You're kidding, right? Particularly after arguing for aliasing
semantics in the previous paragraph for .flat ;-)
>
> Also what exactly is the confused person's notion of the purpose of ``x =
> a.view`` supposed to be? That ``x = a`` is what ``x = a.copy()``
> really does
> and that to create aliases an alias to ``a`` they would have to use
> ``x = a.view``? In that case they'd better read the python
> tutorial before they do
> any more python programming, because they are in for all kinds of
> unpleasant
> surprises (``a = []; b = a; b[1] = 3; print a`` -- oops).
>
This is basically true, though the confusion may be that a.view is
an array object that has different slicing behavior instead of
an non-array object that can be sliced to produce a view. I don't
view it as a major issue but I do see how may mistakenly infer that.
Perry
|
|
From: Tim H. <tim...@ie...> - 2002-06-14 16:13:57
|
<"Perry Greenfield" writes> [SNIP] > Numarray already returns a view of the array if it is contiguous. > Copies are only produced if it is non-contiguous. I assume that > is the behavior you are asking for? This is one horrible aspect of NumPy that I hope you get rid of. I've been burned by this several times -- I expected a view, but silently got a copy because my array was noncontiguous. If you go with copy semantics, this will go away, if you go with view semantics, this should raise an exception instead of silently copying. Ditto with reshape, etc. In my experience, this is a source of hard to find bugs (as opposed to axes issues which tend to produce shallow bugs). [SNIP] > Currently for numarray .flat will fail if it isn't contiguous. It isn't > clear if this should change. If .flat is meant to be a view always, then > it should always fail it the array is not contiguous. Ravel is not > guaranteed to be a view. Ravel should either always return a view or always return a copy -- I don't care which > This is a problematic issue if we decide to switch from view to copy > semantics. If slices produce copies, then does .flat? If so, then > how does one produce a flattened view? x.view.flat? Wouldn't that just produce a copy of the view? Unless you did some weird special casing on view? The following would work, although it's a little clunky. flat_x = x.view[:] # Or however "get me a view" would be spelled. flat_x.shape = (-1,) -tim |
|
From: Alexander S. <a.s...@gm...> - 2002-06-16 22:58:57
|
"Perry Greenfield" <pe...@st...> writes:
> <Alexander Schmolck writes>:
> <Perry Greenfield writes>:
> > > I guess that depends on what you mean by unnecessary copies.
> >
> > In most cases the array of which I desire a flattened representation is
> > contiguous (plus, I usually don't intend to modify it).
> > Consequently, in most
> > cases I don't want to any copies of it to be created (especially
> > not if it is
> > really large -- which is not seldom the case).
> >
> Numarray already returns a view of the array if it is contiguous.
> Copies are only produced if it is non-contiguous. I assume that
> is the behavior you are asking for?
Not at all -- in fact I was rather shocked when my attention was drawn to the
fact that this is also the behavior of Numeric -- I had thought that ravel
would *always* create a copy. I absolutely agree with the other posters that
remarked that different behavior of ravel (creating a copy vs creating a view,
depending on whether the argument is contiguous) is highly undesirable and
error-prone (especially since it is not even possible to determine at compile
time which behavior will occur, if I'm not mistaken). In fact, I think this
behavior is worse than what I incorrectly assumed to be the case.
What I was arguing for is a ravel that always has the same semantics, (namely
creating a copy) but tha -- because it would create the copy only demand --
would be just as efficient as using .flat when
a) its argument were contiguous; and
b) neither the result nor the argument were modified while both are alive.
The reason that I view `.flat` as a hack, is that it is an operation that is
there exclusively for efficiency reasons and has no well defined semantics --
it will only work stochastically, giving better performance in certain
cases. Thus you have to cast lots whether you actually use it at runtime
(calling .iscontiguous) and always have a fall-back scheme (most likely using
ravel) at hand -- there seems to be no way to determine at compile time what's
going to happen.
I don't think a language or a library should have any such constructs or at
least strive to minimize their number. The fact that the current behavior of
ravel actually achieves the effect I want in most cases doesn't justify its
obscure behavior in my eyes, which translates into a variation of the
boiler-plate code previously mentioned (``if a.iscontiguous:...else:``) when
you actually want a *single* ravelled copy and it also is a very likely
candidate for extremely hard to find bugs.
One nice thing about python is that there is very little undefined
behavior. I'd like to keep it that way.
[snipped]
> > I personally don't find it messy. And please keep in mind that
> > the ``view``
> > construct would only very seldomly be used if copy-on-demand is
> > the default
> > -- as I said, I've only needed the aliasing behavior once -- no
> > doubt it was
> > really handy then, but the fact that e.g. matlab doesn't have
> > anything along
> > those lines (AFAIK) suggests that many people will never need it.
> >
> You're kidding, right? Particularly after arguing for aliasing
> semantics in the previous paragraph for .flat ;-)
I didn't argue for any semantics of ``.flat`` -- I just pointed out that I
found the division of labour that I (incorrectly) assumed to be the case an
ugly hack (for the reasons outlined above):
``ravel``: always works, but always creates copy
(which might be undesirable wastage of resources); [this was
mistaken; the real semantics are: always works, creates view if
contiguous, copy otherwise]
``.flat``: behavior undefined at compile time, a runtime-check can be used
to ensure that it can be used as a more efficient alternative to
``ravel`` in some cases.
If I now understand the behavior of both ``ravel`` and ``.flat`` correctly
then I can't currently see *any* raison d'être for a ``.flat`` attribute.
If, as I would hope, the behavior of ravel is changed to always create copies
(ideally on-demand), then matters might look different. In that case, it might
be justifiable to have ``.flat`` as a specialized construct analogous to what
I proposed as``.view``, but only if there is some way to make it work (the
same) for both contiguous and non-contiguous arrays. I'm not sure that it
would be needed at all (especially with a lazy ravel).
alex
--
Alexander Schmolck Postgraduate Research Student
Department of Computer Science
University of Exeter
A.S...@gm... http://www.dcs.ex.ac.uk/people/aschmolc/
|
|
From: John J. L. <jj...@po...> - 2002-06-14 19:21:46
|
On 14 Jun 2002, Alexander Schmolck wrote: [...] > The fact that you can never really be sure whether you can actually use > ``.flat``, without checking beforehand if the array is in fact > contiguous (I don't think there are many guarantees about something > being contiguous, or are there?) and that ravel will always work but has > a huge overhead, suggests to me that something is not quite right. Why does ravel have a huge overhead? It seems it already doesn't copy unless required: search for 'Chacking' -- including the mis-spelling -- in this thread: http://groups.google.com/groups?hl=en&lr=&threadm=abjbfp%241t9%241%40news5.svr.pol.co.uk&rnum=1&prev=/groups%3Fq%3Diterating%2Bover%2Bthe%2Bcells%2Bgroup:comp.lang.python%26hl%3Den%26lr%3D%26scoring%3Dr%26selm%3Dabjbfp%25241t9%25241%2540news5.svr.pol.co.uk%26rnum%3D1 or start up your Python interpreter, if you're less lazy than me. John |
|
From: Konrad H. <hi...@cn...> - 2002-06-14 17:51:04
|
"Perry Greenfield" <pe...@st...> writes: > I didn't mean to imply it wasn't possible, but that it was not > quite as clean. The thing I don't like about this approach (or > Paul's suggestion of a.sub) is the creation of an odd object > that has as its only purpose being sliced. (Even worse, in my Not necessarily. We could decide that array.view is a view of the full array object, and that slicing views returns subviews. > opinion, is making it a different kind of array where slicing > behaves differently. That will lead to the problem we have > discussed for other kinds of array behavior, namely, how do A view could be a different type of object, even though much of the implementation would be shared with arrays. This would help to reduce confusion. > behavior). That could lead to confusion as well. Many may be > under the impression that x = a.view makes x refer to an array > when it doesn't. Users would need to know that a.view without > a '[' is usually an error. Why? It would be a full-size view, which might actually be useful in many situations. My main objection to changing the slicing behaviour is, like with some other proposed changes, compatibility. Even though view behaviour is not required by every NumPy program, there are people out there who use it and finding the locations in the code that need to be changed is a very tricky business. It may keep programmers from switching to Numarray in spite of benefits elsewhere. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- |
|
From: Perry G. <pe...@st...> - 2002-06-14 19:18:42
|
<Konrad Hinsen writes>: <Perry Greenfield writes>: > > > I didn't mean to imply it wasn't possible, but that it was not > > quite as clean. The thing I don't like about this approach (or > > Paul's suggestion of a.sub) is the creation of an odd object > > that has as its only purpose being sliced. (Even worse, in my > > Not necessarily. We could decide that > > array.view > > is a view of the full array object, and that slicing views returns > subviews. > > > opinion, is making it a different kind of array where slicing > > behaves differently. That will lead to the problem we have > > discussed for other kinds of array behavior, namely, how do > > A view could be a different type of object, even though much of the > implementation would be shared with arrays. This would help to > reduce confusion. > I'd be strongly against this. This has the same problem that other customized array objects have (whether regarding slicing behavior, operators, coercion...). In particular, it is clear which kind it is when you create it, but you may pass it to a module that presumes different array behavior. Having different kind of arrays floating around just seems like an invitation for confusion. I'm very much in favor of picking one or the other behaviors and then making some means of explicitly getting the other behavior. > > behavior). That could lead to confusion as well. Many may be > > under the impression that x = a.view makes x refer to an array > > when it doesn't. Users would need to know that a.view without > > a '[' is usually an error. > > Why? It would be a full-size view, which might actually be useful > in many situations. > But one can do that simply by x = a (Though there is the issue that one could do the following which is not the same: x = a.view x.shape = (2,50) so that x is a full array view with a different shape than a) ******** I understand the backward compatibilty issue here, but it is clear that this is an issue that appears to be impossible to get a consensus on. There appear to be significant factions that care passionately about copy vs view and no matter what decision is made many will be unhappy. Perry |
|
From: Konrad H. <hi...@cn...> - 2002-06-15 08:55:56
|
> I'd be strongly against this. This has the same problem that other > customized array objects have (whether regarding slicing behavior, > operators, coercion...). In particular, it is clear which kind it > is when you create it, but you may pass it to a module that > presumes different array behavior. Having different kind of arrays We already have that situation with lists and arrays (and in much of my code netCDF arrays, which have copy semantics) , but in my experience this has never caused confusion. Most general code working on sequences doesn't modify elements at all. When it does, it either clearly requires view semantics (a function you call in order to modify (parts of) an array) or clearly requires copy semantics (a function that uses an array argument as an initial value that it then modifies). > floating around just seems like an invitation for confusion. I'm > very much in favor of picking one or the other behaviors and then > making some means of explicitly getting the other behavior. Then the only solution I see is the current one: default behaviour is view, and when you want a copy yoy copy explicitly. The inverse is not possible, once you made a copy you can't make it behave like a view anymore. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- |
|
From: Scott R. <ra...@ph...> - 2002-06-15 13:12:48
|
On Sat, Jun 15, 2002 at 10:53:17AM +0200, Konrad Hinsen wrote: > > floating around just seems like an invitation for confusion. I'm > > very much in favor of picking one or the other behaviors and then > > making some means of explicitly getting the other behavior. > > Then the only solution I see is the current one: default behaviour is > view, and when you want a copy yoy copy explicitly. The inverse is not > possible, once you made a copy you can't make it behave like a view > anymore. I don't think it is necessary to create the other object _from_ the default one. You could have copy behavior be the default, and if you want a view of some array you simply request one explicitly with .view, .sub, or whatever. Since creating a view is "cheap" compared to creating a copy, there is nothing sacrificed doing things in this manner. Scott -- Scott M. Ransom Address: McGill Univ. Physics Dept. Phone: (514) 398-6492 3600 University St., Rm 338 email: ra...@ph... Montreal, QC Canada H3A 2T8 GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 |
|
From: Konrad H. <hi...@cn...> - 2002-06-17 09:00:46
|
> > Then the only solution I see is the current one: default behaviour is > > view, and when you want a copy yoy copy explicitly. The inverse is not > > possible, once you made a copy you can't make it behave like a view > > anymore. > > I don't think it is necessary to create the other object _from_ > the default one. You could have copy behavior be the default, > and if you want a view of some array you simply request one > explicitly with .view, .sub, or whatever. Let's make this explicit. Given the following four expressions, 1) array 2) array[0] 3) array.view 4) array.view[0] what would the types of each of these objects be according to your proposal? What would the indexing behaviour of those types be? I don't see how you can avoid having either two types or two different behaviours within one type. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- |
|
From: Scott R. <ra...@ph...> - 2002-06-17 16:34:24
|
On June 17, 2002 04:57 am, Konrad Hinsen wrote: > > > Then the only solution I see is the current one: default behaviour is > > > view, and when you want a copy yoy copy explicitly. The inverse is not > > > possible, once you made a copy you can't make it behave like a view > > > anymore. > > > > I don't think it is necessary to create the other object _from_ > > the default one. You could have copy behavior be the default, > > and if you want a view of some array you simply request one > > explicitly with .view, .sub, or whatever. > > Let's make this explicit. Given the following four expressions, > > 1) array > 2) array[0] > 3) array.view > 4) array.view[0] > > what would the types of each of these objects be according to your > proposal? What would the indexing behaviour of those types be? > I don't see how you can avoid having either two types or two > different behaviours within one type. If we assume that a slice returns a copy _always_, then I agree that #4 in your list above would not give a user what they would expect: array.view[0] would give the view of a copy of array[0], _not_ a view of array[0] which is probably what is wanted. I _think_ that this could be fixed by making view (or something similar) an option of the slice rather than a method of the object. For example (assuming that a is an array): Expression: Returns: Slicing Behavior: a or a[:] Copy of all of a Returns a copy of the sub-array a[0] Copy of a[0] Returns a copy of the sub-array a[:,view] View of all of a Returns a copy of the sub-array a[0,view] View of a[0] Returns a copy of the sub-array Notice that it is possible to return a copy of a sub-array from a view since you have access (through a pointer) to the original array data. Scott -- Scott M. Ransom Address: McGill Univ. Physics Dept. Phone: (514) 398-6492 3600 University St., Rm 338 email: ra...@ph... Montreal, QC Canada H3A 2T8 GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 |
|
From: Chris B. <Chr...@no...> - 2002-06-17 22:48:29
|
Konrad Hinsen wrote:
> Let's make this explicit. Given the following four expressions,
> 1) array
> 2) array[0]
> 3) array.view
> 4) array.view[0]
I thought I had I clear idea of what I wanted here, which was the
non-view stuff being the same as Python lists, but I discovered
something: Python lists provide slices that are copies, but they are
shallow copies, so nested lists, which are sort-of the equivalent of
multidimensional arrays, act a lot like the view behavior of NumPy
arrays:
make a "2-d" list
>>> l = [[i, 1+5] for i in range(5)]
>>> l
[[0, 6], [1, 6], [2, 6], [3, 6], [4, 6]]
make an array that is the same:
>>> a = array(l)
array([[0, 6],
[1, 6],
[2, 6],
[3, 6],
[4, 6]])
assign a new binding to the first element:
>>> b = a[0]
>>> m = l[0]
change something in it:
>>> b[0] = 30
>>> a
array([[30, 6],
[ 1, 6],
[ 2, 6],
[ 3, 6],
[ 4, 6]])
The first array is changed
Change something in the first element of the list:
>>> m[0] = 30
>>> l
[[30, 6], [1, 6], [2, 6], [3, 6], [4, 6]]
The first list is changed too.
Now try slices instead:
>>> b = a[2:4]
change an element in the slice:
>>>> b[1,0] = 55
>>> a
array([[30, 6],
[ 1, 6],
[ 2, 6],
[55, 6],
[ 4, 6]])>> a
The first array is changed
Now with the list
>>> m = l[2:4]
>>> m
[[2, 6], [3, 6]]
This is a copy, but it is a shallow copy, so:
>>> m[1][0] = 45
Change an element
>>> l
[[30, 6], [1, 6], [2, 6], [45, 6], [4, 6]]
The list is changed, but:
m[0] = [56,65]
>>> l
[[30, 6], [1, 6], [2, 6], [45, 6], [4, 6]]
The list doesn't change, where:
>>> b[0] = [56,65]
>>> a
array([[30, 6],
[ 1, 6],
[56, 65],
[55, 6],
[ 4, 6]])
The array does change
My conclusion is that nested lists and Arrays simply are different
beasts so we can't expect complete compatibility. I'm also wondering why
lists have that weird behavior of a single index returning a reference,
and a slice returning a copy. Perhaps it has something to so with the
auto-resizing of lists. That being said, I still like the idea of slices
producing copies, so:
> 1) array
An Array like we have now, but slice-is-copy
semantics.
> 2) array[0]
An Array of rank one less than array, sharing data with array
> 3) array.view
An object that can do nothing but create other Arrays that share data
with array. I don't know if is possible but I'd be just as happy if
array.view returned None, and array.view[slice] returned an Array that
shared data with array. Perhaps there is some other notation that could
do this.
> 4) array.view[0]
Same as 2)
To add a few:
5) array[0:1]
An Array with a copy of the data in array[0]
6) array.view[0:1]
An Array sharing data with array
As I write this, I am starting to think that this is all a bit strange.
Even though lists treat slices and indexes differently, perhaps Arrays
should not. They really are different beasts. I also see why it was done
the way it was in the first place!
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
NOAA/OR&R/HAZMAT (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...
|
|
From: Alexander S. <a.s...@gm...> - 2002-06-18 22:22:09
|
Chris Barker <Chr...@no...> writes:
> My conclusion is that nested lists and Arrays simply are different
> beasts so we can't expect complete compatibility. I'm also wondering why
> lists have that weird behavior of a single index returning a reference,
> and a slice returning a copy. Perhaps it has something to so with the
This is not weird at all. Slicing and single item indexing are different
conceptually and what I think you have in mind wouldn't really work. Think of
a real life container, like box with subcompartments. Obviously you should be
able to take out (or put in) an item from the box, which is what single
indexing does (and the item may happen to be another box). My understanding is
that you'd like the box to return copies of whatever was put into it on
indexing, rather than the real thing -- this would not only be
counterintuitive and inefficient, it also means that you could exclusively put
items with a __copy__-method in lists, which would rather limit their
usefulness.
Slicing on the other hand creates a whole new box but this box is filled with
(references to) the same items (a behavior for which a real life equivalent is
more difficult to find :) :
>>> l = 'foobar'
>>> l = ['foobar', 'barfoot']
>>> l2 = l[:]
>>> l2[0] is l[0]
1
Because the l and l2 are different boxes, however, assigning new items to l1
doesn't change l2 and vice versa.
It is true, however that the situation is somewhat different for arrays,
because "multidimensional" lists are just nested boxed, whereas
multidimensional arrays have a different structure. array[1] indexes some part
of itself according to its .shape (which can be modified, thus changing what
array[1] indexes, without modifying the actual array contents in memory),
whereas list[1] indexes some "real" object.
This may mean that the best behavior for ``array[0]`` would be to return a
copy and ``array[:]`` etc. what would be a "deep copy" if it where nested
lists. I think this is the behavior Paul Dubois MA currently has.
> auto-resizing of lists. That being said, I still like the idea of slices
> producing copies, so:
>
> > 1) array
> An Array like we have now, but slice-is-copy
> semantics.
>
> > 2) array[0]
> An Array of rank one less than array, sharing data with array
>
> > 3) array.view
> An object that can do nothing but create other Arrays that share data
> with array. I don't know if is possible but I'd be just as happy if
> array.view returned None, and array.view[slice] returned an Array that
No it is not possible.
> shared data with array. Perhaps there is some other notation that could
> do this.
>
> > 4) array.view[0]
> Same as 2)
I can't see why single-item indexing views would be needed at all if
``array[0]`` doesn't copy as you suggest above.
>
> To add a few:
>
> 5) array[0:1]
> An Array with a copy of the data in array[0]
(I suppose you'd also want array[0:1] and array[0] to have different shape?)
>
> 6) array.view[0:1]
> An Array sharing data with array
>
> As I write this, I am starting to think that this is all a bit strange.
> Even though lists treat slices and indexes differently, perhaps Arrays
> should not. They really are different beasts. I also see why it was done
Yes, arrays and lists are indeed different beasts and a different indexing
behavior (creating copies) for arrays might well be preferable (since array
indexing doesn't refer to "real" objects).
> the way it was in the first place!
>
> -Chris
alex
--
Alexander Schmolck Postgraduate Research Student
Department of Computer Science
University of Exeter
A.S...@gm... http://www.dcs.ex.ac.uk/people/aschmolc/
|
|
From: Chris B. <Chr...@no...> - 2002-06-14 23:19:22
|
Konrad Hinsen wrote:
> Not necessarily. We could decide that
>
> array.view
>
> is a view of the full array object, and that slicing views returns
> subviews.
Please don't!! Having two types of arrays around in a single program
that have the same behaviour except when they are sliced is begging for
confusion and hard to find bugs.
I agree with Perry, that I occasionaly use the view behaviour of
slicing, and it is very usefull when I do, but most of the time I would
be happier with copy symantics. All I want is a way to get at a view of
part of an array, I don't want two different kinds of array around with
different slicing behaviour.
> My main objection to changing the slicing behaviour is, like with some
> other proposed changes, compatibility.
The switch from Numeric to Numarray is a substantial change. I think we
should view it like the mythical Py3k: an oportunity to make
incompatible changes that will really make it better.
By the way, as an old MATLAB user, I have to say that being able to get
views from a slice is one behaviour of NumPy that I really appreciate,
even though I only need it occasionally. MATLAB, howver is a whole
different ball of wax in a lot of ways. There has been a lot of
discussion about the copy on demand idea in MATLAB, but that is
primarily useful because MATLAB has call by value function semantics, so
without copy on demand, you would be making copies of large arrays
passed to functions that weren't even going to change them. I don't
think MATLAB impliments copy on demand for slices anyway, but I could be
wrong there.
Oh, and no function (ie ravel() ) should return a view in some cases,
and a copy in others, that is just asking for bugs!
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
NOAA/OR&R/HAZMAT (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...
|
|
From: Scott R. <ra...@ph...> - 2002-06-14 23:26:04
|
I was going to write an almost identical email, but Chris saved me the trouble. These are my feelings as well. Scott On June 14, 2002 07:01 pm, Chris Barker wrote: > Konrad Hinsen wrote: > > Not necessarily. We could decide that > > > > array.view > > > > is a view of the full array object, and that slicing views returns > > subviews. > > Please don't!! Having two types of arrays around in a single program > that have the same behaviour except when they are sliced is begging for > confusion and hard to find bugs. > > I agree with Perry, that I occasionaly use the view behaviour of > slicing, and it is very usefull when I do, but most of the time I would > be happier with copy symantics. All I want is a way to get at a view of > part of an array, I don't want two different kinds of array around with > different slicing behaviour. > > > My main objection to changing the slicing behaviour is, like with some > > other proposed changes, compatibility. > > The switch from Numeric to Numarray is a substantial change. I think we > should view it like the mythical Py3k: an oportunity to make > incompatible changes that will really make it better. > > By the way, as an old MATLAB user, I have to say that being able to get > views from a slice is one behaviour of NumPy that I really appreciate, > even though I only need it occasionally. MATLAB, howver is a whole > different ball of wax in a lot of ways. There has been a lot of > discussion about the copy on demand idea in MATLAB, but that is > primarily useful because MATLAB has call by value function semantics, so > without copy on demand, you would be making copies of large arrays > passed to functions that weren't even going to change them. I don't > think MATLAB impliments copy on demand for slices anyway, but I could be > wrong there. > > Oh, and no function (ie ravel() ) should return a view in some cases, > and a copy in others, that is just asking for bugs! > > -Chris -- Scott M. Ransom Address: McGill Univ. Physics Dept. Phone: (514) 398-6492 3600 University St., Rm 338 email: ra...@ph... Montreal, QC Canada H3A 2T8 GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 |
|
From: Alexander S. <a.s...@gm...> - 2002-06-12 22:50:02
|
Rick White <rl...@st...> writes: > Here is what I see as the fundamental problem with implementing slicing > in numarray using copy-on-demand instead views. > > Copy-on-demand requires the maintenance of a global list of all the > active views associated with a particular array buffer. Here is a > simple example: > > >>> a = zeros((5000,5000)) > >>> b = a[49:51,50] > >>> c = a[51:53,50] > >>> a[50,50] = 1 > > The assignment to a[50,50] must trigger a copy of the array b; > otherwise b also changes. On the other hand, array c does not need to > be copied since its view does not include element 50,50. You could > instead copy the array a -- but that means copying a 100 Mbyte array > while leaving the original around (since b and c are still using it) -- > not a good idea! Sure, if one wants do perform only the *minimum* amount of copying, things can get rather tricky, but wouldn't it be satisfactory for most cases if attempted modification of the original triggered the delayed copying of the "views" (lazy copies)? In those cases were it isn't satisfactory the user could still explicitly create real (i.e. alias-only) views. > > The bookkeeping can get pretty messy (if you care about memory usage, > which we definitely do). Consider this case: > > >>> a = zeros((5000,5000)) > >>> b = a[0:-10,0:-10] > >>> c = a[49:51,50] > >>> del a > >>> b[50,50] = 1 > > Now what happens? Either we can copy the array for b (which means two ``b`` and ``c`` are copied and then ``a`` is deleted. What does numarray currently keep of a if I do something like the above or: >>> b = a.flat[::-10000] >>> del a ? > copies of the huge (5000,5000) array exist, one used by c and the new > version used by b), or we can be clever and copy c instead. > > Even keeping track of the views associated with a buffer doesn't solve > the problem of an array that is passed to a C extension and is modified > in place. It would seem that passing an array into a C extension would > always require all the associated views to be turned into copies. > Otherwise we can't guarantee that views won't be modifed. Yes -- but only if the C extension is destructive. In that case the user might well be making a mistake in current Numeric if he has views and doesn't want them to be modified by the operation (of course he might know that the inplace operation does not affect the view(s) -- but wouldn't such cases be rather rare?). If he *does* want the views to be modified, he would obviously have to explictly specify them as such in a copy-on-demand scheme and in the other case he has been most likely been prevented from making an error (and can still explicitly use real views if he knows that the inplace operation on the original will not have undesired effects on the "views"). > > This kind of state information with side effects leads to a system that > is hard to develop, hard to debug, and really messes up the behavior of > the program (IMHO). It is *highly* desirable to avoid it if possible. Sure, copy-on-demand is an optimization and optmizations always mess up things. On the other hand, some optimizations also make "nicer" (e.g. less error-prone) semantics computationally viable, so it's often a question between ease and clarity of the implementation vs. ease and clarity of code that uses it. I'm not denying that too much complexity in the implementation also aversely affects users in the form of bugs and that in the particular case of delayed copying the user can also be affected directly by more difficult to understand ressource usage behavior (e.g. a[0] = 1 triggering a monstrous copying operation). Just out of curiosity, has someone already asked the octave people how much trouble it has caused them to implement copy on demand and whether matlab/octave users in practice do experience difficulties because of the more harder to predict runtime behavior (I think, like matlab, octave does copy-on-demand)? > > This is not to deny that copy-on-demand (with explicit views available > on request) would have some desirable advantages for the behavior of > the system. But we've worried these issues to death, and in the end > were convinced that slices == views provided the best compromise > between the desired behavior and a clean implementation. If the implementing copy-on-demand is too difficult and the resulting code would be too messy then this is certainly a valid reason to compromise on the current slicing behavior (especially since people like me who'd like to see copy-on-demand are unlikely to volunteer to implement it :) > Rick > > ------------------------------------------------------------------ > Richard L. White rl...@st... http://sundog.stsci.edu/rick/ > Space Telescope Science Institute > Baltimore, MD > > alex -- Alexander Schmolck Postgraduate Research Student Department of Computer Science University of Exeter A.S...@gm... http://www.dcs.ex.ac.uk/people/aschmolc/ |
|
From: Perry G. <pe...@st...> - 2002-06-13 21:16:25
|
> > Copy-on-demand requires the maintenance of a global list of all the > > active views associated with a particular array buffer. Here is a > > simple example: > > > > >>> a = zeros((5000,5000)) > > >>> b = a[49:51,50] > > >>> c = a[51:53,50] > > >>> a[50,50] = 1 > > > > The assignment to a[50,50] must trigger a copy of the array b; > > otherwise b also changes. On the other hand, array c does not need to > > be copied since its view does not include element 50,50. You could > > instead copy the array a -- but that means copying a 100 Mbyte array > > while leaving the original around (since b and c are still using it) -- > > not a good idea! > > Sure, if one wants do perform only the *minimum* amount of > copying, things can > get rather tricky, but wouldn't it be satisfactory for most cases > if attempted > modification of the original triggered the delayed copying of the "views" > (lazy copies)? In those cases were it isn't satisfactory the > user could still > explicitly create real (i.e. alias-only) views. > I'm not sure what you mean. Are you saying that if anything in the buffer changes, force all views of the buffer to generate copies (rather than try to determine if the change affected only selected views)? If so, yes, it is easier, but it still is a non-trivial capability to implement. > > > > The bookkeeping can get pretty messy (if you care about memory usage, > > which we definitely do). Consider this case: > > > > >>> a = zeros((5000,5000)) > > >>> b = a[0:-10,0:-10] > > >>> c = a[49:51,50] > > >>> del a > > >>> b[50,50] = 1 > > > > Now what happens? Either we can copy the array for b (which means two > > ``b`` and ``c`` are copied and then ``a`` is deleted. > > What does numarray currently keep of a if I do something like the > above or: > > >>> b = a.flat[::-10000] > >>> del a > > ? > The whole buffer remains in both cases. > > copies of the huge (5000,5000) array exist, one used by c and the new > > version used by b), or we can be clever and copy c instead. > > > > Even keeping track of the views associated with a buffer doesn't solve > > the problem of an array that is passed to a C extension and is modified > > in place. It would seem that passing an array into a C extension would > > always require all the associated views to be turned into copies. > > Otherwise we can't guarantee that views won't be modifed. > > Yes -- but only if the C extension is destructive. In that case > the user might > well be making a mistake in current Numeric if he has views and > doesn't want > them to be modified by the operation (of course he might know > that the inplace > operation does not affect the view(s) -- but wouldn't such cases be rather > rare?). If he *does* want the views to be modified, he would > obviously have to > explictly specify them as such in a copy-on-demand scheme and in the other > case he has been most likely been prevented from making an error (and can > still explicitly use real views if he knows that the inplace > operation on the > original will not have undesired effects on the "views"). > If the point is that views are susceptible to unexpected changes made in place by a C extension, yes, certainly (just as they are for changes made in place in Python). But I'm not sure what that has to do with the implied copy (even if delayed) being broken by extensions written in C. Promising a copy, and not honoring it is not the same as not promising it in the first place. But I may be misunderstanding your point. Perry |
|
From: Alexander S. <a.s...@gm...> - 2002-06-14 00:35:14
|
"Perry Greenfield" <pe...@st...> writes:
> I'm not sure what you mean. Are you saying that if anything in the
> buffer changes, force all views of the buffer to generate copies
> (rather than try to determine if the change affected only selected
Yes (I suspect that this will be be sufficient in practice).
> views)? If so, yes, it is easier, but it still is a non-trivial
> capability to implement.
Sure. But since copy-on-demand is only an optimization and as such doesn't
affect the semantics, it could also be implemented at a later point if the
resources are currently not available. I have little doubt that someone will
eventually add copy-on-demand, if the option is kept open and in the meantime
one could still get all the performance (and alias behavior) of the current
implementation by explicitly using ``.view`` (or ``.sub`` if you prefer) to
create aliases.
I'm becoming increasingly convinced (see below) that copy-slicing-semantics
are much to be preferred as the default, so given the above I don't think that
performance concerns should sway one towards alias-slicing, if enough people
feel that copy semantics as such are preferable.
> > >
> > > The bookkeeping can get pretty messy (if you care about memory usage,
> > > which we definitely do). Consider this case:
> > >
> > > >>> a = zeros((5000,5000))
> > > >>> b = a[0:-10,0:-10]
> > > >>> c = a[49:51,50]
> > > >>> del a
> > > >>> b[50,50] = 1
> > >
> > > Now what happens? Either we can copy the array for b (which means two
> >
> > ``b`` and ``c`` are copied and then ``a`` is deleted.
> >
> > What does numarray currently keep of a if I do something like the
> > above or:
> >
> > >>> b = a.flat[::-10000]
> > >>> del a
> >
> > ?
> >
> The whole buffer remains in both cases.
OK, so this is then a nice example where even eager copy slicing behavior
would be *significantly* more efficient than the current aliasing behavior --
so copy-on-demand would then on the whole seem to be not just nearly equally
but *more* efficient than alias slicing.
And as far as difficult to understand runtime behavior is concerned, the extra
~100MB useless baggage carried around by b (second case) are, I'd venture to
suspect, less than obvious to the casual observer. In fact I remember one of
my fellow phd-students having significant problems with mysterious memory
consumption (a couple of arrays taking up more than 1GB rather than a few
hundred MB) -- maybe something like the above was involved. That
``A = A[::-1]`` doesn't work (as pointed out by Paul Barrett) will also come as a
surprise to most people.
If I understand all this correctly, I consider it a rather strong case against
alias slicing as default behavior.
> > > Even keeping track of the views associated with a buffer doesn't solve
> > > the problem of an array that is passed to a C extension and is modified
> > > in place. It would seem that passing an array into a C extension would
> > > always require all the associated views to be turned into copies.
> > > Otherwise we can't guarantee that views won't be modifed.
> >
> > Yes -- but only if the C extension is destructive. In that case
> > the user might
> > well be making a mistake in current Numeric if he has views and
> > doesn't want
> > them to be modified by the operation (of course he might know
> > that the inplace
> > operation does not affect the view(s) -- but wouldn't such cases be rather
> > rare?). If he *does* want the views to be modified, he would
> > obviously have to
> > explictly specify them as such in a copy-on-demand scheme and in the other
> > case he has been most likely been prevented from making an error (and can
> > still explicitly use real views if he knows that the inplace
> > operation on the
> > original will not have undesired effects on the "views").
> >
> If the point is that views are susceptible to unexpected changes
> made in place by a C extension, yes, certainly (just as they
> are for changes made in place in Python). But I'm not sure what
> that has to do with the implied copy (even if delayed) being
> broken by extensions written in C. Promising a copy, and not
> honoring it is not the same as not promising it in the first
> place. But I may be misunderstanding your point.
>
OK, I'll try again, hopefully this is clearer.
In a sentence: I don't see any problems with C extensions in particular that
would arise from copy-on-demand (I might well be overlooking something,
though).
Rick was saying that passing an array to a C extension that performs an
inplace operation on it means that all copies of all its (lazy) views must be
performed.
My point was that this is correct, but I can't see any problem with that,
neither from the point of extension writer, nor from the point of performance
nor from the point of the user, nor indeed from the point of the numarray
implementors (obviously the copy-on-demand scheme *as such* will be an
effort).
All that is needed is a separate interface for (the minority of) C extensions
that destructively modify their arguments (they only need to call some
function `actualize_views(the_array_or_view)` or whatever at the start -- this
function will obviously be necessary regardless of the C extensions). So
nothing will break, the promises are kept and no extra work.
It won't be any slower than what would happen with current Numeric, either,
because either the (Numeric) user intended his (aliased) views to modified as
well or it was a bug. If he intended the views to be modified, he would
explicitly use alias-views under the new scheme and everything would behave
exactly the same.
alex
--
Alexander Schmolck Postgraduate Research Student
Department of Computer Science
University of Exeter
A.S...@gm... http://www.dcs.ex.ac.uk/people/aschmolc/
|
|
From: Konrad H. <hi...@cn...> - 2002-06-17 08:45:49
|
> Konrad Hinsen <hi...@cn...> writes: > > [did you mean this to be off-list? If not, please just forward it to the > list.] No, I sent the mail to the list as well, but one out of three mails I send to the list never arrive there at first try... In this case, the copy sent to myself got lost as well, so I don't have any copy left, sorry. > > <rant> > > > > I don't know about the others out there, but I have 30000 lines of > > published Python code plus a lot of unpublished code (scripts), all of > > which use NumPy arrays almost everywhere. There are also a few places > > where views are created intentionally, which are then passed around to > > other code and can end up anywhere. The time required to update all > > that to new slicing semantics would be enormous, and I don't see how I > > could justify it to myself or to my employer. I'd also have to stop > > advertising Python as a time-efficient development tool. > > I sympathize with this view. However, I think the solution to this problem > should be a compatibility wrapper rather than a design compromise. > > There are at least 2 reasons why: > > 1. Numarray has quite a few incompatibilities to Numeric anyway, so even > without this change you'd be forced to rewrite all or most of those scripts The question is how much effort it is to update code. If it is easy, most people will do it sooner or later. If it is difficult, they won't. And that will lead to a split in the user community, which I think is highly detrimental to the further development of NumPy and Numarray. A compatibility wrapper won't change this. Assume that I have tons of code that I can't update because it's too much effort. Instead I use the compatbility wrapper. When I add a line or a function to that code, it will of course stick to the old conventions. When I add a new module, I will also prefer the old conventions, for consistency. And other people working with the code will pick up the old conventions as well. At the same time, other people will use the new conventions. There will be two parts of the community that cannot easily read each other's code. So unless we can reach a concensus that will guarantee that 90% of existing code will be adapted to the new interfaces, there will be a split. > (or use the wrapper), but none of the incompatibilities I'm currently aware > of would, in my eyes, buy one as much as introducing copy-indexing > semantics would. So if things get broken anyway, one might as well take I agree, but it also comes at the highest cost. There is absolute no way to identify automatically the code that needs to be adapted, and there is no run-time error message in case of failure - just a wrong result. None of the other proposed changes is as risky as this one. > this step (especially since intentional views are, on the whole, used > rather sparingly -- although tracking down these uses in retrospect might > admittedly be unpleasant). It is not merely unpleasant, the cost is simply prohibitive. > 2. Numarray is supposed to be incorporated into the core. Compromising the > consistency of core python (and code that depends on it) is in my eyes > worse than compromising code written for Numeric. I don't see view behaviour as inconsistent with Python. Python has one mutable sequence type, the list, with copy behaviour. One type is hardly enough to establish a rule. > As a third reason I could claim that there is some hope of a much more > widespread adoption of Numeric/numarray as an alternative to matlab etc. in > the next couple of years, so that it might be wise to fix things now, but I'd > understand if you'd remain unimpressed by that :) I'd like to see any supporting evidence. I think this argument is based on the reasoning "I would prefer it to be this way, so many others would certainly also prefer it, so they would start using NumPy if only these changes were made." This is not how decision processes work in real life. On the contrary, people might look at the history of NumPy and decide that it is too unreliable to base a serious project on - if they changed the interface once, they might do it again. This is a particularly important aspect in the OpenSource universe, where there are no contracts that promise anything. If you want people to use your code, you have to demonstrate that it is reliable, and that applies to both the code and the interfaces. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- |
|
From: Alexander S. <a.s...@gm...> - 2002-06-17 15:11:15
|
Konrad Hinsen <hi...@cn...> writes:
[Konrad wants to keep alias-slicing behavior for backward-compatibility]
> > I sympathize with this view. However, I think the solution to this problem
> > should be a compatibility wrapper rather than a design compromise.
> >
> > There are at least 2 reasons why:
> >
> > 1. Numarray has quite a few incompatibilities to Numeric anyway, so even
> > without this change you'd be forced to rewrite all or most of those scripts
>
> The question is how much effort it is to update code. If it is easy,
> most people will do it sooner or later. If it is difficult, they won't.
> And that will lead to a split in the user community, which I think
> is highly detrimental to the further development of NumPy and Numarray.
I agree that avoiding a split of the Numeric user community is a crucial issue
and that efforts have to be taken to make transition painless enough to happen
(in most cases; maybe it needs to be even 90% or more as you say).
>
> A compatibility wrapper won't change this. Assume that I have tons of
> code that I can't update because it's too much effort. Instead I use
> the compatbility wrapper. When I add a line or a function to that
> code, it will of course stick to the old conventions. When I add a new
> module, I will also prefer the old conventions, for consistency. And
> other people working with the code will pick up the old conventions as
> well. At the same time, other people will use the new conventions.
> There will be two parts of the community that cannot easily read each
> other's code.
I don't think the situation is quite so bleak. Yes, library code should be
converted, and although a compatibility wrapper might be helpful in the
process, I agree that it isn't a full solution for the reasons you cite above.
But there is plenty of code that is mainly used internally and no longer
changes (much), for which I think a compatibility wrapper is a fine solution
(and might be preferable to conversion, even if it involves little effort). If
I had some matlab (or C) code that fulfills similar criteria, I'd also rather
wrap it somehow rather than to convert it to python.
>
> So unless we can reach a concensus that will guarantee that 90% of
> existing code will be adapted to the new interfaces, there will be a
> split.
>
> > (or use the wrapper), but none of the incompatibilities I'm currently aware
> > of would, in my eyes, buy one as much as introducing copy-indexing
> > semantics would. So if things get broken anyway, one might as well take
>
> I agree, but it also comes at the highest cost. There is absolute no
> way to identify automatically the code that needs to be adapted, and
> there is no run-time error message in case of failure - just a wrong
> result. None of the other proposed changes is as risky as this one.
Wouldn't an (almost) automatic solution be to simply replace (almost) all
instances of a[b:c] with a.view[b:c] in your legacy code? Even for unusual
cases (like if you heavily mix arrays and lists) you could still autoconvert
by inserting ``if type(foo) == ArrayType:...``, although this would admittedly
be rather messy. The unnecessary ``.view``s can be eliminated over time and
even if they aren't, no one would have to learn or switch between two
libraries.
>
> > this step (especially since intentional views are, on the whole, used
> > rather sparingly -- although tracking down these uses in retrospect might
> > admittedly be unpleasant).
>
> It is not merely unpleasant, the cost is simply prohibitive.
See above. I personally hope that even without resorting to something like the
above, converting my code to copy behavior wouldn't be too much of an effort,
but my code-base is much smaller than yours and I can't currently recall more
than one case of intended aliasing that would require a couple of changes and
my estimate might also prove quite wrong. I have no idea which scenario is
typical.
>
> > 2. Numarray is supposed to be incorporated into the core. Compromising the
> > consistency of core python (and code that depends on it) is in my eyes
> > worse than compromising code written for Numeric.
>
> I don't see view behaviour as inconsistent with Python. Python has one
> mutable sequence type, the list, with copy behaviour. One type is
> hardly enough to establish a rule.
Well, AFAIK there are actually three mutable sequence types in python core and
all have copy-slicing behavior: list, UserList and array:
>>> import array
>>> aa = array.array('d', [1.0, 2.0, 3.0, 4.0, 5.0, 6.0])
>>> bb = aa[:]
>>> bb is aa
0
I would suppose that in the grand scheme of things numarray.array is intended
as an eventual replacement for array.array, or not?
Furthermore list is such a fundamental data type in python that I think it is
actually enough to establish a rule (if the vast majority of 3rd party modules
sequence types don't have the same semantics, I'd regard it as a strong
argument for your position, but I haven't checked).
>
> > As a third reason I could claim that there is some hope of a much more
> > widespread adoption of Numeric/numarray as an alternative to matlab etc. in
> > the next couple of years, so that it might be wise to fix things now, but I'd
> > understand if you'd remain unimpressed by that :)
>
> I'd like to see any supporting evidence. I think this argument is
> based on the reasoning "I would prefer it to be this way, so many
> others would certainly also prefer it, so they would start using NumPy
> if only these changes were made." This is not how decision processes
> work in real life.
Sure, but I didn't try to imply this causality anyway:) My argument wasn't so
much "lets make it really good (where good is what *I* say) then loads of
people will adopt it", it was more: "Numeric has a good chance to grow
considerably in popularity over the next years, so it will be much easier to
fix things now than later" (for slicing behavior, now is likely to be the last
chance).
The fact that matlab users are used to copy-on-demand and the fact that many
people, (including you if I understand you correctly) think that copy-slicing
semantics as such (without backward compatibility concerns) are preferable,
might have a small influence on people's decision to adopt Numeric, but I
perfectly agree that this influence will be minor compared to other issues.
>
> On the contrary, people might look at the history of NumPy and decide
> that it is too unreliable to base a serious project on - if they
> changed the interface once, they might do it again. This is a
> particularly important aspect in the OpenSource universe, where there
> are no contracts that promise anything. If you want people to use your
I don't think matlab or similar alternatives make legally binding promises
about backwards compatibility, or do they? It guess it is actually more
difficult to *force* incompatible changes on people with an open source
project than with commercial software, but I agree that splitting or
lighthearted sacrifices of backwards compatibility are more of a temptation
with open source, for one thing because there are usually less financial
stakes involved for the authors.
> code, you have to demonstrate that it is reliable, and that applies to
> both the code and the interfaces.
Yes, this is very important and I very much appreciate that you stress these
and similar points in your postings.
But reliability to me also includes the ability for growth -- I not only want
my old code to work in a couple of years, I also want the tool I wrote it in
to remain competitive and this can conflict with backwards-compatibility. I
like the balance python strikes here so far -- the language has improved
significantly (and in my eyes has remained superior to newer competitors such
as ruby) but at the same time for me and most other people transitions between
versions haven't caused too much trouble. This increases the value of my
code-base to me: I can assume that it will still work (or be adapted without
too much effort) in years to come and yet be written in an excellent language
for the job.
Striking this balance is however quite difficult (as can be seen by the heated
discussions in c.l.p), so getting it right will most likely involve
considerable effort (and controversy) within the Numeric community.
alex
--
Alexander Schmolck Postgraduate Research Student
Department of Computer Science
University of Exeter
A.S...@gm... http://www.dcs.ex.ac.uk/people/aschmolc/
|
|
From: Konrad H. <hi...@cn...> - 2002-06-07 16:00:57
|
> For binary operations between a Python scalar and array, there is > no coercion performed on the array type if the scalar is of the > same kind as the array (but not same size or precision). For example > (assuming ints happen to be 32 bit in this case) That solves one problem and creates another... Two, in fact. One is the inconsistency problem: Python type coercion always promotes "smaller" to "bigger" types, it would be good to make no exceptions from this rule. Besides, there are still situations in which types, ranks, and indexing operations depend on each other in a strange way. With a = array([1., 2.], Float) b = array([3., 4.], Float32) the result of a*b is of type Float, whereas a[0]*b is of type Float32 - if and only if a has rank 1. > (Yes, it would be easiest to deal with if Python had all these types, > but I think that will never happen, nor should it happen.) Python doesn't need to have them as standard types, an add-on package can provide them as well. NumPy seems like the obvious one. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X