numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] copy on demand

[Scott R. <ra...@ph...>]

On June 17, 2002 04:57 am, Konrad Hinsen wrote:
> > > Then the only solution I see is the current one: default behaviour is
> > > view, and when you want a copy yoy copy explicitly. The inverse is not
> > > possible, once you made a copy you can't make it behave like a view
> > > anymore.
> >
> > I don't think it is necessary to create the other object _from_
> > the default one.  You could have copy behavior be the default,
> > and if you want a view of some array you simply request one
> > explicitly with .view, .sub, or whatever.
>
> Let's make this explicit. Given the following four expressions,
>
> 1) array
> 2) array[0]
> 3) array.view
> 4) array.view[0]
>
> what would the types of each of these objects be according to your
> proposal? What would the indexing behaviour of those types be?
> I don't see how you can avoid having either two types or two
> different behaviours within one type.

If we assume that a slice returns a copy _always_, then I agree that #4 in 
your list above would not give a user what they would expect:

array.view[0] would give the view of a copy of array[0], _not_ a view of 
array[0] which is probably what is wanted.  I _think_ that this could be 
fixed by making view (or something similar) an option of the slice rather 
than a method of the object.  For example (assuming that a is an array):

Expression:      Returns:             Slicing Behavior:
a or a[:]        Copy of all of a     Returns a copy of the sub-array
a[0]             Copy of a[0]         Returns a copy of the sub-array
a[:,view]        View of all of a     Returns a copy of the sub-array
a[0,view]        View of a[0]         Returns a copy of the sub-array

Notice that it is possible to return a copy of a sub-array from a view since 
you have access (through a pointer) to the original array data.

Scott

-- 
Scott M. Ransom              Address:  McGill Univ. Physics Dept.
Phone:  (514) 398-6492                 3600 University St., Rm 338
email:  ra...@ph...       Montreal, QC  Canada H3A 2T8 
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989

Re: [Numpy-discussion] copy on demand

[Alexander S. <a.s...@gm...>]

Konrad Hinsen <hi...@cn...> writes:

[Konrad wants to keep alias-slicing behavior for backward-compatibility]
> > I sympathize with this view. However, I think the solution to this problem
> > should be a compatibility wrapper rather than a design compromise.
> > 
> > There are at least 2 reasons why:
> > 
> > 1. Numarray has quite a few incompatibilities to Numeric anyway, so even
> >    without this change you'd be forced to rewrite all or most of those scripts
> 
> The question is how much effort it is to update code. If it is easy,
> most people will do it sooner or later. If it is difficult, they won't.
> And that will lead to a split in the user community, which I think
> is highly detrimental to the further development of NumPy and Numarray.

I agree that avoiding a split of the Numeric user community is a crucial issue
and that efforts have to be taken to make transition painless enough to happen
(in most cases; maybe it needs to be even 90% or more as you say).

> 
> A compatibility wrapper won't change this. Assume that I have tons of
> code that I can't update because it's too much effort. Instead I use
> the compatbility wrapper. When I add a line or a function to that
> code, it will of course stick to the old conventions. When I add a new
> module, I will also prefer the old conventions, for consistency. And
> other people working with the code will pick up the old conventions as
> well. At the same time, other people will use the new conventions.
> There will be two parts of the community that cannot easily read each
> other's code.

I don't think the situation is quite so bleak. Yes, library code should be
converted, and although a compatibility wrapper might be helpful in the
process, I agree that it isn't a full solution for the reasons you cite above.

But there is plenty of code that is mainly used internally and no longer
changes (much), for which I think a compatibility wrapper is a fine solution
(and might be preferable to conversion, even if it involves little effort). If
I had some matlab (or C) code that fulfills similar criteria, I'd also rather
wrap it somehow rather than to convert it to python.

> 
> So unless we can reach a concensus that will guarantee that 90% of
> existing code will be adapted to the new interfaces, there will be a
> split.
> 
> >    (or use the wrapper), but none of the incompatibilities I'm currently aware
> >    of would, in my eyes, buy one as much as introducing copy-indexing
> >    semantics would. So if things get broken anyway, one might as well take
> 
> I agree, but it also comes at the highest cost. There is absolute no
> way to identify automatically the code that needs to be adapted, and
> there is no run-time error message in case of failure - just a wrong
> result. None of the other proposed changes is as risky as this one.

Wouldn't an (almost) automatic solution be to simply replace (almost) all
instances of a[b:c] with a.view[b:c] in your legacy code? Even for unusual
cases (like if you heavily mix arrays and lists) you could still autoconvert
by inserting ``if type(foo) == ArrayType:...``, although this would admittedly
be rather messy.  The unnecessary ``.view``s can be eliminated over time and
even if they aren't, no one would have to learn or switch between two
libraries.

> 
> >    this step (especially since intentional views are, on the whole, used
> >    rather sparingly -- although tracking down these uses in retrospect might
> >    admittedly be unpleasant).
> 
> It is not merely unpleasant, the cost is simply prohibitive.

See above. I personally hope that even without resorting to something like the
above, converting my code to copy behavior wouldn't be too much of an effort,
but my code-base is much smaller than yours and I can't currently recall more
than one case of intended aliasing that would require a couple of changes and
my estimate might also prove quite wrong. I have no idea which scenario is
typical.

> 
> > 2. Numarray is supposed to be incorporated into the core. Compromising the
> >    consistency of core python (and code that depends on it) is in my eyes
> >    worse than compromising code written for Numeric.
> 
> I don't see view behaviour as inconsistent with Python. Python has one
> mutable sequence type, the list, with copy behaviour. One type is
> hardly enough to establish a rule.

Well, AFAIK there are actually three mutable sequence types in python core and
all have copy-slicing behavior: list, UserList and array:

    >>> import array
    >>> aa = array.array('d', [1.0, 2.0, 3.0, 4.0, 5.0, 6.0])
    >>> bb = aa[:]
    >>> bb is aa
    0

I would suppose that in the grand scheme of things numarray.array is intended
as an eventual replacement for array.array, or not?

Furthermore list is such a fundamental data type in python that I think it is
actually enough to establish a rule (if the vast majority of 3rd party modules
sequence types don't have the same semantics, I'd regard it as a strong
argument for your position, but I haven't checked).

> 
> > As a third reason I could claim that there is some hope of a much more
> > widespread adoption of Numeric/numarray as an alternative to matlab etc. in
> > the next couple of years, so that it might be wise to fix things now, but I'd
> > understand if you'd remain unimpressed by that :)
> 
> I'd like to see any supporting evidence. I think this argument is
> based on the reasoning "I would prefer it to be this way, so many
> others would certainly also prefer it, so they would start using NumPy
> if only these changes were made." This is not how decision processes
> work in real life.

Sure, but I didn't try to imply this causality anyway:) My argument wasn't so
much "lets make it really good (where good is what *I* say) then loads of
people will adopt it", it was more: "Numeric has a good chance to grow
considerably in popularity over the next years, so it will be much easier to
fix things now than later" (for slicing behavior, now is likely to be the last
chance).

The fact that matlab users are used to copy-on-demand and the fact that many
people, (including you if I understand you correctly) think that copy-slicing
semantics as such (without backward compatibility concerns) are preferable,
might have a small influence on people's decision to adopt Numeric, but I
perfectly agree that this influence will be minor compared to other issues.

> 
> On the contrary, people might look at the history of NumPy and decide
> that it is too unreliable to base a serious project on - if they
> changed the interface once, they might do it again. This is a
> particularly important aspect in the OpenSource universe, where there
> are no contracts that promise anything. If you want people to use your

I don't think matlab or similar alternatives make legally binding promises
about backwards compatibility, or do they? It guess it is actually more
difficult to *force* incompatible changes on people with an open source
project than with commercial software, but I agree that splitting or
lighthearted sacrifices of backwards compatibility are more of a temptation
with open source, for one thing because there are usually less financial
stakes involved for the authors.

> code, you have to demonstrate that it is reliable, and that applies to
> both the code and the interfaces.

Yes, this is very important and I very much appreciate that you stress these
and similar points in your postings.

But reliability to me also includes the ability for growth -- I not only want
my old code to work in a couple of years, I also want the tool I wrote it in
to remain competitive and this can conflict with backwards-compatibility. I
like the balance python strikes here so far -- the language has improved
significantly (and in my eyes has remained superior to newer competitors such
as ruby) but at the same time for me and most other people transitions between
versions haven't caused too much trouble. This increases the value of my
code-base to me: I can assume that it will still work (or be adapted without
too much effort) in years to come and yet be written in an excellent language
for the job.

Striking this balance is however quite difficult (as can be seen by the heated
discussions in c.l.p), so getting it right will most likely involve
considerable effort (and controversy) within the Numeric community.

alex

-- 
Alexander Schmolck     Postgraduate Research Student
                       Department of Computer Science
                       University of Exeter
A.S...@gm...     http://www.dcs.ex.ac.uk/people/aschmolc/

Re: [Numpy-discussion] copy on demand

[Konrad H. <hi...@cn...>]

> > Then the only solution I see is the current one: default behaviour is
> > view, and when you want a copy yoy copy explicitly. The inverse is not
> > possible, once you made a copy you can't make it behave like a view
> > anymore.
> 
> I don't think it is necessary to create the other object _from_
> the default one.  You could have copy behavior be the default,
> and if you want a view of some array you simply request one
> explicitly with .view, .sub, or whatever.

Let's make this explicit. Given the following four expressions,

1) array
2) array[0]
3) array.view
4) array.view[0]

what would the types of each of these objects be according to your
proposal? What would the indexing behaviour of those types be?
I don't see how you can avoid having either two types or two
different behaviours within one type.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] copy on demand

[Konrad H. <hi...@cn...>]

> Konrad Hinsen <hi...@cn...> writes:
> 
> [did you mean this to be off-list? If not, please just forward it to the
> list.]

No, I sent the mail to the list as well, but one out of three mails I
send to the list never arrive there at first try... In this case, the
copy sent to myself got lost as well, so I don't have any copy left,
sorry.

> > <rant>
> > 
> > I don't know about the others out there, but I have 30000 lines of
> > published Python code plus a lot of unpublished code (scripts), all of
> > which use NumPy arrays almost everywhere. There are also a few places
> > where views are created intentionally, which are then passed around to
> > other code and can end up anywhere. The time required to update all
> > that to new slicing semantics would be enormous, and I don't see how I
> > could justify it to myself or to my employer. I'd also have to stop
> > advertising Python as a time-efficient development tool.
> 
> I sympathize with this view. However, I think the solution to this problem
> should be a compatibility wrapper rather than a design compromise.
> 
> There are at least 2 reasons why:
> 
> 1. Numarray has quite a few incompatibilities to Numeric anyway, so even
>    without this change you'd be forced to rewrite all or most of those scripts

The question is how much effort it is to update code. If it is easy,
most people will do it sooner or later. If it is difficult, they won't.
And that will lead to a split in the user community, which I think
is highly detrimental to the further development of NumPy and Numarray.

A compatibility wrapper won't change this. Assume that I have tons of
code that I can't update because it's too much effort. Instead I use
the compatbility wrapper. When I add a line or a function to that
code, it will of course stick to the old conventions. When I add a new
module, I will also prefer the old conventions, for consistency. And
other people working with the code will pick up the old conventions as
well. At the same time, other people will use the new conventions.
There will be two parts of the community that cannot easily read each
other's code.

So unless we can reach a concensus that will guarantee that 90% of
existing code will be adapted to the new interfaces, there will be a
split.

>    (or use the wrapper), but none of the incompatibilities I'm currently aware
>    of would, in my eyes, buy one as much as introducing copy-indexing
>    semantics would. So if things get broken anyway, one might as well take

I agree, but it also comes at the highest cost. There is absolute no
way to identify automatically the code that needs to be adapted, and
there is no run-time error message in case of failure - just a wrong
result. None of the other proposed changes is as risky as this one.

>    this step (especially since intentional views are, on the whole, used
>    rather sparingly -- although tracking down these uses in retrospect might
>    admittedly be unpleasant).

It is not merely unpleasant, the cost is simply prohibitive.

> 2. Numarray is supposed to be incorporated into the core. Compromising the
>    consistency of core python (and code that depends on it) is in my eyes
>    worse than compromising code written for Numeric.

I don't see view behaviour as inconsistent with Python. Python has one
mutable sequence type, the list, with copy behaviour. One type is
hardly enough to establish a rule.

> As a third reason I could claim that there is some hope of a much more
> widespread adoption of Numeric/numarray as an alternative to matlab etc. in
> the next couple of years, so that it might be wise to fix things now, but I'd
> understand if you'd remain unimpressed by that :)

I'd like to see any supporting evidence. I think this argument is
based on the reasoning "I would prefer it to be this way, so many
others would certainly also prefer it, so they would start using NumPy
if only these changes were made." This is not how decision processes
work in real life.

On the contrary, people might look at the history of NumPy and decide
that it is too unreliable to base a serious project on - if they
changed the interface once, they might do it again. This is a
particularly important aspect in the OpenSource universe, where there
are no contracts that promise anything. If you want people to use your
code, you have to demonstrate that it is reliable, and that applies to
both the code and the interfaces.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] copy on demand

[Alexander S. <a.s...@gm...>]

"Perry Greenfield" <pe...@st...> writes:

> <Alexander Schmolck writes>:
>     <Perry Greenfield writes>:
> > > I guess that depends on what you mean by unnecessary copies.
> >
> > In most cases the array of which I desire a flattened representation is
> > contiguous (plus, I usually don't intend to modify it).
> > Consequently, in most
> > cases I don't want to any copies of it to be created (especially
> > not if it is
> > really large -- which is not seldom the case).
> >
> Numarray already returns a view of the array if it is contiguous.
> Copies are only produced if it is non-contiguous. I assume that
> is the behavior you are asking for?

Not at all -- in fact I was rather shocked when my attention was drawn to the
fact that this is also the behavior of Numeric -- I had thought that ravel
would *always* create a copy. I absolutely agree with the other posters that
remarked that different behavior of ravel (creating a copy vs creating a view,
depending on whether the argument is contiguous) is highly undesirable and
error-prone (especially since it is not even possible to determine at compile
time which behavior will occur, if I'm not mistaken). In fact, I think this
behavior is worse than what I incorrectly assumed to be the case.

What I was arguing for is a ravel that always has the same semantics, (namely
creating a copy) but tha -- because it would create the copy only demand --
would be just as efficient as using .flat when 

a) its argument were contiguous; and
b) neither the result nor the argument were modified while both are alive.

The reason that I view `.flat` as a hack, is that it is an operation that is
there exclusively for efficiency reasons and has no well defined semantics --
it will only work stochastically, giving better performance in certain
cases. Thus you have to cast lots whether you actually use it at runtime
(calling .iscontiguous) and always have a fall-back scheme (most likely using
ravel) at hand -- there seems to be no way to determine at compile time what's
going to happen.

I don't think a language or a library should have any such constructs or at
least strive to minimize their number. The fact that the current behavior of
ravel actually achieves the effect I want in most cases doesn't justify its
obscure behavior in my eyes, which translates into a variation of the
boiler-plate code previously mentioned (``if a.iscontiguous:...else:``) when
you actually want a *single* ravelled copy and it also is a very likely
candidate for extremely hard to find bugs.

One nice thing about python is that there is very little undefined
behavior. I'd like to keep it that way.

[snipped]
> > I personally don't find it messy.  And please keep in mind that
> > the ``view``
> > construct would only very seldomly be used if copy-on-demand is
> > the default
> > -- as I said, I've only needed the aliasing behavior once -- no
> > doubt it was
> > really handy then, but the fact that e.g. matlab doesn't have
> > anything along
> > those lines (AFAIK) suggests that many people will never need it.
> >
> You're kidding, right? Particularly after arguing for aliasing
> semantics in the previous paragraph for .flat ;-)

I didn't argue for any semantics of ``.flat`` -- I just pointed out that I
found the division of labour that I (incorrectly) assumed to be the case an
ugly hack (for the reasons outlined above):

``ravel``: always works, but always creates copy
            (which might be undesirable wastage of resources); [this was
            mistaken; the real semantics are: always works, creates view if
            contiguous, copy otherwise]

``.flat``: behavior undefined at compile time, a runtime-check can be used
           to ensure that it can be used as a more efficient alternative to
           ``ravel`` in some cases.

If I now understand the behavior of both ``ravel`` and ``.flat`` correctly
then I can't currently see *any* raison d'être for a ``.flat`` attribute.

If, as I would hope, the behavior of ravel is changed to always create copies
(ideally on-demand), then matters might look different. In that case, it might
be justifiable to have ``.flat`` as a specialized construct analogous to what
I proposed as``.view``, but only if there is some way to make it work (the
same) for both contiguous and non-contiguous arrays. I'm not sure that it
would be needed at all (especially with a lazy ravel).

alex

-- 
Alexander Schmolck     Postgraduate Research Student
                       Department of Computer Science
                       University of Exeter
A.S...@gm...     http://www.dcs.ex.ac.uk/people/aschmolc/

[Numpy-discussion] Personal

[victor i. n. <vic...@fa...>]

Dear Sir,
I am the Chairman Contract Review Committee of
National Electric Power Authority (NEPA).
Although this proposal might come to you as a surprise
since it is coming from someone you do not know or
ever seen before, but after due deliberation with my
colleagues, I decided to contact you based onIntuition.
We are soliciting for your humble and confidential
assistance to take custody of Seventy One Million,
Five Hundred Thousand United StatesDollars.{US$71,500,000.00}.
This sum (US$71.5M) is an over invoiced contract sum
which is currently in offshore payment account of the
Central Bank of Nigeria as an unclaimed contract
entitlement which can easily be withdrawn or drafted
or paid to any recommended beneficiary by my committee.
On this note, you will be presented as a contractor to
NEPA who has executed a contract to a tune of the
above sum and has not been paid.
Proposed Sharing Partern (%):
1. 70% for me and my colleagues.
2. 20% for you as a partner/fronting for us.
3. 10% for expenses that may be incure by both parties
during the cause of this transacton.
Our law prohibits a civil servant from operating a
foreign account, hence we are contacting you.
If this proposal satisfies you, do response as soon as
possible with the following information:
1. The name you wish to use as the beneficiary of thefund.
2. Your Confidential Phone and Fax Numbers.
Further discussion will be centered on how the fund
shall be transferred and full details on how to accomplish this great opportunity of ours.
Thank you and God bless.

Best regards,

victor ichaka nabia

Re: [Numpy-discussion] copy on demand

[Scott R. <ra...@ph...>]

On Sat, Jun 15, 2002 at 10:53:17AM +0200, Konrad Hinsen wrote:

> > floating around just seems like an invitation for confusion. I'm
> > very much in favor of picking one or the other behaviors and then
> > making some means of explicitly getting the other behavior.
> 
> Then the only solution I see is the current one: default behaviour is
> view, and when you want a copy yoy copy explicitly. The inverse is not
> possible, once you made a copy you can't make it behave like a view
> anymore.

I don't think it is necessary to create the other object _from_
the default one.  You could have copy behavior be the default,
and if you want a view of some array you simply request one
explicitly with .view, .sub, or whatever.

Since creating a view is "cheap" compared to creating a copy,
there is nothing sacrificed doing things in this manner.

Scott

-- 
Scott M. Ransom              Address:  McGill Univ. Physics Dept.
Phone:  (514) 398-6492                 3600 University St., Rm 338
email:  ra...@ph...       Montreal, QC  Canada H3A 2T8 
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989

Re: [Numpy-discussion] copy on demand

[Konrad H. <hi...@cn...>]

> I'd be strongly against this. This has the same problem that other
> customized array objects have (whether regarding slicing behavior,
> operators, coercion...). In particular, it is clear which kind it
> is when you create it, but you may pass it to a module that
> presumes different array behavior. Having different kind of arrays

We already have that situation with lists and arrays (and in much of
my code netCDF arrays, which have copy semantics) , but in my
experience this has never caused confusion. Most general code working
on sequences doesn't modify elements at all. When it does, it either
clearly requires view semantics (a function you call in order to
modify (parts of) an array) or clearly requires copy semantics (a
function that uses an array argument as an initial value that it then
modifies).

> floating around just seems like an invitation for confusion. I'm
> very much in favor of picking one or the other behaviors and then
> making some means of explicitly getting the other behavior.

Then the only solution I see is the current one: default behaviour is
view, and when you want a copy yoy copy explicitly. The inverse is not
possible, once you made a copy you can't make it behave like a view
anymore.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] copy on demand

[Scott R. <ra...@ph...>]

I was going to write an almost identical email, but Chris saved me the 
trouble.  These are my feelings as well.

Scott

On June 14, 2002 07:01 pm, Chris Barker wrote:
> Konrad Hinsen wrote:
> > Not necessarily. We could decide that
> >
> >    array.view
> >
> > is a view of the full array object, and that slicing views returns
> > subviews.
>
> Please don't!! Having two types of arrays around in a single program
> that have the same behaviour except when they are sliced is begging for
> confusion and hard to find bugs.
>
> I agree with Perry, that I occasionaly use the view behaviour of
> slicing, and it is very usefull when I do, but most of the time I would
> be happier with copy symantics. All I want is a way to get at a view of
> part of an array, I don't want two different kinds of array around with
> different slicing behaviour.
>
> > My main objection to changing the slicing behaviour is, like with some
> > other proposed changes, compatibility.
>
> The switch from Numeric to Numarray is a substantial change. I think we
> should view it like the mythical Py3k: an oportunity to make
> incompatible changes that will really make it better.
>
> By the way, as an old MATLAB user, I have to say that being able to get
> views from a slice is one behaviour of NumPy that I really appreciate,
> even though I only need it occasionally. MATLAB, howver is a whole
> different ball of wax in a lot of ways. There has been a lot of
> discussion about the copy on demand idea in MATLAB, but that is
> primarily useful because MATLAB has call by value function semantics, so
> without copy on demand, you would be making copies of large arrays
> passed to functions that weren't even going to change them. I don't
> think MATLAB impliments copy on demand for slices anyway, but I could be
> wrong there.
>
> Oh, and no function (ie ravel() ) should return a view in some cases,
> and a copy in others, that is just asking for bugs!
>
> -Chris

-- 
Scott M. Ransom              Address:  McGill Univ. Physics Dept.
Phone:  (514) 398-6492                 3600 University St., Rm 338
email:  ra...@ph...       Montreal, QC  Canada H3A 2T8 
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989

Re: [Numpy-discussion] copy on demand

[Chris B. <Chr...@no...>]

Konrad Hinsen wrote:
> Not necessarily. We could decide that
> 
>    array.view
> 
> is a view of the full array object, and that slicing views returns
> subviews.

Please don't!! Having two types of arrays around in a single program
that have the same behaviour except when they are sliced is begging for
confusion and hard to find bugs.

I agree with Perry, that I occasionaly use the view behaviour of
slicing, and it is very usefull when I do, but most of the time I would
be happier with copy symantics. All I want is a way to get at a view of
part of an array, I don't want two different kinds of array around with
different slicing behaviour.

> My main objection to changing the slicing behaviour is, like with some
> other proposed changes, compatibility. 

The switch from Numeric to Numarray is a substantial change. I think we
should view it like the mythical Py3k: an oportunity to make
incompatible changes that will really make it better.

By the way, as an old MATLAB user, I have to say that being able to get
views from a slice is one behaviour of NumPy that I really appreciate,
even though I only need it occasionally. MATLAB, howver is a whole
different ball of wax in a lot of ways. There has been a lot of
discussion about the copy on demand idea in MATLAB, but that is
primarily useful because MATLAB has call by value function semantics, so
without copy on demand, you would be making copies of large arrays
passed to functions that weren't even going to change them. I don't
think MATLAB impliments copy on demand for slices anyway, but I could be
wrong there.

Oh, and no function (ie ravel() ) should return a view in some cases,
and a copy in others, that is just asking for bugs!

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] copy on demand

[John J. L. <jj...@po...>]

On 14 Jun 2002, Alexander Schmolck wrote:
[...]
> The fact that you can never really be sure whether you can actually use
> ``.flat``, without checking beforehand if the array is in fact
> contiguous (I don't think there are many guarantees about something
> being contiguous, or are there?) and that ravel will always work but has
> a huge overhead, suggests to me that something is not quite right.

Why does ravel have a huge overhead?  It seems it already doesn't copy
unless required: search for 'Chacking' -- including the mis-spelling -- in
this thread:

http://groups.google.com/groups?hl=en&lr=&threadm=abjbfp%241t9%241%40news5.svr.pol.co.uk&rnum=1&prev=/groups%3Fq%3Diterating%2Bover%2Bthe%2Bcells%2Bgroup:comp.lang.python%26hl%3Den%26lr%3D%26scoring%3Dr%26selm%3Dabjbfp%25241t9%25241%2540news5.svr.pol.co.uk%26rnum%3D1

or start up your Python interpreter, if you're less lazy than me.


John

RE: [Numpy-discussion] copy on demand

[Perry G. <pe...@st...>]

<Konrad Hinsen writes>:
   <Perry Greenfield writes>:
> 
> > I didn't mean to imply it wasn't possible, but that it was not
> > quite as clean. The thing I don't like about this approach (or
> > Paul's suggestion of a.sub) is the creation of an odd object
> > that has as its only purpose being sliced. (Even worse, in my
> 
> Not necessarily. We could decide that
> 
>    array.view
> 
> is a view of the full array object, and that slicing views returns
> subviews.
> 
> > opinion, is making it a different kind of array where slicing
> > behaves differently. That will lead to the problem we have
> > discussed for other kinds of array behavior, namely, how do
> 
> A view could be a different type of object, even though much of the
> implementation would be shared with arrays. This would help to
> reduce confusion.
> 
I'd be strongly against this. This has the same problem that other
customized array objects have (whether regarding slicing behavior,
operators, coercion...). In particular, it is clear which kind it
is when you create it, but you may pass it to a module that
presumes different array behavior. Having different kind of arrays
floating around just seems like an invitation for confusion. I'm
very much in favor of picking one or the other behaviors and then
making some means of explicitly getting the other behavior.

> > behavior). That could lead to confusion as well. Many may be
> > under the impression that x = a.view makes x refer to an array
> > when it doesn't. Users would need to know that a.view without
> > a '[' is usually an error.
> 
> Why? It would be a full-size view, which might actually be useful
> in many situations.
> 
But one can do that simply by

x = a

(Though there is the issue that one could do the following which is
not the same:

x = a.view
x.shape = (2,50)

so that x is a full array view with a different shape than a)

********

I understand the backward compatibilty issue here, but it is clear that
this is an issue that appears to be impossible to get a consensus on.
There appear to be significant factions that care passionately about
copy vs view and no matter what decision is made many will be unhappy.

Perry

Re: [Numpy-discussion] copy on demand

[Konrad H. <hi...@cn...>]

"Perry Greenfield" <pe...@st...> writes:

> I didn't mean to imply it wasn't possible, but that it was not
> quite as clean. The thing I don't like about this approach (or
> Paul's suggestion of a.sub) is the creation of an odd object
> that has as its only purpose being sliced. (Even worse, in my

Not necessarily. We could decide that

   array.view

is a view of the full array object, and that slicing views returns
subviews.

> opinion, is making it a different kind of array where slicing
> behaves differently. That will lead to the problem we have
> discussed for other kinds of array behavior, namely, how do

A view could be a different type of object, even though much of the
implementation would be shared with arrays. This would help to
reduce confusion.

> behavior). That could lead to confusion as well. Many may be
> under the impression that x = a.view makes x refer to an array
> when it doesn't. Users would need to know that a.view without
> a '[' is usually an error.

Why? It would be a full-size view, which might actually be useful
in many situations.

My main objection to changing the slicing behaviour is, like with some
other proposed changes, compatibility. Even though view behaviour is
not required by every NumPy program, there are people out there who
use it and finding the locations in the code that need to be changed
is a very tricky business. It may keep programmers from switching to
Numarray in spite of benefits elsewhere.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] copy on demand

[Tim H. <tim...@ie...>]

<"Perry Greenfield" writes>
[SNIP]
> Numarray already returns a view of the array if it is contiguous.
> Copies are only produced if it is non-contiguous. I assume that
> is the behavior you are asking for?

This is one horrible aspect of NumPy that I hope you get rid of. I've been
burned by this several times -- I expected a view, but silently got a copy
because my array was noncontiguous. If you go with copy semantics, this will
go away, if you go with view semantics, this should raise an exception
instead of silently copying. Ditto with reshape,  etc. In my experience,
this is a source of hard to find bugs (as opposed to axes issues which tend
to produce shallow bugs).

[SNIP]

> Currently for numarray .flat will fail if it isn't contiguous. It isn't
> clear if this should change. If .flat is meant to be a view always, then
> it should always fail it the array is not contiguous. Ravel is not
> guaranteed to be a view.

Ravel should either always return a view or always return a copy -- I don't
care which

> This is a problematic issue if we decide to switch from view to copy
> semantics. If slices produce copies, then does .flat? If so, then
> how does one produce a flattened view? x.view.flat?

Wouldn't that just produce a copy of the view? Unless you did some weird
special casing on view? The following would work, although it's a little
clunky.

flat_x = x.view[:] # Or however "get me a view" would be spelled.
flat_x.shape = (-1,)


-tim

RE: [Numpy-discussion] copy on demand

[Perry G. <pe...@st...>]

<Alexander Schmolck writes>:
    <Perry Greenfield writes>:
> > I guess that depends on what you mean by unnecessary copies.
>
> In most cases the array of which I desire a flattened representation is
> contiguous (plus, I usually don't intend to modify it).
> Consequently, in most
> cases I don't want to any copies of it to be created (especially
> not if it is
> really large -- which is not seldom the case).
>
Numarray already returns a view of the array if it is contiguous.
Copies are only produced if it is non-contiguous. I assume that
is the behavior you are asking for?

> The fact that you can never really be sure whether you can actually use
> ``.flat``, without checking beforehand if the array is in fact
> contiguous (I
> don't think there are many guarantees about something being
> contiguous, or are
> there?) and that ravel will always work but has a huge overhead,
> suggests to
> me that something is not quite right.
>
Not for numarray, at least in this context.

> > If the array is non-contiguous what would you have it do?
>
> Simple -- in that case 'lazy ravel' would do the same as 'ravel' currently
> does, create a copy (or alternatively rearrange the memory
> representation to
> make it non-contiguous and then create a lazy copy, but I don't
> know whether
> this would be a good or even feasible idea).
>
> A lazy version of ravel would have the same semantics as ravel
> but only create
> an actual copy if necessary-- which means as long as no modification takes
> place and the array is non-contiguous, it will be sufficient to return the
> ``.flat`` (for starters). If it is contiguous than the copying can't be
> helped, but these cases are rare and currently you either have to test for
> them explicitly or slow everything down and waste memory by just
> always using
> ``ravel()``.
>
Currently for numarray .flat will fail if it isn't contiguous. It isn't
clear if this should change. If .flat is meant to be a view always, then
it should always fail it the array is not contiguous. Ravel is not
guaranteed to be a view.

This is a problematic issue if we decide to switch from view to copy
semantics. If slices produce copies, then does .flat? If so, then
how does one produce a flattened view? x.view.flat?

> For example, if bar is contiguous ``foo = ravel(bar)`` would be
> computationally equivalent to ``bar.flat``, as long as neither of them is
> modified, but semantically equivalent to the current ``foo =
> ravel(bar)`` in
> all cases.
>
> Thus you could now write:
>
> >>> a = ravel(a)[20:]
>
> wherever you've written this boiler-plate code before:
>
> >>> if a.iscontiguous():
> >>>    a = a.flat[20:]
> >>> else:
> >>>    a = ravel(a)[20:]
>
> without any loss of performance.
>
I believe this is already true in numarray.

>
> I personally don't find it messy.  And please keep in mind that
> the ``view``
> construct would only very seldomly be used if copy-on-demand is
> the default
> -- as I said, I've only needed the aliasing behavior once -- no
> doubt it was
> really handy then, but the fact that e.g. matlab doesn't have
> anything along
> those lines (AFAIK) suggests that many people will never need it.
>
You're kidding, right? Particularly after arguing for aliasing
semantics in the previous paragraph for .flat ;-)

>
> Also what exactly is the confused person's notion of the purpose of ``x =
> a.view`` supposed to be? That ``x = a`` is what ``x = a.copy()``
> really does
> and that to create aliases an alias to ``a`` they would have to use
> ``x = a.view``? In that case they'd better read the python
> tutorial before they do
> any more python programming, because they are in for all kinds of
> unpleasant
> surprises (``a = []; b = a; b[1] = 3; print a`` -- oops).
>
This is basically true, though the confusion may be that a.view is
an array object that has different slicing behavior instead of
an non-array object that can be sliced to produce a view. I don't
view it as a major issue but I do see how may mistakenly infer that.

Perry

Re: [Numpy-discussion] copy on demand

[Alexander S. <a.s...@gm...>]

"Perry Greenfield" <pe...@st...> writes:

> > I guess the best reason to bite the bullet and carry around state
> > information
> > would be if there were significant other cases where one also
> > would want to
> > optimize operations under the hood. If there isn't much else in
> > this direction
> > then the effort involved might not be justified. One thing that bugs me in
> > Numeric (and that might already have been solved in numarray) is that
> > e.g. ``ravel`` (and I think also ``transpose``) creates
> > unnecessary copies,
> > whereas ``.flat`` doesn't, but won't work in all cases (viz. when
> > the array is
> > non-contiguous), so I can either have ugly or inefficient code.
> >
> I guess that depends on what you mean by unnecessary copies. 

In most cases the array of which I desire a flattened representation is
contiguous (plus, I usually don't intend to modify it). Consequently, in most
cases I don't want to any copies of it to be created (especially not if it is
really large -- which is not seldom the case).

The fact that you can never really be sure whether you can actually use
``.flat``, without checking beforehand if the array is in fact contiguous (I
don't think there are many guarantees about something being contiguous, or are
there?) and that ravel will always work but has a huge overhead, suggests to
me that something is not quite right.

> If the array is non-contiguous what would you have it do?

Simple -- in that case 'lazy ravel' would do the same as 'ravel' currently
does, create a copy (or alternatively rearrange the memory representation to
make it non-contiguous and then create a lazy copy, but I don't know whether
this would be a good or even feasible idea).

A lazy version of ravel would have the same semantics as ravel but only create
an actual copy if necessary-- which means as long as no modification takes
place and the array is non-contiguous, it will be sufficient to return the
``.flat`` (for starters). If it is contiguous than the copying can't be
helped, but these cases are rare and currently you either have to test for
them explicitly or slow everything down and waste memory by just always using
``ravel()``.

For example, if bar is contiguous ``foo = ravel(bar)`` would be
computationally equivalent to ``bar.flat``, as long as neither of them is
modified, but semantically equivalent to the current ``foo = ravel(bar)`` in
all cases.

Thus you could now write:

>>> a = ravel(a)[20:]

wherever you've written this boiler-plate code before:

>>> if a.iscontiguous(): 
>>>    a = a.flat[20:]
>>> else:
>>>    a = ravel(a)[20:]

without any loss of performance.

> 
> > > a feature one wants even though they are not the default, it turns
> > > out that it isn't all that simple to obtain views without sacrificing
> > > ordinary slicing syntax to obtain a view. It is simple to obtain
> > > copies of view slices though.
> >
> > I'm not sure I understand the above.  What is the problem with
> > ``a.view[1:3]``
> > (or``a.view()[1:3])?
> >
> I didn't mean to imply it wasn't possible, but that it was not
> quite as clean. The thing I don't like about this approach (or
> Paul's suggestion of a.sub) is the creation of an odd object
> that has as its only purpose being sliced. (Even worse, in my

I personally don't find it messy.  And please keep in mind that the ``view``
construct would only very seldomly be used if copy-on-demand is the default
-- as I said, I've only needed the aliasing behavior once -- no doubt it was
really handy then, but the fact that e.g. matlab doesn't have anything along
those lines (AFAIK) suggests that many people will never need it.

So even if ``.view`` is messy, I'd rather have something messy that is almost
never used, in exchange for (what I perceive as) significantly nicer and
cleaner semantics for something that is used all the time (array slicing;
alias slicing is messy in at least the respect that it breaks standard usage
and generic sequence code as well as causing potentially devious
bugs. Unexpected behaviors like phantom buffers kept alive in their entirety
by partial views etc. or what ``A = A[::-1]`` does are not exactly pretty
either).


> opinion, is making it a different kind of array where slicing
> behaves differently. That will lead to the problem we have
> discussed for other kinds of array behavior, namely, how do
> you keep from being confused about a particular array's slicing
> behavior). That could lead to confusion as well. Many may be

I don't see that problem, frankly. The view is *not* an array. It doesn't need
(and shouldn't have) anything except a method to access slices (__getitem__).

As mentioned before, I also regard it as highly desirable that ``b =
a.view[3:10]`` sticks out immediately. This signals "warning -- potentially
tricky code ahead". Nothing in ``b = a[3:10]`` tells you that someone intends
to modify a and b depedently (because in more than 9 out of 10 cases he won't)
-- now *this* is confusing.

> under the impression that x = a.view makes x refer to an array
> when it doesn't. Users would need to know that a.view without
> a '[' is usually an error.

Since the ``.view`` shouldn't allow anything except slicing, they'll soon find
out ("Error: you can't multiply me, I'm a view and not an array"). And I can't
see why that would be harder to figure out (or look up in the docu) than that
a[1:3] creates an alias and *not* a copy contrary to *everything* else you've
ever heard or read about python sequences (especially since in most cases it
will work as intended).

Also what exactly is the confused person's notion of the purpose of ``x =
a.view`` supposed to be? That ``x = a`` is what ``x = a.copy()`` really does
and that to create aliases an alias to ``a`` they would have to use 
``x = a.view``? In that case they'd better read the python tutorial before they do
any more python programming, because they are in for all kinds of unpleasant
surprises (``a = []; b = a; b[1] = 3; print a`` -- oops).


alex

-- 
Alexander Schmolck     Postgraduate Research Student
                       Department of Computer Science
                       University of Exeter
A.S...@gm...     http://www.dcs.ex.ac.uk/people/aschmolc/

Re: [Numpy-discussion] copy on demand

[Alexander S. <a.s...@gm...>]

"Perry Greenfield" <pe...@st...> writes:

> I'm not sure what you mean. Are you saying that if anything in the
> buffer changes, force all views of the buffer to generate copies
> (rather than try to determine if the change affected only selected

Yes (I suspect that this will be be sufficient in practice).

> views)? If so, yes, it is easier, but it still is a non-trivial
> capability to implement.

Sure. But since copy-on-demand is only an optimization and as such doesn't
affect the semantics, it could also be implemented at a later point if the
resources are currently not available. I have little doubt that someone will
eventually add copy-on-demand, if the option is kept open and in the meantime
one could still get all the performance (and alias behavior) of the current
implementation by explicitly using ``.view`` (or ``.sub`` if you prefer) to
create aliases.

I'm becoming increasingly convinced (see below) that copy-slicing-semantics
are much to be preferred as the default, so given the above I don't think that
performance concerns should sway one towards alias-slicing, if enough people
feel that copy semantics as such are preferable.

> > >
> > > The bookkeeping can get pretty messy (if you care about memory usage,
> > > which we definitely do).  Consider this case:
> > >
> > >     >>> a = zeros((5000,5000))
> > >     >>> b = a[0:-10,0:-10]
> > >     >>> c = a[49:51,50]
> > >     >>> del a
> > >     >>> b[50,50] = 1
> > >
> > > Now what happens?  Either we can copy the array for b (which means two
> >
> > ``b`` and ``c`` are copied and then ``a`` is deleted.
> >
> > What does numarray currently keep of a if I do something like the
> > above or:
> >
> > >>> b = a.flat[::-10000]
> > >>> del a
> >
> > ?
> >
> The whole buffer remains in both cases.

OK, so this is then a nice example where even eager copy slicing behavior
would be *significantly* more efficient than the current aliasing behavior --
so copy-on-demand would then on the whole seem to be not just nearly equally
but *more* efficient than alias slicing.

And as far as difficult to understand runtime behavior is concerned, the extra
~100MB useless baggage carried around by b (second case) are, I'd venture to
suspect, less than obvious to the casual observer. In fact I remember one of
my fellow phd-students having significant problems with mysterious memory
consumption (a couple of arrays taking up more than 1GB rather than a few
hundred MB) -- maybe something like the above was involved. That
 ``A = A[::-1]`` doesn't work (as pointed out by Paul Barrett) will also come as a
surprise to most people.

If I understand all this correctly, I consider it a rather strong case against
alias slicing as default behavior.


> > > Even keeping track of the views associated with a buffer doesn't solve
> > > the problem of an array that is passed to a C extension and is modified
> > > in place.  It would seem that passing an array into a C extension would
> > > always require all the associated views to be turned into copies.
> > > Otherwise we can't guarantee that views won't be modifed.
> >
> > Yes -- but only if the C extension is destructive. In that case
> > the user might
> > well be making a mistake in current Numeric if he has views and
> > doesn't want
> > them to be modified by the operation (of course he might know
> > that the inplace
> > operation does not affect the view(s) -- but wouldn't such cases be rather
> > rare?). If he *does* want the views to be modified, he would
> > obviously have to
> > explictly specify them as such in a copy-on-demand scheme and in the other
> > case he has been most likely been prevented from making an error (and can
> > still explicitly use real views if he knows that the inplace
> > operation on the
> > original will not have undesired effects on the "views").
> >
> If the point is that views are susceptible to unexpected changes
> made in place by a C extension, yes, certainly (just as they
> are for changes made in place in Python). But I'm not sure what
> that has to do with the implied copy (even if delayed) being
> broken by extensions written in C. Promising a copy, and not
> honoring it is not the same as not promising it in the first
> place. But I may be misunderstanding your point.
> 

OK, I'll try again, hopefully this is clearer.

In a sentence: I don't see any problems with C extensions in particular that
would arise from copy-on-demand (I might well be overlooking something,
though).

Rick was saying that passing an array to a C extension that performs an
inplace operation on it means that all copies of all its (lazy) views must be
performed.

My point was that this is correct, but I can't see any problem with that,
neither from the point of extension writer, nor from the point of performance
nor from the point of the user, nor indeed from the point of the numarray
implementors (obviously the copy-on-demand scheme *as such* will be an
effort).

All that is needed is a separate interface for (the minority of) C extensions
that destructively modify their arguments (they only need to call some
function `actualize_views(the_array_or_view)` or whatever at the start -- this
function will obviously be necessary regardless of the C extensions). So
nothing will break, the promises are kept and no extra work.

It won't be any slower than what would happen with current Numeric, either,
because either the (Numeric) user intended his (aliased) views to modified as
well or it was a bug. If he intended the views to be modified, he would
explicitly use alias-views under the new scheme and everything would behave
exactly the same.



alex
-- 
Alexander Schmolck     Postgraduate Research Student
                       Department of Computer Science
                       University of Exeter
A.S...@gm...     http://www.dcs.ex.ac.uk/people/aschmolc/

[Numpy-discussion] Some initial thoughts about the past week's discussions

[Perry G. <pe...@st...>]

Impressions so far on various issues raised regarding numarray
interfaces

1) We are mostly persuaded that rank-0 arrays are the way to go.
   We will pursue the issue of whether it is possible to have 
   Python accept these as indices for sequence objects with python-dev.
2) We are still mulling over the axis order issue. Regardless of 
   which convention we choose, we are almost certainly going to 
   make it consistent (always the same axis as default). A compatibility
   module will be provided to replicate Numeric defaults.
3) repr. Finally, a consensus! Even unanimity.
4) Complex comparisons. Implement equality, non-equality, predictable
   sorting. Make >,<,>=,<= illegal.
5) Copy vs view. Open to more input (but no delayed copying or such).

Perry

RE: [Numpy-discussion] copy on demand

[Perry G. <pe...@st...>]

> > Copy-on-demand requires the maintenance of a global list of all the
> > active views associated with a particular array buffer.  Here is a
> > simple example:
> >
> >     >>> a = zeros((5000,5000))
> >     >>> b = a[49:51,50]
> >     >>> c = a[51:53,50]
> >     >>> a[50,50] = 1
> >
> > The assignment to a[50,50] must trigger a copy of the array b;
> > otherwise b also changes.  On the other hand, array c does not need to
> > be copied since its view does not include element 50,50.  You could
> > instead copy the array a -- but that means copying a 100 Mbyte array
> > while leaving the original around (since b and c are still using it) --
> > not a good idea!
>
> Sure, if one wants do perform only the *minimum* amount of
> copying, things can
> get rather tricky, but wouldn't it be satisfactory for most cases
> if attempted
> modification of the original triggered the delayed copying of the "views"
> (lazy copies)?  In those cases were it isn't satisfactory the
> user could still
> explicitly create real (i.e. alias-only) views.
>
I'm not sure what you mean. Are you saying that if anything in the
buffer changes, force all views of the buffer to generate copies
(rather than try to determine if the change affected only selected
views)? If so, yes, it is easier, but it still is a non-trivial
capability to implement.

> >
> > The bookkeeping can get pretty messy (if you care about memory usage,
> > which we definitely do).  Consider this case:
> >
> >     >>> a = zeros((5000,5000))
> >     >>> b = a[0:-10,0:-10]
> >     >>> c = a[49:51,50]
> >     >>> del a
> >     >>> b[50,50] = 1
> >
> > Now what happens?  Either we can copy the array for b (which means two
>
> ``b`` and ``c`` are copied and then ``a`` is deleted.
>
> What does numarray currently keep of a if I do something like the
> above or:
>
> >>> b = a.flat[::-10000]
> >>> del a
>
> ?
>
The whole buffer remains in both cases.

> > copies of the huge (5000,5000) array exist, one used by c and the new
> > version used by b), or we can be clever and copy c instead.
> >
> > Even keeping track of the views associated with a buffer doesn't solve
> > the problem of an array that is passed to a C extension and is modified
> > in place.  It would seem that passing an array into a C extension would
> > always require all the associated views to be turned into copies.
> > Otherwise we can't guarantee that views won't be modifed.
>
> Yes -- but only if the C extension is destructive. In that case
> the user might
> well be making a mistake in current Numeric if he has views and
> doesn't want
> them to be modified by the operation (of course he might know
> that the inplace
> operation does not affect the view(s) -- but wouldn't such cases be rather
> rare?). If he *does* want the views to be modified, he would
> obviously have to
> explictly specify them as such in a copy-on-demand scheme and in the other
> case he has been most likely been prevented from making an error (and can
> still explicitly use real views if he knows that the inplace
> operation on the
> original will not have undesired effects on the "views").
>
If the point is that views are susceptible to unexpected changes
made in place by a C extension, yes, certainly (just as they
are for changes made in place in Python). But I'm not sure what
that has to do with the implied copy (even if delayed) being
broken by extensions written in C. Promising a copy, and not
honoring it is not the same as not promising it in the first
place. But I may be misunderstanding your point.

Perry

RE: [Numpy-discussion] copy on demand

[Perry G. <pe...@st...>]

> I guess the best reason to bite the bullet and carry around state
> information
> would be if there were significant other cases where one also
> would want to
> optimize operations under the hood. If there isn't much else in
> this direction
> then the effort involved might not be justified. One thing that bugs me in
> Numeric (and that might already have been solved in numarray) is that
> e.g. ``ravel`` (and I think also ``transpose``) creates
> unnecessary copies,
> whereas ``.flat`` doesn't, but won't work in all cases (viz. when
> the array is
> non-contiguous), so I can either have ugly or inefficient code.
>
I guess that depends on what you mean by unnecessary copies. If the
array is non-contiguous what would you have it do?

> > a feature one wants even though they are not the default, it turns
> > out that it isn't all that simple to obtain views without sacrificing
> > ordinary slicing syntax to obtain a view. It is simple to obtain
> > copies of view slices though.
>
> I'm not sure I understand the above.  What is the problem with
> ``a.view[1:3]``
> (or``a.view()[1:3])?
>
I didn't mean to imply it wasn't possible, but that it was not
quite as clean. The thing I don't like about this approach (or
Paul's suggestion of a.sub) is the creation of an odd object
that has as its only purpose being sliced. (Even worse, in my
opinion, is making it a different kind of array where slicing
behaves differently. That will lead to the problem we have
discussed for other kinds of array behavior, namely, how do
you keep from being confused about a particular array's slicing
behavior). That could lead to confusion as well. Many may be
under the impression that x = a.view makes x refer to an array
when it doesn't. Users would need to know that a.view without
a '[' is usually an error.

Sure it's not hard to implement. But I don't view it as that
clean a solution. On the other hand, a[1:3].copy() (or alternatively,
a[1:3].copy) is another array just like any other.
>
Perry

RE: [Numpy-discussion] RE: default axis for numarray

[Perry G. <pe...@st...>]

<Paul Dubois writes>: 
> There are now dead horses all over the landscape, and I for one am going
> to shut up.
> 
Not enough dead horses for me :-). But seriously, I would like to
hear from others about this issue (I already knew what Paul, Paul,
Eric, Travis and Konrad felt about this before it started up).
You can either post to the mailing list or email directly if
you are the shy, retiring type.

Perry

Re: [Numpy-discussion] (a and b) != (b and a) ?

[Andrew P. L. <bs...@ma...>]

On Wed, 12 Jun 2002, Scott Gilbert wrote:
> C++ has overloadable && and || operators, but overloading them is frowned
> on by many.  C++ has the advantage over Python in that it knows the actual
> types at compile time.

Actually, overloading && and || isn't just frowned upon in C++, it's
effectively banned.  The reason is that it replaces short-circuit
semantics with function call semantics and screws up the standard idioms
(if ((a != NULL) && (*a == "a")) { ... } ).  See "Effective C++" by Scott
Meyers.  As far as I know, *none* of the C++ literati hold the opposing
view.

-a

Re: [Numpy-discussion] (a and b) != (b and a) ?

[Scott G. <xs...@ya...>]

--- Chris Barker <Chr...@no...> wrote:
> 
> Well, yes, but it wasn't possible with <,>,== and friends untill rich
> comparisons were added in Python 2.1. So I am still wondering why the
> same extension wasn't made to "and" and "or". In fact, given that Guido
> is adding a bool type, this may be a time to re-visit the question,
> unless there really is a compelling reason not to, which is quite
> likely. 
> 

The "and" and "or" operators do short circuit evaluation.  So in addition
to acting like boolean operations, they are also control flow.  For "and",
the second expression is not evaluated if the first one is false.  For
"or", the second expression is not evaluated if the first one is true.  

I'm not clever enough to figure out how an overloaded and/or operator could
implement control flow for the outer expressions.  The outer expressions
"self" and "other" would already be evaluated by the time your
__operator__(self, other) function was called.

C++ has overloadable && and || operators, but overloading them is frowned
on by many.  C++ has the advantage over Python in that it knows the actual
types at compile time.




__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com

Re: [Numpy-discussion] (a and b) != (b and a) ?

[Chris B. <Chr...@no...>]

Tim Hochberg wrote:
> > I imagine there is a compelling reason that "and" and "or" have not been
> > overridden like the comparison operators, but it sure would be nice!
> 
> Because it's not possible?

Well, yes, but it wasn't possible with <,>,== and friends untill rich
comparisons were added in Python 2.1. So I am still wondering why the
same extension wasn't made to "and" and "or". In fact, given that Guido
is adding a bool type, this may be a time to re-visit the question,
unless there really is a compelling reason not to, which is quite
likely. 

> In practice I haven't found this to be much of a problem. Nearly every time
> I need to and two arrays together, "&" works just as well as logical_and.

This has always worked for me, as well, so maybe the answer is that
there is no compelling reason to make a change.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] copy on demand

[Alexander S. <a.s...@gm...>]

"Perry Greenfield" <pe...@st...> writes:

> <Rick White writes> :
> 
> > This kind of state information with side effects leads to a system that
> > is hard to develop, hard to debug, and really messes up the behavior of
> > the program (IMHO).  It is *highly* desirable to avoid it if possible.
> > 
> Rick beat me to the punch. The requirement for copy-on-demand
> definitely leads to a far more complex implementation with 
> much more potential for misunderstood memory usage. You could
> do one small thing and suddenly force a spate of copies (perhaps
> cascading). There is no way we would taken on a redesign of 

Yes, but I would suspect that cases were a little innocuous a[0] = 3 triggers
excessive processing should be rather unusual (matlab or octave users will know).

> Numeric with this requirement with the resources we have available.

Fair enough -- if implementing copy-on-demand is too much work then we'll have
to live without it (especially if view-slicing doesn't stand in the way of a
future inclusion into the python core).

I guess the best reason to bite the bullet and carry around state information
would be if there were significant other cases where one also would want to
optimize operations under the hood. If there isn't much else in this direction
then the effort involved might not be justified. One thing that bugs me in
Numeric (and that might already have been solved in numarray) is that
e.g. ``ravel`` (and I think also ``transpose``) creates unnecessary copies,
whereas ``.flat`` doesn't, but won't work in all cases (viz. when the array is
non-contiguous), so I can either have ugly or inefficient code.

> 
> > This is not to deny that copy-on-demand (with explicit views available
> > on request) would have some desirable advantages for the behavior of
> > the system.  But we've worried these issues to death, and in the end
> > were convinced that slices == views provided the best compromise
> > between the desired behavior and a clean implementation.
> > 
> Rick's explanation doesn't really address the other position which
> is slices should force immediate copies. This isn't a difficult
> implementation issue by itself. But it does raise some related
> implementation questions. Supposing one does feel that views are
> a feature one wants even though they are not the default, it turns
> out that it isn't all that simple to obtain views without sacrificing
> ordinary slicing syntax to obtain a view. It is simple to obtain
> copies of view slices though.

I'm not sure I understand the above.  What is the problem with ``a.view[1:3]``
(or``a.view()[1:3])?

> 
> Slicing views may not be important to everyone. It is important
> to us (and others) and we do see a number of situations where
> forcing copies to operate on array subsets would be a serious
> performance problem. We did discuss this issue with Guido and

Sure, no one denies that even if with copy-on-demand (explicitly) aliased
views would still be useful.

> he did not  indicate that having different behavior on slicing
> with arrays would be a show stopper for acceptance into the
> Standard Library. We are also aware that there is no great
> consensus on this issue (even internally at STScI :-).
> 

Yep, I just saw Paul Barrett's post :)

> Perry Greenfield
> 
> 


alex
-- 
Alexander Schmolck     Postgraduate Research Student
                       Department of Computer Science
                       University of Exeter
A.S...@gm...     http://www.dcs.ex.ac.uk/people/aschmolc/