numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] adding a .M attribute to the array.

[<a.s...@gm...>]

Konrad Hinsen <hi...@cn...> writes:

> Computational and notational efficiency are rather well separated,
> fortunately. Both the current dot function and an hypothetical matrix

Yes, the only thing they have in common is that both are currently
unsatisfactory (for matrix operations) in numpy, at least for my needs.
Although I've solved my most pressing performance problems by patching Numeric
[1], I'm obviously interested in a more official solution (i.e. one that is
maintained by others :)

[...] [order changed by me]

> a.s...@gm... (A.Schmolck) writes:
> > My impression is that the best path also very much depends on the what the
> > feature aspirations and divisions of labor of numpy/numarray and scipy are
    ^^^^^^^
Darn, I made a confusing mistake -- this should read _future_.

> > going to be. For example, scipy is really aimed at scientific users, which
> > need performance, and are willing to buy it with inconvenience (like the
> 
> I see the main difference in distribution philosophy. NumPy is an
> add-on package to Python, which is in turn used by other add-on
> packages in a modular way. SciPy is rather a monolithic
> super-distribution for scientific users.
> 
> Personally I strongly favour the modular package approach, and in fact
> I haven't installed SciPy on my system for that reason, although I
> would be interested in some of its components.

[...]

> The same approach as for XML could be used: a slim-line version in the
> standard distribution that could be replaced by a high-efficiency
> extended version for those who care.

[...]

I personally agree with all your above points -- if you have a look at our
"dotblas"-patch mentioned earlier (see [1]), you will find that it aims to do
provide that -- have dot run anywhere without a hassle but run (much) faster
if the user is willing to install atlas.

My main concern was that the argument should shift away a bit from syntactic
and implementation details to what audiences and what needs numpy/numarray and
are supposed to address and, in this light, how to best strike the balance
between convinience for users and maitainers, speed and bloat, generality and
efficiency etc.

As an example, adding the dotblas patch [1] to Numeric is, I think more
convinient for the users (granting a few assumptions (like that it actually
works :) for the sake of the argument) -- it gives users that have atlas
better-performance and those who don't won't (or at least shouldn't) notice.

It is however inconvinient for the maintainers. Whether one should bother
including it in this or some other way depends, among the obvious question of
whether there is a better way to achieve what it does for both groups (like
creating a dedicated Matrix class), also on what numpy is really supposed to
achieve. I'm not entirely clear on that. For example I don't know how many
numpy users deeply care about their matrix multiplications for big (1000x1000)
matrices being 40 times faster.

The monolithic approach is not entirely without its charms (remember python's
"batteries included" jinggle)? Apart from convinience factors it also has the
not unconsiderable advantage that people use _one_ standard module for a
certain thing -- rather than 20 different solutions. This certainly helps to
improve code quality. Not least because someone goes through the trouble of
deciding what merrit's inclusion in the "Big Thing", possibly urging changes
but at least almost certainly taking more time for evalutation than an
indivdual programmer who just wants to get a certain job done. It also makes
life easier for module writers -- they can rely on certain stuff being around
(and don't have to reinvent the wheel, another potential improvement to code
quality). As such it makes live easier for maintainers, as does the scipy
commandment that you have to install atlas/lapack, full-stop (and if it
doesn't run on your machine -- well at least it works fast for some people and
that might well be better than working slow for everyone in this context).

So, I think what's good really depends on what you're aiming at, that's why
I'd like to know what users and developers think about these matters.

My points regarding scipy and numpy/numarray were just one attempt at
interpreting what these respective libraries try to/should/could attempt to be
or become. Now, not being a developer for either of them (I've only submitted
a few minor patches to scipy), I'm not in a particular good position to
venture such interpretations, but I hoped that it would provoke other and more
knowledgeable people to share their opinions and insights on this matter (as
indeed you did).

> I'd love to have efficient matrices without having to install the
> whole SciPy package!

Welcome to the linear algebra lobby group ;) yep, that would be nice but my
impression was that the scipy folks are currently more concerned about
performance issues than the numpy/numarray folks and I could live with either
package providing what I want.

Ideally , I'd like to see a slim core numarray, without any frills (and more
streamlined to behave like standard python containers (e.g. indexing and
type/casts behavior)) for the python core, something more enabled and
efficient for numerics (including matrices!) as a seperate package (like the
XML example you quote). And then maybe a bigger pre-bundled collection of
(ideally rather modular) numerical libraries for really hard-core scientific
users (maybe in the spirit of xemacs-packages and sumo-tar-balls -- no bloat
if you don't need it, plenty of features in an instant if you do).

Anyway, is there at least general agreement that there should be some new and
wonderful matrix class (plus supporting libraries) somewhere (rather than
souping up array)?

alex


Footnotes: 
[1]  patch for faster dot product in Numeric
     http://www.scipy.org/Members/aschmolck


-- 
Alexander Schmolck     Postgraduate Research Student
                       Department of Computer Science
                       University of Exeter
A.S...@gm...     http://www.dcs.ex.ac.uk/people/aschmolc/

Re: [SciPy-dev] Re: [Numpy-discussion] adding a .M attribute to the array.

[Pearu P. <pe...@ce...>]

<blink>Off topic warning</blink>

On 18 Mar 2002, Konrad Hinsen wrote:

> I see the main difference in distribution philosophy. NumPy is an
> add-on package to Python, which is in turn used by other add-on
> packages in a modular way. SciPy is rather a monolithic
> super-distribution for scientific users.
> 
> Personally I strongly favour the modular package approach, and in fact
> I haven't installed SciPy on my system for that reason, although I
> would be interested in some of its components.

Me too. In what I have contributed to SciPy, I have tried to follow this
modularity approach. Modularity is also important property from the
development point of view: it minimizes possible interference with
other unreleated modules and their bugs.

What I am trying to say here is that SciPy can (and should?, +1
from me) provide its components separately, though, currently only few of
its components seem to be available in that way without some changes.

Pearu

Re: [Numpy-discussion] adding a .M attribute to the array.

[Konrad H. <hi...@cn...>]

a.s...@gm... (A.Schmolck) writes:

> Linear algebra functionality is currently exclusively provided by `array` and
> libraries that operate on and return `array`s, but the computational and
> notational efficiency leaves to be desired (compared to e.g. Matlab) in some
> areas, importantly matrix multiplications (which are up to 40 times slower)
> and really awkward to write (and much more importantly, decipher afterwards).

Computational and notational efficiency are rather well separated,
fortunately. Both the current dot function and an hypothetical matrix
multiply operator could be implemented in straightforward C code or
using a high-performance library such as Atlas. In fact, this should
even be an installation choice in my opinion, as installing Atlas
isn't trivial on all machines (e.g. with some gcc versions), and I
consider it important for fundamental libraries that they work
everywhere easily, even if not optimally.

> My impression is that the best path also very much depends on the what the
> feature aspirations and divisions of labor of numpy/numarray and scipy are
> going to be. For example, scipy is really aimed at scientific users, which
> need performance, and are willing to buy it with inconvenience (like the

I see the main difference in distribution philosophy. NumPy is an
add-on package to Python, which is in turn used by other add-on
packages in a modular way. SciPy is rather a monolithic
super-distribution for scientific users.

Personally I strongly favour the modular package approach, and in fact
I haven't installed SciPy on my system for that reason, although I
would be interested in some of its components.

> algorithms, without doing any numerical computations on the arrays). So maybe
> (a subset of) numpy should make it into the python core (or an as yet

This has been discussed already, and it might well happen one day, but
not with the current NumPy implementation. Numarray looks like a much
better candidate, but isn't ready yet.

> In such a scenario, where numpy remains relatively general (and
> might even aim at incorporation into the core), it would be a no-no
> to bloat it with too much code aimed at improving efficiency
> (calling blas when possible, sparse storage etc.). On the other hand

The same approach as for XML could be used: a slim-line version in the
standard distribution that could be replaced by a high-efficiency
extended version for those who care.

> attractive solution do incorporate good matrix functionality (and
> possibly other improvements for hard core number crunchers) in scipy
> only (or at least limit the efficient _implementation_ of matrices
> to scipy, providing at only a pure python class or so in numpy). I'm

I'd love to have efficient matrices without having to install the
whole SciPy package!

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] adding a .M attribute to the array.

[<a.s...@gm...>]

[Sorry about the crossposting, but it also seemed relevant to both scipy and
numpy...]

Huaiyu Zhu <hua...@ya...> writes:
[...] 
> I'd like to hear any suggestions on how to proceed.  My own favorite would
> be to have separate array and matrix classes with easy but explicit
> conversions between them.  Without conversions, arrays and matrices would
> be completely independent semantically.  In other words, I'm mostly in
> favor of Konrad Hinsen's position, with the addition of using ~ operators
> for elementwise operations for matrix-like classes.  The PEP itself also
> discussed ideas of extending the meaning of ~ to other parts of Python for
> elementwise operations on aggregate types, but my impressions of people's
> impressions is that it has a better chance without that part.
> 

Well, from my impression of the previous discussions, the situation (both for
numpy and scipy) seems to boil down to me as follows:

Either `array` currently is too much of a matrix, or too little:

Linear algebra functionality is currently exclusively provided by `array` and
libraries that operate on and return `array`s, but the computational and
notational efficiency leaves to be desired (compared to e.g. Matlab) in some
areas, importantly matrix multiplications (which are up to 40 times slower)
and really awkward to write (and much more importantly, decipher afterwards).

So I think what one should really do is discuss the advantages and
disadvantages of the two possible ways out of this situation, namely
providing:

1) a new (efficient) `matrix` class/type (and appropriate libraries that
   operate on it) [The Matrix class that comes with Numeric is more some
   syntactic sugar wrapper -- AFAIK it's not use as a return type or argument
   in any of the functions that only make sense for arrays that are matrices].

2) the additional functionality that is needed for linear algebra in `array`
   and the libraries that operate on it.

(see [1] below for what I feel is currently missing and could be done
either in way 1) or 2))

I think it might be helpful to investigate these "macro"-issues before one
gets bogged down in discussions about operators (I admit these are not
entirely unrelated -- given that one of the reasons for the creation of a
Matrix type would be that '*' is already taken in 'array's and there is no way
to add a new operator without modifying the python core -- just for the record
and ignoring my own advice, _iff_ there is a chance of getting '~*' into the
language, I'd rather have '*' do the same for both matrices and arrays).
 
My impression is that the best path also very much depends on the what the
feature aspirations and divisions of labor of numpy/numarray and scipy are
going to be. For example, scipy is really aimed at scientific users, which
need performance, and are willing to buy it with inconvenience (like the
necessity to install other libraries on one's machine, most prominently atlas
and blas). The `array` type and the functions in `Numeric`, on the other hand,
potentially target a much wider community -- the efficient storage and
indexing facilities (rich comparisons, strides, the take, choose
etc. functions) make it highly useful for code that is not necessarily
numeric, (as an example I'm currently using it for feature selection
algorithms, without doing any numerical computations on the arrays). So maybe
(a subset of) numpy should make it into the python core (or an as yet
`non-existent sumo-distribution`) [BTW, I also wonder whether the python-core
array module could be superseded/merged with numpy's `array`? One potential
show stopper seems to be that it is e.g. `pop`able].

In such a scenario, where numpy remains relatively general (and might even aim
at incorporation into the core), it would be a no-no to bloat it with too much
code aimed at improving efficiency (calling blas when possible, sparse storage
etc.). On the other hand people who want to do serious numerical work will
need this -- and the scipy community already requires atlas etc. and targets a
more specialized audience. Under this consideration it might be an attractive
solution do incorporate good matrix functionality (and possibly other
improvements for hard core number crunchers) in scipy only (or at least limit
the efficient _implementation_ of matrices to scipy, providing at only a pure
python class or so in numpy). I'm not suggesting, BTW, to necessarily put all
of [1] into a single class -- it seems sensible to have a couple of subclasses
(for masked, sparse representations etc.) to `matrix` (maybe the parent-class
should even be a relatively naïve Numpy implementation, with the good stuff as
subclasses in scipy...).

In any event, creating a new matrix class/type would also mean that matrix
functionality in libraries should use and return this class (existing
libraries should presumably largely still operate on arrays for
backwards-compatibily (or both -- after a typecheck), and
some matrix operations are so useful that it makes sense to provide array
versions for them (e.g. dot) -- but on the whole it makes little sense to have
a computationally and space efficient matrix type if one has to cast it around
all the time). A `matrix` class is more specialized than an `array` and since
the operations one will often do on it are consequently more limited, I think
it should provide most important functionality as methods (rather than as
external functions; see [2] for a list of suggestions).

Approach 1) on the other hand would have the advantage that the current
interface would stay pretty much the same, and as long as 2D arrays can just
be regarded as matrices, there is no absolutely compelling reason not to stuff
everything into array (at least the scipy-version thereof).

Another important question to ask before deciding what to change how and if,
is obviously how many people in the scipy/numpy community do lots of linear
algebra (and how many deflectors from matlab etc. one could hope to win if one
spiced things up a bit for them...), but I would suppose there must be quite a
few (but I'm certainly biased ;).

Unfortunately, I've really got to do some work again now, but before I return
to number-crunching I'd like to say that I'd be happy to help with the
implementation of a matrix class/type in python (I guess a .py-prototype would
be helpful to start with, but ultimately a (subclassable) C(++)-type will be
called for, at least in scipy).

--alex

Footnotes: 
[1]  The required improvements for serious linear algebra seem to be:

     - optional use (atlas) blas routines for real and complex matrix, matrix
       `dot`s if atlas is available on the build machine (see
       http://www.scipy.org/Members/aschmolck for a patch -- it produces
       speedups of more than factor 40 for big matrices; I'd be willing to
       provide an equivalent patch for the scipy distribution if there is
       interest)
  
     - making sure that no unnecessary copies are created (e.g. when
       transposing a matrix to use it in `dot` -- AFAIK although the transpose
       itself only creates a new view, using it for dot results in a copy (but
       I might be wrong here ))
  
     - allowing more space efficient storage forms for special cases
       (e.g. sparse matrices, upper triangular etc.). IO libraries that can
       save and load such representations are also needed (methods and static
       methods might be a good choice to keep things transparent to the user).

     - providing a convinient and above all legible notation for common matrix
       operations (better than `dot(tranpose(A),B)` etc. -- possibilities
       include A * B.T or A ~* B.T or A * B ** T (by overloding __rpow__ as
       suggested in a previous post))

     - (in the case of a new `matrix` class): indexing functionality
       (e.g. `where`, `choose` etc. should be available without having to
       cast, e.g. for the common case that I want to set everything under a
       certain threshold to 0., I don't want to have to cast my sparse matrix
       to an array etc.)


[2]  What should a matrix class contain?

     - a dot operator (certainly eventually, but if there is a good chance to
       get ~* into python, maybe '*' should remain unimplemented till this can
       be decided)

     - most or all of what scipy's linalg module does

     - possibly IO, (reading  as a static method)

     - indexing (the like of take, choose etc. (some should maybe be functions
       or static methods))
  

-- 
Alexander Schmolck     Postgraduate Research Student
                       Department of Computer Science
                       University of Exeter
A.S...@gm...     http://www.dcs.ex.ac.uk/people/aschmolc/

Re: [Numpy-discussion] adding a .M attribute to the array.

[Huaiyu Z. <hua...@ya...>]

I'm a little late to this discussion, but it gives me a chance to read all
the existing comments.  I'd like to offer some background to one possible
solution.

On Thu, 7 Mar 2002, eric wrote:
> 
> http://python.sourceforge.net/peps/pep-0225.html
> 
>     This one proposes decorating the current binary ops with some
>     symbols to indicate that they have different behavior than
>     the standard binary ops.  This is similar to Matlab's use of
>     * for matrix multiplication and .* for element-wise multiplication
>     or to R's use of * for element-wise multiplication and %*% for
>     "object-wise" multiplication.
> 
>     It proposes prepending ~ to operators to change their behavior so
>     that ~* would become matrix multiply.
> 
>     The PEP is a little more general, but this gives the flavor.
> 
> My hunch is that some form of the second (perhaps drastically reduced) would
> meet with more success.  The suggested ~* or even the %*% operator are both
> palitable.  Such details can be decided later.  The question is whether there is
> sufficient interest to try and push the operator idea through?  It would take
> much longer than choosing something we can do ourselves (like .M), but the
> operator solution seems more desirable to me.
> 

I'm one of the coauthor of this PEP.  I'm very glad to see additional
interest in this proposal.  It is not just a proposal - there was actually
a patch made by Gregory Lielens for the ~op operators for Python 2.0.  It
redefines ~ so that it can be combined with + - * / ** to form new
operators.  It is quite ingenious in that all the original bitwise
operations on ~ alone are still valid.  The new operator can be assigned
any semantics with hooks like __tmul__ and __rtmul__.  The idea is that a
matrix class would define __mul__ so that * is matrix multiplication and
define __tmul__ so that ~* is elementwise operation.  There is a test
implementation on the MatPy homepage (matpy.sourceforge.net).

So what was holding it back?  Well, last time around when this was
discussed, it appears that most of the heavy weights in the Numeric
community favored either keeping the status quo, or using ~* symbol for
arrays.  We hoped to use the MatPy package as a test case to show that it
is possible to have two entirely different kinds of objects, where the
meaning of * and ~* are switched.  However, for various reasons I was not
able to act upon it for months, and Python evolved into 2.1 and 2.2.  I
never had much time to update the patch, and felt the attempt was futile
as 1) Python was evolving quite fast, 2) I did not heard much about this
issue since then.  I often feel guilty about the lapse.

Now it might be a good time to revive this proposal, as the idea of having
matrices and arrays with independent semantics but possibly related
implementation appears to be gaining some additional acceptance.  Some
ancillary issues that hindered the implementation at that time have also
been solved.  For example, using .I for inverse, .T for transpose, etc,
was costly because of the need to override __getattr__ and __coerce__,
making a matrix class less attractive in practice. These can now be
implemented efficiently using the new set/get mechanism.

I'd like to hear any suggestions on how to proceed.  My own favorite would
be to have separate array and matrix classes with easy but explicit
conversions between them.  Without conversions, arrays and matrices would
be completely independent semantically.  In other words, I'm mostly in
favor of Konrad Hinsen's position, with the addition of using ~ operators
for elementwise operations for matrix-like classes.  The PEP itself also
discussed ideas of extending the meaning of ~ to other parts of Python for
elementwise operations on aggregate types, but my impressions of people's
impressions is that it has a better chance without that part.

Huaiyu

[Numpy-discussion] fft_help

[Karshi <kar...@ut...>]

Hi all,

 How do use Matlab like " fft_shift" in NumPy?
Thanks

Re: [Numpy-discussion] Numerical Python 21.0 released

[Konrad H. <hi...@cn...>]

Gerard Vermeulen <gve...@po...> writes:

> Because all RPM based Linux distributions have subtle incompatibilities, it 
> is impossible to write a setup.py script that produces RPMs that will work on 
> all those distributions.

Or at least not simple.

I like the idea of providing RPMs for as many distributions as
possible, and I volunteer to participate in the effort. In fact, I
always make RPMs of all my Python-related packages already for my own
use (seven machines), for RedHat 7.x systems.

However, I have had difficulties with uploading to SourceForge for
almost a full year now, and I don't expect it to get better soon.
While building RPMs is a small job for me, uploading the result costs
me an enormous effort each time, and I am not willing to waste time on
that. I don't know how many people are in a similar situation, but
perhaps we could get more RPM packaging volunteers by opening an
RPM archive elsewhere.

Konrad.

Re: [Numpy-discussion] Numerical Python 21.0 released

[Gerard V. <gve...@po...>]

Because all RPM based Linux distributions have subtle incompatibilities, it 
is impossible to write a setup.py script that produces RPMs that will work on 
all those distributions.

Normally, RPMs build on a particular version of a particular distribution
will work on other systems with exactly the same version of the same 
distribution (Paul's setup is not "normal", because his Python interpreter
lives in his home directory).

If anybody wants to provide RPMs, please code distribution+version
in the name of the RPM. The RPM.README in the tar.gz explains how
to do this.

Gerard

On Thursday 14 March 2002 20:42, Paul F Dubois wrote:
> Gerard Vermeulen asked me to remove the RPMs from SourceForge, and I
> have done so. Apparently what is made by default is not useful except
> for me.
>
> I absolutely refuse to learn about RPMs; it is not useful for my job. If
> the RPM-cult wants this stuff to work then I need patches to setup.py
> that ALWAYS produce a suitable RPM, or a volunteer who will make and
> install such files for each release. I
> will run an automated test if supplied but it can't interfere with my
> system or require superuser privs, because I don't have that.
>
> I appreciate people trying to help and I'm sorry if it wasn't clear just
> how incompetent I intend to be on this.
>

RE: [Numpy-discussion] Numerical Python 21.0 released

[Paul F D. <pa...@pf...>]

Gerard Vermeulen asked me to remove the RPMs from SourceForge, and I
have done so. Apparently what is made by default is not useful except
for me.

I absolutely refuse to learn about RPMs; it is not useful for my job. If
the RPM-cult wants this stuff to work then I need patches to setup.py
that ALWAYS produce a suitable RPM, or a volunteer who will make and
install such files for each release. I 
will run an automated test if supplied but it can't interfere with my
system or require superuser privs, because I don't have that.

I appreciate people trying to help and I'm sorry if it wasn't clear just
how incompetent I intend to be on this.

RE: [Numpy-discussion] Plans for RandomArray support in numarray?

[Perry G. <pe...@st...>]

> 
> Are there any plans for supporting RandomArray in numarray type arrays? 
> 
> Amund
> http://www.idi.ntnu.no/~amundt/
> 
> 

Yes, we plan to support that and some version of the other existing
libraries available (e.g. linear algebra, fft...).

Now that we are done (mostly anyway) with the changes needed to
add safety checks to numarray, our short-term plans are:

1) Further changes to make numarray more backward compatible.
   (This is fairly minor work and probably involves less than
   a weeks work).

2) Documenting the C-API by:
   a) Adding the appropriate chapters to the numarray manual.
   b) providing examples of various kinds of C functions, e.g.,
      i) How to add a ufunc.
      ii) how to add simple C functions...
      iii) how to add more sophisticated C functions...
      iv) (possibly) how to use SWIG with numarray

3) To help us do 2) better, add some existing Numeric libraries.
   In particular:
   a) FFT
   b) RandomArray
   c) linear algebra
   Probably in that order.

Items 2) or 3) may or may not require new releases of numarray.
If no basic changes to numarray are required. We probably will
try to release the work piecemeal (e.g., updates to the manual,
examples, or libraries as add-ons). Hopefully, you will begin to
see results here starting in 2 to 4 weeks.

Perry

[Numpy-discussion] Plans for RandomArray support in numarray?

[Amund T. <am...@pv...>]

Are there any plans for supporting RandomArray in numarray type arrays? 

Amund
http://www.idi.ntnu.no/~amundt/

[Numpy-discussion] Numerical Python 21.0 released

[Paul F D. <pa...@pf...>]

Version 21.0 March 13, 2002
Fixed bugs:
    [ #482603 ] Memory leak in MA/Numeric/Python
                Reported by Reggie Dugard. Turned out to be 
                *two* memory leaks in one case in a routine in Numeric, 
                array_objectype. (Dubois)
    [ none    ] if vals was a null-array array([]) putmask and put would
                   crash.  Fixed with check.
    [ #469951 ] n = n1[0] gives array which shares dimension of n1
array. 
                  This causes bugs if shape of n1 is changed (n didn't
used
		  to have it's own dimensions array  (Travis Oliphant)
    [ #514588 ] MLab.cov(x,x) != MLab.cov(x) (Travis Oliphant)
    [ #518702 ] segfault when invalid typecode for asarray (Travis
Oliphant)
    [ #497530 ] MA __getitem__ prevents 0 len arrays (Reggie Duggard)
    [ #508363 ] outerproduct of noncontiguous arrays (Martin Wiechert)
    [ #513010 ] memory leak in comparisons (Byran Nollett)
    [ #512223 ] Character typecode not defined (Jochen Kupper)
    [ #500784 ] MLab.py diff error (anonymous, fixed by Dubois)
    [ #503741 ] accuracy of MLab.std(x) (Katsunori Waragai)
    [ #507568 ] overlapping copy a[2:5] = a[3:6]
                Change uses of memcpy to memmove which allows overlaps.
    [ numpy-Patches-499722 ] size of buffer created from array is bad
(Michel Sanner).
    [ #502186 ] a BUG in RandomArray.normal (introduced by last bug fix
in 20.3)      
               (Katsunori Waragai).
      
Fixed errors for Mac (Jack Jensen).

Make rpm's properly, better Windows installers. (Gerard Vermeulen)
    Added files setup.cfg; setup calculates rpm_install.sh to use
current Python.
    New setup.py, eliminate setup_all.py. Use os.path.join everywhere.
Revision in b6
    added file README.RPM, further improvements.

Implement true division operations for Python 2.2. (Bruce Sherwood)
    Note: true division of all integer types results in an array of
floats, 
    not doubles. This decision is arbitrary and there are arguments
either way,
    so users of this new feature should be aware that the decision may 
    change in the future. 

New functions in Numeric; they work on any sequence a that can be
converted to a
Numeric array. Similar change to average in MA. (Dubois)

    def rank (a):
        "Get the rank of a (the number of dimensions, not a matrix
rank)"

    def shape (a):
        "Get the shape of a"

    def size (a, axis=None):
        "Get the number of elements in a, or along a certain axis."
                            
    def average (a, axis=0, weights=None, returned = 0):
        """average(a, axis=0, weights=None)
           Computes average along indicated axis. 
           If axis is None, average over the entire array.
           Inputs can be integer or floating types; result is type
Float.
       
           If weights are given, result is:
               sum(a*weights)/(sum(weights))
           weights must have a's shape or be the 1-d with length the
size
           of a in the given axis. Integer weights are converted to
Float.

           Not supplying weights is equivalent to supply weights that
are
           all 1.
    
           If returned, return a tuple: the result and the sum of the
weights 
           or count of values. The shape of these two results will be
the same.
    
           raises ZeroDivisionError if appropriate when result is
scalar.
           (The version in MA does not -- it returns masked values).
        """

[Numpy-discussion] ANN: numarray-0.3

[Todd M. <jm...@st...>]

 Numarray 0.3
------------
Numarray is a Numeric replacement which features c-code generated from
python template scripts, the capacity to operate directly on arrays in
files, and improved type promotion semantics.

Numarray-0.3 incorporates safety checks to prevent crashing Python
when a user accidentally changes private variables in numarray.

The new safety checks ensure that:

1. Numarray C-functions are called with properly sized buffers.
2. Numarray C-functions are called with properly aligned buffers.
3. Parameters match the C-function in count and i/o direction.
4. The correct generic function wrapper is used to call each C-function.
5. All indices implied by the array strides are valid.

Failed checks result in python exceptions.

A new memory object fixes an unforunate limitation of the python
buffer object, namely the lack of guaranteed double aligned storage.

The largest generated source module, _ufuncmodule.c, has been
partitioned by data type into several smaller, more gcc-friendly
modules, e.g. _ufuncFloat64module.c.

The sort and argsort functions are fixed.
The dot function is fixed for 1D arrays.
Transpose, swapaxes, and reshape once again return views.

WHERE
-----------
Numarray-0.3 windows executable installers and source code tar ball is 
here:

http://sourceforge.net/project/showfiles.php?group_id=1369

Numarray is hosted by Source Forge in the same project which hosts Numeric:

http://sourceforge.net/projects/numpy/

The web page for Numarray information is at:

http://stsdas.stsci.edu/numarray/index.html

Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at
the Source Forge project for NumPy at:

http://sourceforge.net/tracker/?group_id=1369

REQUIREMENTS
--------------------------

numarray-0.3 requires Python 2.0 or greater.


AUTHORS, LICENSE
------------------------------

Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC
Hsu, Paul Barrett, and Phil Hodge at the Space Telescope Science
Institute.  Numarray is made available under a BSD-style License.  See
LICENSE.txt in the source distribution for details.


-- 
Todd Miller             jm...@st... <mailto:jm...@st...> 
<mailto:jm...@st...>
STSCI / SSG            (410) 338 4576


-- 
Todd Miller 			jm...@st...
STSCI / SSG			(410) 338 4576

Re: [Numpy-discussion] adding a .M attribute to the array.

[Konrad H. <hi...@cn...>]

"John J. Lee" <jj...@po...> writes:

> (Konrad also complains about Perl's nasty syntax.  This is frequently
> complained about, but do you really think the syntax is the problem --
> surely it's Perl's horribly complicated semantics that is the real issue?
> The syntax is just inconvenient, in comparison at least.  Sorry, a bit
> OT...)

It's both, of course. I don't really wish to decide which is worse,
especially not because I'd have to read more Perl code to reach such a
decision ;-)

But syntax is an issue for readability. There are some symbols that
are generally used as operators in computer languages, and I think
Python uses all of them already. Moreover, the general semantics are
quite uniform as well: * stands for multiplication, for example,
although the details of what multiplication means can vary.

Symbols like @ are not operators everywhere, and where they are there
is no uniform meaning attached to them, so they create confusion.
As a test, take a Python program and replace all * by @. It does
look weird.

Konrad.

RE: [Numpy-discussion] RE: Python 2.2 seriously crippled for numerical computation?

[Tim P. <ti...@co...>]

I hope the bogus underflow problem is fixed in CVS Python now.  Since my
platform didn't have the problem (and still doesn't, of course), we won't
know for sure until people try it and report back.

Python used to link with -lieee, but somebody changed it for a reason
history may have lost.  See comments attached to Huaiyu Zhu's bug report for
more on that:

<http://sf.net/tracker/?func=detail&aid=525705&group_id=5470&atid=105470>

If someone on a box that had the problem can build current CVS Python with
and without -lieee (I'm told it defaults to "without" today, although that
appears to mixed up with whether or not __fpu_control is found outside of
libieee (search for "ieee" in configure.in)), please try the following in a
shell:

import math
x = 1e200
y = 1/x

math.pow(x, 2)  # expect OverflowError
math.pow(y, 2)  # expect 0.

pow(x, 2)       # expect OverflowError
pow(y, 2)       # expect 0.

x**2            # expect OverflowError
y**2            # expect 0.

If those all work as annotated on a box that had the problem, please report
the success *as a comment to the bug report* (link above).  If one or more
still fail, likewise please give box details and paste the failures into a
comment on the bug report.

Thanks!

Re: [Numpy-discussion] big picture? One proposal

[Andrew P. L. <bs...@al...>]

On Fri, 8 Mar 2002, Pearu Peterson wrote:

> This is how I interpret the raised exception as behind the scenes matrix
> and array are the same (in the sense of data representation).

Matricies can have many different storage formats.  Sparse, Banded, Dense,
Triangular, etc.  Arrays and Matricies are the same behind the scenes *for
the moment*.  For examples of matrix storage formats, check the BLAST
Technical Forum documentation at netlib.  Unfortunately, www.netlib.org
appears to be down right now.

Locking the assumption that Matrix and Array are "the same behind the
scenes" into the main Python specification is not a good idea.

-a

Re: [Numpy-discussion] adding a .M attribute to the array.

[John J. L. <jj...@po...>]

On Thu, 7 Mar 2002, Konrad Hinsen wrote:

> "eric" <er...@en...> writes:
>
> > Matrix.Matrix objects. This attribute approach will work, but I
> > wonder if trying the "adding an operator to Python" approach one
> > more time would be worth while. At Python10 developer's day, Guido
[...]
> If you want to go the "operator way", the goal should rather be
> something like APL, with composite operators. Matrix multiplication
[...]

How about general operator - function equivalence, as explained here by
Alex Martelli?  The change is large in one sense, but it is conceptually
very simple:

http://groups.google.com/groups?q=operator+Martelli+Haskell+group:comp.lang.python&hl=en&selm=8t4dl301a4%40news2.newsguy.com&rnum=1

>   2 div 3
> or
>   div(2,3)
> or
>   2 `div 3
> [Haskell-ishly syntax-sugar note: Haskell lets you
> use any 2-operand function as an infix operator by
> just enclosing its name in ``; in Py3K, I think a
> single leading ` would suffice -- far nicer than the
> silly current use of ` for the rare need of repr --
> and we might also, with pleasing symmetry, let any
> operator be used as a normal function a la
>     `+(a,b)
> i.e., the ` marker could lexically switch functions
> to operators and operators to functions, without
> needing to 'import operator' and recall what the
> operator-name for a given operator IS...!-).  The
> priority and associativity of these infinitely
> many "new operators" could be fixed ones...].

Since GvR seems to have given up the idea of 'Py3K' in favour of gradual
changes, perhaps this is a real possibility?

Travis'

r = a.M * b.M

would then be written as

M = Numeric.matrixmultiply
r = a `M b

(Konrad also complains about Perl's nasty syntax.  This is frequently
complained about, but do you really think the syntax is the problem --
surely it's Perl's horribly complicated semantics that is the real issue?
The syntax is just inconvenient, in comparison at least.  Sorry, a bit
OT...)


John

Re: [Numpy-discussion] big picture? One proposal

[Konrad H. <hi...@cn...>]

> The Python core has long had at least 2 examples of operators which
> act as object constructors: 'j' which performs complex() and 'L' which
> performs long() (you can't get much more `pythonic' than a built-in
> type).  

Those are suffixes for constants, not operators. If they were
operators, you could apply them to variables - which you can't.

More importantly, the L suffix wouldn't even work as an operator,
as the preceding number might extend the range of integers before
it has a chance of being converted to a long integer.

> I would venture to say that the numeric community is pretty high up
> there in importance if not size, given the early appearance of the
> complex number type and strong math capacity not to mention GvR's

The complex type was introduced for the benefit of NumPy (I remember
it all too well, as I did the initial implementation), but after a
long discussion on the Python list, with many expressing
disapprovement because of its special-need status. I'd say it shows
the limits of what one can get accepted.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

[Numpy-discussion] RE: An historical precedent for matrix operation symbols

[Perry G. <pe...@st...>]

Paul Dubois writes:

[...]
Paul puts this very well and I agree with virtually everything he
says.
>
> One should not have any illusions: putting such operators on the array
> class is just expediency, a way to give the arrays a bit of a dual life.
> But a real matrix facility would have an abstract base class, be
> restricted to <= 2 dimensions, have realizations including symmetric,
> Hermitian, sparse, tridiagonal, yada yada yada.
>
A few comments on this point. I do think that this is correct. If a matrix
is stored as an array intrinsically, then constructing an array
representation
(e.g., array(b) where b is a matrix) would be very efficient since a new
array object consists of creating a new object that still uses the same
underlying data buffer the matrix object does. There is no significant
increase in memory. On the other hand, if a matrix class does use other
representations, particularly such as sparse or tridiagnonal, then there
naturally would be some cost to creating an array object since a new copy
of the data would be required. Having such various matrix representations
would certainly be useful (particularly sparse) but will certainly require
work to support (not something we (STScI) can sign up to do, but I hope
someone else is willing).

Perry

[Numpy-discussion] How the a**T trick works

[Paul F D. <pa...@pf...>]

I confess I admire the a**T suggestion for a notation for transpose(a).
The question was raised as to whether this really works and an empirical
proof was offered that it does. Here is how it works.

In a**T the first thing tried is to ask a.__pow__ if it can manage with
T as an argument. The array says, hmm, T is an instance of a class that
I never heard of, and it doesn't have an __array__ attribute to call in
order to make it an array. I pass. Then T's class' __rpow__ is given a
turn, and asked if it can do the job with a as an argument, which it
can.

This is how 2**a works, or 2 - a, too. The int type disavows any
knowledge of what to do, so the other operand gets a chance to save the
day. So this is a "trick" we use every day.

[Numpy-discussion] An historical precedent for matrix operation symbols

[Paul F D. <pa...@pf...>]

To sum up my previous postings: I prefer the "constructor" notation, not
the funny-attribute notation. However, I believe an efficient matrix
class can be done in Python on top of a  numeric array object. Shadow
classing just doesn't seem to be an overhead problem in most cases. The
coding for MA is very complicated because of its semantics are so
different; for Matrix it would be much less complicated.

When I designed Basis (1984) I was faced with the operator issue. The
vast majority of the operations were going to be elementwise but some
users would also want matrix multiply and the solution of linear
systems. (Aside: a/b meaning the solution x of bx = a, is best
calculated not as (b**-1) * a but by solving the system without forming
the inverse.)

My choice was: matrix multiply and divide were *!, /!. 

This was successful in two senses: the users found it easy to remember,
and you could implement it in the tokenizer or just in the grammar. I
chose to make it a token so as to forbid internal spaces, but for Python
it could be done without touching the tokenizer, and it doesn't use up
any new symbols.

Symbols are precious; when I designed Basis I had a keyboard map and I
would cross out the keys I had "used up". If I were Guido I would be
very reluctant to give up anything valuable like @ for our purposes.

One should not have any illusions: putting such operators on the array
class is just expediency, a way to give the arrays a bit of a dual life.
But a real matrix facility would have an abstract base class, be
restricted to <= 2 dimensions, have realizations including symmetric,
Hermitian, sparse, tridiagonal, yada yada yada.

Another aside: There are mathematical operations whose output type
depends not just on the input type but some of these other
considerations. This led me to believe that the Numpy approach is
essentially correct, that the type of the elements be variable rather
than having separate classes for each type.

Re: [Numpy-discussion] big picture? One proposal

[David B. <df...@mr...>]

On Fri, 8 Mar 2002, Konrad Hinsen wrote:

> > regarding to consider arrays as a mathematical concept too. Just in
> > mathematics I never (need to) use arrays in that way (my fields are
> > mathematical analysis, integrable systems, and not computer science nor
>
> I meant "mathematical" as a school of thought (going from the abstract
> to the concrete), not as a domain of research. I don't know any area
> of mathematics either that uses the array concept, but it is
> definitely common in computer science (as a structured collection of
> similar data). Image data is a good example.


Just my 2c worth: I count myself in the "mathematical" school despite
being a physicist. I look at matrices as having a specific algebra which,
for instance, cannot be easily made to apply to higher-dimensional arrays.
Therefore they are not just arrays looked at in a different way. For
object-oriented thinkers this means they are different objects.  They may
"inherit" a lot of attributes from arrays but are not arrays.

Another point to note is that a specific complaint earlier in the
thread was the computational inefficiency of using numpy arrays for
matrix-intensive operations. It seems to me that it would be far easier to
write an optimised set of code for matrices if they were known to be
a separate class. An example (which is probably not useful, but serves
for illustration) is that one could "cache" or delay transposes etc,
knowing that a matrix-multiply was likely to be about to come up. This
sort of thing would be more difficult if the result of the transpose would
have to be sensible when followed by a generic array operation.

David

RE: [Numpy-discussion] big picture? One proposal

[Pearu P. <pe...@ce...>]

On Fri, 8 Mar 2002, Perry Greenfield wrote:

> It's safer, but it isn't safe. Besides, one could still do this and raise
> exceptions on mixed types. Is the issue that people strongly want to do
> (if both a an b are arrays)
> 
> a.M * b  (or matrix(a) * b)
> 
> instead 
> 
> a.M * b.M (or matrix(a) * matrix(b))
> 
> to get matrix behavior?

Just to be clear, my suggestion consisted on the following points:

1) not to introduce any new type or concept such as matrix
2) to forget the current Matrix class
3) all objects are arrays, including a.M
4) a.M has a temporary bit set only for the following operation

So, with this setup there is no issue with mixed types at all and it is
easy to implement.
But if there will be introduced a new type, matrix, then this setup does
not work. The reason why I proposed the above setup was exactly because I
didn't like that a.M would return a different object type in the middle of
an expresssion.

Pearu

RE: [Numpy-discussion] big picture? One proposal

[Perry G. <pe...@st...>]

Pearu Peterson writes:

[...]
> 
> I find separating array and matrix instances (in a sense of raising
> exception when doing <matrix> <op> <array>) not a very simple solution:
>   New concepts are introduced that actually do not solve the simplicity
> problem of representing matrix operations. As I see it, they
> only introduce restrictions and the main assumption behind the rationale
> is that "users are dumb and they don't know what is best for them".
> This is how I interpret the raised exception as behind the scenes matrix
> and array are the same (in the sense of data representation). 
> 
I don't think the issue is whether users are "dumb" but rather, will it
be more or less transparent to them what is supposed to happen. Remember,
this particular proposal affects in no way the notational convenience
when operands are of the same type. It doesn't even affect the notational
convenience of most of the examples presented (e.g., a.M * b.M) as
long as the resulting operands are of the same type. It only affects
cases involving mixed types. Do we really want 

<matrix> * <array>   (yes, it would be possible to have one dominate over
<array> * <matrix>    the other always regardless of order)

to mean two different things for example? Will a user always be aware
that a module function returns arrays rather than matrices. Yes, users
ought to check the documentation, but they often don't or they 
misremember. The more I think about it the more I come to think it
really is better to be safer in this case. It will not be hard
for users to explicitly convert, nor should it be notationally
cumbersome. E.g. (just to use one the proposed options)

matrix(a) * b
a * array(b)

I don't see this as a big burden. I would rather do it this way myself
for my own code.

[...]
> (there have been two implementation approaches proposed for this: (i) a.M
> returns a Matrix instance, (ii) a.M returns the same array with a
> temporarily set bit saying that the following operation is somehow
> special).
> To me, this looks like a safe solution. Though it is a hack, at least it
> is simple and understandable in anywhere where it is used (having a * b
> where b can be either matrix or array, it is not predictable from just
> looking the code what the result will be -- not very pythonic indeed).
>
It's safer, but it isn't safe. Besides, one could still do this and raise
exceptions on mixed types. Is the issue that people strongly want to do
(if both a an b are arrays)

a.M * b  (or matrix(a) * b)

instead 

a.M * b.M (or matrix(a) * matrix(b))

to get matrix behavior?
 
Perry
 

Re: [Numpy-discussion] big picture? One proposal

[Konrad H. <hi...@cn...>]

> regarding to consider arrays as a mathematical concept too. Just in
> mathematics I never (need to) use arrays in that way (my fields are
> mathematical analysis, integrable systems, and not computer science nor

I meant "mathematical" as a school of thought (going from the abstract
to the concrete), not as a domain of research. I don't know any area
of mathematics either that uses the array concept, but it is
definitely common in computer science (as a structured collection of
similar data). Image data is a good example.

> something can be accomplished efficiently. To be honest, I don't like
> programming  in Python because it has a nice style, but because I can
> accomplish a lot with it in a very efficient way (and not only by using

I want both :-)

> > Moreover, I claim that Python *does* provide a good solution, it is
> > merely a very different one.
> 
> So, what is it?

Separate matrix and array objects, with computationally efficient but
explicit (verbose) interconversion.

> Yes, it works:
> >>> from Numeric import *
> >>> class T: __rpow__ = lambda s,o: transpose(o)
> ...
> >>> print array([[1,2],[3,4]]) ** T()
> [[1 3]
>  [2 4]]

Right, it works as long as the left argument doesn't try to do the
power operation itself.

> And I don't understand why it is abusive (because it is a different
> approach?). It's just an idea.

For me, "power" is a shorthand for repeated multiplication, with
certain properties attached to it. I have no problem with using the **
operator for something else, but then on different data types. The
idea that a**b could be completely different operations for the same a
as a function of b is not very appealing to me. In fact, the idea that
an operand instead of the operator defines the operation is not very
appealing to me.

There's also a more pragmatic objection which is purely technical, I
like to stay away from playing tricks with the binary operator type
coercion system in Python. Sooner or later it always bites back. And
the details have changed over Python releases, which is a
compatibility nightmare.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------