You can subscribe to this list here.
2000 |
Jan
(8) |
Feb
(49) |
Mar
(48) |
Apr
(28) |
May
(37) |
Jun
(28) |
Jul
(16) |
Aug
(16) |
Sep
(44) |
Oct
(61) |
Nov
(31) |
Dec
(24) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2001 |
Jan
(56) |
Feb
(54) |
Mar
(41) |
Apr
(71) |
May
(48) |
Jun
(32) |
Jul
(53) |
Aug
(91) |
Sep
(56) |
Oct
(33) |
Nov
(81) |
Dec
(54) |
2002 |
Jan
(72) |
Feb
(37) |
Mar
(126) |
Apr
(62) |
May
(34) |
Jun
(124) |
Jul
(36) |
Aug
(34) |
Sep
(60) |
Oct
(37) |
Nov
(23) |
Dec
(104) |
2003 |
Jan
(110) |
Feb
(73) |
Mar
(42) |
Apr
(8) |
May
(76) |
Jun
(14) |
Jul
(52) |
Aug
(26) |
Sep
(108) |
Oct
(82) |
Nov
(89) |
Dec
(94) |
2004 |
Jan
(117) |
Feb
(86) |
Mar
(75) |
Apr
(55) |
May
(75) |
Jun
(160) |
Jul
(152) |
Aug
(86) |
Sep
(75) |
Oct
(134) |
Nov
(62) |
Dec
(60) |
2005 |
Jan
(187) |
Feb
(318) |
Mar
(296) |
Apr
(205) |
May
(84) |
Jun
(63) |
Jul
(122) |
Aug
(59) |
Sep
(66) |
Oct
(148) |
Nov
(120) |
Dec
(70) |
2006 |
Jan
(460) |
Feb
(683) |
Mar
(589) |
Apr
(559) |
May
(445) |
Jun
(712) |
Jul
(815) |
Aug
(663) |
Sep
(559) |
Oct
(930) |
Nov
(373) |
Dec
|
From: <a.s...@gm...> - 2002-03-18 22:32:14
|
Konrad Hinsen <hi...@cn...> writes: > Computational and notational efficiency are rather well separated, > fortunately. Both the current dot function and an hypothetical matrix Yes, the only thing they have in common is that both are currently unsatisfactory (for matrix operations) in numpy, at least for my needs. Although I've solved my most pressing performance problems by patching Numeric [1], I'm obviously interested in a more official solution (i.e. one that is maintained by others :) [...] [order changed by me] > a.s...@gm... (A.Schmolck) writes: > > My impression is that the best path also very much depends on the what the > > feature aspirations and divisions of labor of numpy/numarray and scipy are ^^^^^^^ Darn, I made a confusing mistake -- this should read _future_. > > going to be. For example, scipy is really aimed at scientific users, which > > need performance, and are willing to buy it with inconvenience (like the > > I see the main difference in distribution philosophy. NumPy is an > add-on package to Python, which is in turn used by other add-on > packages in a modular way. SciPy is rather a monolithic > super-distribution for scientific users. > > Personally I strongly favour the modular package approach, and in fact > I haven't installed SciPy on my system for that reason, although I > would be interested in some of its components. [...] > The same approach as for XML could be used: a slim-line version in the > standard distribution that could be replaced by a high-efficiency > extended version for those who care. [...] I personally agree with all your above points -- if you have a look at our "dotblas"-patch mentioned earlier (see [1]), you will find that it aims to do provide that -- have dot run anywhere without a hassle but run (much) faster if the user is willing to install atlas. My main concern was that the argument should shift away a bit from syntactic and implementation details to what audiences and what needs numpy/numarray and are supposed to address and, in this light, how to best strike the balance between convinience for users and maitainers, speed and bloat, generality and efficiency etc. As an example, adding the dotblas patch [1] to Numeric is, I think more convinient for the users (granting a few assumptions (like that it actually works :) for the sake of the argument) -- it gives users that have atlas better-performance and those who don't won't (or at least shouldn't) notice. It is however inconvinient for the maintainers. Whether one should bother including it in this or some other way depends, among the obvious question of whether there is a better way to achieve what it does for both groups (like creating a dedicated Matrix class), also on what numpy is really supposed to achieve. I'm not entirely clear on that. For example I don't know how many numpy users deeply care about their matrix multiplications for big (1000x1000) matrices being 40 times faster. The monolithic approach is not entirely without its charms (remember python's "batteries included" jinggle)? Apart from convinience factors it also has the not unconsiderable advantage that people use _one_ standard module for a certain thing -- rather than 20 different solutions. This certainly helps to improve code quality. Not least because someone goes through the trouble of deciding what merrit's inclusion in the "Big Thing", possibly urging changes but at least almost certainly taking more time for evalutation than an indivdual programmer who just wants to get a certain job done. It also makes life easier for module writers -- they can rely on certain stuff being around (and don't have to reinvent the wheel, another potential improvement to code quality). As such it makes live easier for maintainers, as does the scipy commandment that you have to install atlas/lapack, full-stop (and if it doesn't run on your machine -- well at least it works fast for some people and that might well be better than working slow for everyone in this context). So, I think what's good really depends on what you're aiming at, that's why I'd like to know what users and developers think about these matters. My points regarding scipy and numpy/numarray were just one attempt at interpreting what these respective libraries try to/should/could attempt to be or become. Now, not being a developer for either of them (I've only submitted a few minor patches to scipy), I'm not in a particular good position to venture such interpretations, but I hoped that it would provoke other and more knowledgeable people to share their opinions and insights on this matter (as indeed you did). > I'd love to have efficient matrices without having to install the > whole SciPy package! Welcome to the linear algebra lobby group ;) yep, that would be nice but my impression was that the scipy folks are currently more concerned about performance issues than the numpy/numarray folks and I could live with either package providing what I want. Ideally , I'd like to see a slim core numarray, without any frills (and more streamlined to behave like standard python containers (e.g. indexing and type/casts behavior)) for the python core, something more enabled and efficient for numerics (including matrices!) as a seperate package (like the XML example you quote). And then maybe a bigger pre-bundled collection of (ideally rather modular) numerical libraries for really hard-core scientific users (maybe in the spirit of xemacs-packages and sumo-tar-balls -- no bloat if you don't need it, plenty of features in an instant if you do). Anyway, is there at least general agreement that there should be some new and wonderful matrix class (plus supporting libraries) somewhere (rather than souping up array)? alex Footnotes: [1] patch for faster dot product in Numeric http://www.scipy.org/Members/aschmolck -- Alexander Schmolck Postgraduate Research Student Department of Computer Science University of Exeter A.S...@gm... http://www.dcs.ex.ac.uk/people/aschmolc/ |
From: Pearu P. <pe...@ce...> - 2002-03-18 20:32:09
|
<blink>Off topic warning</blink> On 18 Mar 2002, Konrad Hinsen wrote: > I see the main difference in distribution philosophy. NumPy is an > add-on package to Python, which is in turn used by other add-on > packages in a modular way. SciPy is rather a monolithic > super-distribution for scientific users. > > Personally I strongly favour the modular package approach, and in fact > I haven't installed SciPy on my system for that reason, although I > would be interested in some of its components. Me too. In what I have contributed to SciPy, I have tried to follow this modularity approach. Modularity is also important property from the development point of view: it minimizes possible interference with other unreleated modules and their bugs. What I am trying to say here is that SciPy can (and should?, +1 from me) provide its components separately, though, currently only few of its components seem to be available in that way without some changes. Pearu |
From: Konrad H. <hi...@cn...> - 2002-03-18 15:59:27
|
a.s...@gm... (A.Schmolck) writes: > Linear algebra functionality is currently exclusively provided by `array` and > libraries that operate on and return `array`s, but the computational and > notational efficiency leaves to be desired (compared to e.g. Matlab) in some > areas, importantly matrix multiplications (which are up to 40 times slower) > and really awkward to write (and much more importantly, decipher afterwards). Computational and notational efficiency are rather well separated, fortunately. Both the current dot function and an hypothetical matrix multiply operator could be implemented in straightforward C code or using a high-performance library such as Atlas. In fact, this should even be an installation choice in my opinion, as installing Atlas isn't trivial on all machines (e.g. with some gcc versions), and I consider it important for fundamental libraries that they work everywhere easily, even if not optimally. > My impression is that the best path also very much depends on the what the > feature aspirations and divisions of labor of numpy/numarray and scipy are > going to be. For example, scipy is really aimed at scientific users, which > need performance, and are willing to buy it with inconvenience (like the I see the main difference in distribution philosophy. NumPy is an add-on package to Python, which is in turn used by other add-on packages in a modular way. SciPy is rather a monolithic super-distribution for scientific users. Personally I strongly favour the modular package approach, and in fact I haven't installed SciPy on my system for that reason, although I would be interested in some of its components. > algorithms, without doing any numerical computations on the arrays). So maybe > (a subset of) numpy should make it into the python core (or an as yet This has been discussed already, and it might well happen one day, but not with the current NumPy implementation. Numarray looks like a much better candidate, but isn't ready yet. > In such a scenario, where numpy remains relatively general (and > might even aim at incorporation into the core), it would be a no-no > to bloat it with too much code aimed at improving efficiency > (calling blas when possible, sparse storage etc.). On the other hand The same approach as for XML could be used: a slim-line version in the standard distribution that could be replaced by a high-efficiency extended version for those who care. > attractive solution do incorporate good matrix functionality (and > possibly other improvements for hard core number crunchers) in scipy > only (or at least limit the efficient _implementation_ of matrices > to scipy, providing at only a pure python class or so in numpy). I'm I'd love to have efficient matrices without having to install the whole SciPy package! Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- |
From: <a.s...@gm...> - 2002-03-18 14:55:00
|
[Sorry about the crossposting, but it also seemed relevant to both scipy and numpy...] Huaiyu Zhu <hua...@ya...> writes: [...] > I'd like to hear any suggestions on how to proceed. My own favorite would > be to have separate array and matrix classes with easy but explicit > conversions between them. Without conversions, arrays and matrices would > be completely independent semantically. In other words, I'm mostly in > favor of Konrad Hinsen's position, with the addition of using ~ operators > for elementwise operations for matrix-like classes. The PEP itself also > discussed ideas of extending the meaning of ~ to other parts of Python for > elementwise operations on aggregate types, but my impressions of people's > impressions is that it has a better chance without that part. > Well, from my impression of the previous discussions, the situation (both for numpy and scipy) seems to boil down to me as follows: Either `array` currently is too much of a matrix, or too little: Linear algebra functionality is currently exclusively provided by `array` and libraries that operate on and return `array`s, but the computational and notational efficiency leaves to be desired (compared to e.g. Matlab) in some areas, importantly matrix multiplications (which are up to 40 times slower) and really awkward to write (and much more importantly, decipher afterwards). So I think what one should really do is discuss the advantages and disadvantages of the two possible ways out of this situation, namely providing: 1) a new (efficient) `matrix` class/type (and appropriate libraries that operate on it) [The Matrix class that comes with Numeric is more some syntactic sugar wrapper -- AFAIK it's not use as a return type or argument in any of the functions that only make sense for arrays that are matrices]. 2) the additional functionality that is needed for linear algebra in `array` and the libraries that operate on it. (see [1] below for what I feel is currently missing and could be done either in way 1) or 2)) I think it might be helpful to investigate these "macro"-issues before one gets bogged down in discussions about operators (I admit these are not entirely unrelated -- given that one of the reasons for the creation of a Matrix type would be that '*' is already taken in 'array's and there is no way to add a new operator without modifying the python core -- just for the record and ignoring my own advice, _iff_ there is a chance of getting '~*' into the language, I'd rather have '*' do the same for both matrices and arrays). My impression is that the best path also very much depends on the what the feature aspirations and divisions of labor of numpy/numarray and scipy are going to be. For example, scipy is really aimed at scientific users, which need performance, and are willing to buy it with inconvenience (like the necessity to install other libraries on one's machine, most prominently atlas and blas). The `array` type and the functions in `Numeric`, on the other hand, potentially target a much wider community -- the efficient storage and indexing facilities (rich comparisons, strides, the take, choose etc. functions) make it highly useful for code that is not necessarily numeric, (as an example I'm currently using it for feature selection algorithms, without doing any numerical computations on the arrays). So maybe (a subset of) numpy should make it into the python core (or an as yet `non-existent sumo-distribution`) [BTW, I also wonder whether the python-core array module could be superseded/merged with numpy's `array`? One potential show stopper seems to be that it is e.g. `pop`able]. In such a scenario, where numpy remains relatively general (and might even aim at incorporation into the core), it would be a no-no to bloat it with too much code aimed at improving efficiency (calling blas when possible, sparse storage etc.). On the other hand people who want to do serious numerical work will need this -- and the scipy community already requires atlas etc. and targets a more specialized audience. Under this consideration it might be an attractive solution do incorporate good matrix functionality (and possibly other improvements for hard core number crunchers) in scipy only (or at least limit the efficient _implementation_ of matrices to scipy, providing at only a pure python class or so in numpy). I'm not suggesting, BTW, to necessarily put all of [1] into a single class -- it seems sensible to have a couple of subclasses (for masked, sparse representations etc.) to `matrix` (maybe the parent-class should even be a relatively naïve Numpy implementation, with the good stuff as subclasses in scipy...). In any event, creating a new matrix class/type would also mean that matrix functionality in libraries should use and return this class (existing libraries should presumably largely still operate on arrays for backwards-compatibily (or both -- after a typecheck), and some matrix operations are so useful that it makes sense to provide array versions for them (e.g. dot) -- but on the whole it makes little sense to have a computationally and space efficient matrix type if one has to cast it around all the time). A `matrix` class is more specialized than an `array` and since the operations one will often do on it are consequently more limited, I think it should provide most important functionality as methods (rather than as external functions; see [2] for a list of suggestions). Approach 1) on the other hand would have the advantage that the current interface would stay pretty much the same, and as long as 2D arrays can just be regarded as matrices, there is no absolutely compelling reason not to stuff everything into array (at least the scipy-version thereof). Another important question to ask before deciding what to change how and if, is obviously how many people in the scipy/numpy community do lots of linear algebra (and how many deflectors from matlab etc. one could hope to win if one spiced things up a bit for them...), but I would suppose there must be quite a few (but I'm certainly biased ;). Unfortunately, I've really got to do some work again now, but before I return to number-crunching I'd like to say that I'd be happy to help with the implementation of a matrix class/type in python (I guess a .py-prototype would be helpful to start with, but ultimately a (subclassable) C(++)-type will be called for, at least in scipy). --alex Footnotes: [1] The required improvements for serious linear algebra seem to be: - optional use (atlas) blas routines for real and complex matrix, matrix `dot`s if atlas is available on the build machine (see http://www.scipy.org/Members/aschmolck for a patch -- it produces speedups of more than factor 40 for big matrices; I'd be willing to provide an equivalent patch for the scipy distribution if there is interest) - making sure that no unnecessary copies are created (e.g. when transposing a matrix to use it in `dot` -- AFAIK although the transpose itself only creates a new view, using it for dot results in a copy (but I might be wrong here )) - allowing more space efficient storage forms for special cases (e.g. sparse matrices, upper triangular etc.). IO libraries that can save and load such representations are also needed (methods and static methods might be a good choice to keep things transparent to the user). - providing a convinient and above all legible notation for common matrix operations (better than `dot(tranpose(A),B)` etc. -- possibilities include A * B.T or A ~* B.T or A * B ** T (by overloding __rpow__ as suggested in a previous post)) - (in the case of a new `matrix` class): indexing functionality (e.g. `where`, `choose` etc. should be available without having to cast, e.g. for the common case that I want to set everything under a certain threshold to 0., I don't want to have to cast my sparse matrix to an array etc.) [2] What should a matrix class contain? - a dot operator (certainly eventually, but if there is a good chance to get ~* into python, maybe '*' should remain unimplemented till this can be decided) - most or all of what scipy's linalg module does - possibly IO, (reading as a static method) - indexing (the like of take, choose etc. (some should maybe be functions or static methods)) -- Alexander Schmolck Postgraduate Research Student Department of Computer Science University of Exeter A.S...@gm... http://www.dcs.ex.ac.uk/people/aschmolc/ |
From: Huaiyu Z. <hua...@ya...> - 2002-03-18 08:39:56
|
I'm a little late to this discussion, but it gives me a chance to read all the existing comments. I'd like to offer some background to one possible solution. On Thu, 7 Mar 2002, eric wrote: > > http://python.sourceforge.net/peps/pep-0225.html > > This one proposes decorating the current binary ops with some > symbols to indicate that they have different behavior than > the standard binary ops. This is similar to Matlab's use of > * for matrix multiplication and .* for element-wise multiplication > or to R's use of * for element-wise multiplication and %*% for > "object-wise" multiplication. > > It proposes prepending ~ to operators to change their behavior so > that ~* would become matrix multiply. > > The PEP is a little more general, but this gives the flavor. > > My hunch is that some form of the second (perhaps drastically reduced) would > meet with more success. The suggested ~* or even the %*% operator are both > palitable. Such details can be decided later. The question is whether there is > sufficient interest to try and push the operator idea through? It would take > much longer than choosing something we can do ourselves (like .M), but the > operator solution seems more desirable to me. > I'm one of the coauthor of this PEP. I'm very glad to see additional interest in this proposal. It is not just a proposal - there was actually a patch made by Gregory Lielens for the ~op operators for Python 2.0. It redefines ~ so that it can be combined with + - * / ** to form new operators. It is quite ingenious in that all the original bitwise operations on ~ alone are still valid. The new operator can be assigned any semantics with hooks like __tmul__ and __rtmul__. The idea is that a matrix class would define __mul__ so that * is matrix multiplication and define __tmul__ so that ~* is elementwise operation. There is a test implementation on the MatPy homepage (matpy.sourceforge.net). So what was holding it back? Well, last time around when this was discussed, it appears that most of the heavy weights in the Numeric community favored either keeping the status quo, or using ~* symbol for arrays. We hoped to use the MatPy package as a test case to show that it is possible to have two entirely different kinds of objects, where the meaning of * and ~* are switched. However, for various reasons I was not able to act upon it for months, and Python evolved into 2.1 and 2.2. I never had much time to update the patch, and felt the attempt was futile as 1) Python was evolving quite fast, 2) I did not heard much about this issue since then. I often feel guilty about the lapse. Now it might be a good time to revive this proposal, as the idea of having matrices and arrays with independent semantics but possibly related implementation appears to be gaining some additional acceptance. Some ancillary issues that hindered the implementation at that time have also been solved. For example, using .I for inverse, .T for transpose, etc, was costly because of the need to override __getattr__ and __coerce__, making a matrix class less attractive in practice. These can now be implemented efficiently using the new set/get mechanism. I'd like to hear any suggestions on how to proceed. My own favorite would be to have separate array and matrix classes with easy but explicit conversions between them. Without conversions, arrays and matrices would be completely independent semantically. In other words, I'm mostly in favor of Konrad Hinsen's position, with the addition of using ~ operators for elementwise operations for matrix-like classes. The PEP itself also discussed ideas of extending the meaning of ~ to other parts of Python for elementwise operations on aggregate types, but my impressions of people's impressions is that it has a better chance without that part. Huaiyu |
From: Karshi <kar...@ut...> - 2002-03-16 04:20:28
|
Hi all, How do use Matlab like " fft_shift" in NumPy? Thanks |
From: Konrad H. <hi...@cn...> - 2002-03-15 09:34:46
|
Gerard Vermeulen <gve...@po...> writes: > Because all RPM based Linux distributions have subtle incompatibilities, it > is impossible to write a setup.py script that produces RPMs that will work on > all those distributions. Or at least not simple. I like the idea of providing RPMs for as many distributions as possible, and I volunteer to participate in the effort. In fact, I always make RPMs of all my Python-related packages already for my own use (seven machines), for RedHat 7.x systems. However, I have had difficulties with uploading to SourceForge for almost a full year now, and I don't expect it to get better soon. While building RPMs is a small job for me, uploading the result costs me an enormous effort each time, and I am not willing to waste time on that. I don't know how many people are in a similar situation, but perhaps we could get more RPM packaging volunteers by opening an RPM archive elsewhere. Konrad. |
From: Gerard V. <gve...@po...> - 2002-03-15 07:40:42
|
Because all RPM based Linux distributions have subtle incompatibilities, it is impossible to write a setup.py script that produces RPMs that will work on all those distributions. Normally, RPMs build on a particular version of a particular distribution will work on other systems with exactly the same version of the same distribution (Paul's setup is not "normal", because his Python interpreter lives in his home directory). If anybody wants to provide RPMs, please code distribution+version in the name of the RPM. The RPM.README in the tar.gz explains how to do this. Gerard On Thursday 14 March 2002 20:42, Paul F Dubois wrote: > Gerard Vermeulen asked me to remove the RPMs from SourceForge, and I > have done so. Apparently what is made by default is not useful except > for me. > > I absolutely refuse to learn about RPMs; it is not useful for my job. If > the RPM-cult wants this stuff to work then I need patches to setup.py > that ALWAYS produce a suitable RPM, or a volunteer who will make and > install such files for each release. I > will run an automated test if supplied but it can't interfere with my > system or require superuser privs, because I don't have that. > > I appreciate people trying to help and I'm sorry if it wasn't clear just > how incompetent I intend to be on this. > |
From: Paul F D. <pa...@pf...> - 2002-03-14 19:42:50
|
Gerard Vermeulen asked me to remove the RPMs from SourceForge, and I have done so. Apparently what is made by default is not useful except for me. I absolutely refuse to learn about RPMs; it is not useful for my job. If the RPM-cult wants this stuff to work then I need patches to setup.py that ALWAYS produce a suitable RPM, or a volunteer who will make and install such files for each release. I will run an automated test if supplied but it can't interfere with my system or require superuser privs, because I don't have that. I appreciate people trying to help and I'm sorry if it wasn't clear just how incompetent I intend to be on this. |
From: Perry G. <pe...@st...> - 2002-03-14 14:29:40
|
> > Are there any plans for supporting RandomArray in numarray type arrays? > > Amund > http://www.idi.ntnu.no/~amundt/ > > Yes, we plan to support that and some version of the other existing libraries available (e.g. linear algebra, fft...). Now that we are done (mostly anyway) with the changes needed to add safety checks to numarray, our short-term plans are: 1) Further changes to make numarray more backward compatible. (This is fairly minor work and probably involves less than a weeks work). 2) Documenting the C-API by: a) Adding the appropriate chapters to the numarray manual. b) providing examples of various kinds of C functions, e.g., i) How to add a ufunc. ii) how to add simple C functions... iii) how to add more sophisticated C functions... iv) (possibly) how to use SWIG with numarray 3) To help us do 2) better, add some existing Numeric libraries. In particular: a) FFT b) RandomArray c) linear algebra Probably in that order. Items 2) or 3) may or may not require new releases of numarray. If no basic changes to numarray are required. We probably will try to release the work piecemeal (e.g., updates to the manual, examples, or libraries as add-ons). Hopefully, you will begin to see results here starting in 2 to 4 weeks. Perry |
From: Amund T. <am...@pv...> - 2002-03-14 13:35:54
|
Are there any plans for supporting RandomArray in numarray type arrays? Amund http://www.idi.ntnu.no/~amundt/ |
From: Paul F D. <pa...@pf...> - 2002-03-13 23:01:50
|
Version 21.0 March 13, 2002 Fixed bugs: [ #482603 ] Memory leak in MA/Numeric/Python Reported by Reggie Dugard. Turned out to be *two* memory leaks in one case in a routine in Numeric, array_objectype. (Dubois) [ none ] if vals was a null-array array([]) putmask and put would crash. Fixed with check. [ #469951 ] n = n1[0] gives array which shares dimension of n1 array. This causes bugs if shape of n1 is changed (n didn't used to have it's own dimensions array (Travis Oliphant) [ #514588 ] MLab.cov(x,x) != MLab.cov(x) (Travis Oliphant) [ #518702 ] segfault when invalid typecode for asarray (Travis Oliphant) [ #497530 ] MA __getitem__ prevents 0 len arrays (Reggie Duggard) [ #508363 ] outerproduct of noncontiguous arrays (Martin Wiechert) [ #513010 ] memory leak in comparisons (Byran Nollett) [ #512223 ] Character typecode not defined (Jochen Kupper) [ #500784 ] MLab.py diff error (anonymous, fixed by Dubois) [ #503741 ] accuracy of MLab.std(x) (Katsunori Waragai) [ #507568 ] overlapping copy a[2:5] = a[3:6] Change uses of memcpy to memmove which allows overlaps. [ numpy-Patches-499722 ] size of buffer created from array is bad (Michel Sanner). [ #502186 ] a BUG in RandomArray.normal (introduced by last bug fix in 20.3) (Katsunori Waragai). Fixed errors for Mac (Jack Jensen). Make rpm's properly, better Windows installers. (Gerard Vermeulen) Added files setup.cfg; setup calculates rpm_install.sh to use current Python. New setup.py, eliminate setup_all.py. Use os.path.join everywhere. Revision in b6 added file README.RPM, further improvements. Implement true division operations for Python 2.2. (Bruce Sherwood) Note: true division of all integer types results in an array of floats, not doubles. This decision is arbitrary and there are arguments either way, so users of this new feature should be aware that the decision may change in the future. New functions in Numeric; they work on any sequence a that can be converted to a Numeric array. Similar change to average in MA. (Dubois) def rank (a): "Get the rank of a (the number of dimensions, not a matrix rank)" def shape (a): "Get the shape of a" def size (a, axis=None): "Get the number of elements in a, or along a certain axis." def average (a, axis=0, weights=None, returned = 0): """average(a, axis=0, weights=None) Computes average along indicated axis. If axis is None, average over the entire array. Inputs can be integer or floating types; result is type Float. If weights are given, result is: sum(a*weights)/(sum(weights)) weights must have a's shape or be the 1-d with length the size of a in the given axis. Integer weights are converted to Float. Not supplying weights is equivalent to supply weights that are all 1. If returned, return a tuple: the result and the sum of the weights or count of values. The shape of these two results will be the same. raises ZeroDivisionError if appropriate when result is scalar. (The version in MA does not -- it returns masked values). """ |
From: Todd M. <jm...@st...> - 2002-03-12 22:20:46
|
Numarray 0.3 ------------ Numarray is a Numeric replacement which features c-code generated from python template scripts, the capacity to operate directly on arrays in files, and improved type promotion semantics. Numarray-0.3 incorporates safety checks to prevent crashing Python when a user accidentally changes private variables in numarray. The new safety checks ensure that: 1. Numarray C-functions are called with properly sized buffers. 2. Numarray C-functions are called with properly aligned buffers. 3. Parameters match the C-function in count and i/o direction. 4. The correct generic function wrapper is used to call each C-function. 5. All indices implied by the array strides are valid. Failed checks result in python exceptions. A new memory object fixes an unforunate limitation of the python buffer object, namely the lack of guaranteed double aligned storage. The largest generated source module, _ufuncmodule.c, has been partitioned by data type into several smaller, more gcc-friendly modules, e.g. _ufuncFloat64module.c. The sort and argsort functions are fixed. The dot function is fixed for 1D arrays. Transpose, swapaxes, and reshape once again return views. WHERE ----------- Numarray-0.3 windows executable installers and source code tar ball is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://stsdas.stsci.edu/numarray/index.html Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS -------------------------- numarray-0.3 requires Python 2.0 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, and Phil Hodge at the Space Telescope Science Institute. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jm...@st... <mailto:jm...@st...> <mailto:jm...@st...> STSCI / SSG (410) 338 4576 -- Todd Miller jm...@st... STSCI / SSG (410) 338 4576 |
From: Konrad H. <hi...@cn...> - 2002-03-09 22:37:17
|
"John J. Lee" <jj...@po...> writes: > (Konrad also complains about Perl's nasty syntax. This is frequently > complained about, but do you really think the syntax is the problem -- > surely it's Perl's horribly complicated semantics that is the real issue? > The syntax is just inconvenient, in comparison at least. Sorry, a bit > OT...) It's both, of course. I don't really wish to decide which is worse, especially not because I'd have to read more Perl code to reach such a decision ;-) But syntax is an issue for readability. There are some symbols that are generally used as operators in computer languages, and I think Python uses all of them already. Moreover, the general semantics are quite uniform as well: * stands for multiplication, for example, although the details of what multiplication means can vary. Symbols like @ are not operators everywhere, and where they are there is no uniform meaning attached to them, so they create confusion. As a test, take a Python program and replace all * by @. It does look weird. Konrad. |
From: Tim P. <ti...@co...> - 2002-03-09 05:17:35
|
I hope the bogus underflow problem is fixed in CVS Python now. Since my platform didn't have the problem (and still doesn't, of course), we won't know for sure until people try it and report back. Python used to link with -lieee, but somebody changed it for a reason history may have lost. See comments attached to Huaiyu Zhu's bug report for more on that: <http://sf.net/tracker/?func=detail&aid=525705&group_id=5470&atid=105470> If someone on a box that had the problem can build current CVS Python with and without -lieee (I'm told it defaults to "without" today, although that appears to mixed up with whether or not __fpu_control is found outside of libieee (search for "ieee" in configure.in)), please try the following in a shell: import math x = 1e200 y = 1/x math.pow(x, 2) # expect OverflowError math.pow(y, 2) # expect 0. pow(x, 2) # expect OverflowError pow(y, 2) # expect 0. x**2 # expect OverflowError y**2 # expect 0. If those all work as annotated on a box that had the problem, please report the success *as a comment to the bug report* (link above). If one or more still fail, likewise please give box details and paste the failures into a comment on the bug report. Thanks! |
From: Andrew P. L. <bs...@al...> - 2002-03-08 23:35:19
|
On Fri, 8 Mar 2002, Pearu Peterson wrote: > This is how I interpret the raised exception as behind the scenes matrix > and array are the same (in the sense of data representation). Matricies can have many different storage formats. Sparse, Banded, Dense, Triangular, etc. Arrays and Matricies are the same behind the scenes *for the moment*. For examples of matrix storage formats, check the BLAST Technical Forum documentation at netlib. Unfortunately, www.netlib.org appears to be down right now. Locking the assumption that Matrix and Array are "the same behind the scenes" into the main Python specification is not a good idea. -a |
From: John J. L. <jj...@po...> - 2002-03-08 21:41:18
|
On Thu, 7 Mar 2002, Konrad Hinsen wrote: > "eric" <er...@en...> writes: > > > Matrix.Matrix objects. This attribute approach will work, but I > > wonder if trying the "adding an operator to Python" approach one > > more time would be worth while. At Python10 developer's day, Guido [...] > If you want to go the "operator way", the goal should rather be > something like APL, with composite operators. Matrix multiplication [...] How about general operator - function equivalence, as explained here by Alex Martelli? The change is large in one sense, but it is conceptually very simple: http://groups.google.com/groups?q=operator+Martelli+Haskell+group:comp.lang.python&hl=en&selm=8t4dl301a4%40news2.newsguy.com&rnum=1 > 2 div 3 > or > div(2,3) > or > 2 `div 3 > [Haskell-ishly syntax-sugar note: Haskell lets you > use any 2-operand function as an infix operator by > just enclosing its name in ``; in Py3K, I think a > single leading ` would suffice -- far nicer than the > silly current use of ` for the rare need of repr -- > and we might also, with pleasing symmetry, let any > operator be used as a normal function a la > `+(a,b) > i.e., the ` marker could lexically switch functions > to operators and operators to functions, without > needing to 'import operator' and recall what the > operator-name for a given operator IS...!-). The > priority and associativity of these infinitely > many "new operators" could be fixed ones...]. Since GvR seems to have given up the idea of 'Py3K' in favour of gradual changes, perhaps this is a real possibility? Travis' r = a.M * b.M would then be written as M = Numeric.matrixmultiply r = a `M b (Konrad also complains about Perl's nasty syntax. This is frequently complained about, but do you really think the syntax is the problem -- surely it's Perl's horribly complicated semantics that is the real issue? The syntax is just inconvenient, in comparison at least. Sorry, a bit OT...) John |
From: Konrad H. <hi...@cn...> - 2002-03-08 17:55:08
|
> The Python core has long had at least 2 examples of operators which > act as object constructors: 'j' which performs complex() and 'L' which > performs long() (you can't get much more `pythonic' than a built-in > type). Those are suffixes for constants, not operators. If they were operators, you could apply them to variables - which you can't. More importantly, the L suffix wouldn't even work as an operator, as the preceding number might extend the range of integers before it has a chance of being converted to a long integer. > I would venture to say that the numeric community is pretty high up > there in importance if not size, given the early appearance of the > complex number type and strong math capacity not to mention GvR's The complex type was introduced for the benefit of NumPy (I remember it all too well, as I did the initial implementation), but after a long discussion on the Python list, with many expressing disapprovement because of its special-need status. I'd say it shows the limits of what one can get accepted. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- |
From: Perry G. <pe...@st...> - 2002-03-08 16:47:43
|
Paul Dubois writes: [...] Paul puts this very well and I agree with virtually everything he says. > > One should not have any illusions: putting such operators on the array > class is just expediency, a way to give the arrays a bit of a dual life. > But a real matrix facility would have an abstract base class, be > restricted to <= 2 dimensions, have realizations including symmetric, > Hermitian, sparse, tridiagonal, yada yada yada. > A few comments on this point. I do think that this is correct. If a matrix is stored as an array intrinsically, then constructing an array representation (e.g., array(b) where b is a matrix) would be very efficient since a new array object consists of creating a new object that still uses the same underlying data buffer the matrix object does. There is no significant increase in memory. On the other hand, if a matrix class does use other representations, particularly such as sparse or tridiagnonal, then there naturally would be some cost to creating an array object since a new copy of the data would be required. Having such various matrix representations would certainly be useful (particularly sparse) but will certainly require work to support (not something we (STScI) can sign up to do, but I hope someone else is willing). Perry |
From: Paul F D. <pa...@pf...> - 2002-03-08 16:45:17
|
I confess I admire the a**T suggestion for a notation for transpose(a). The question was raised as to whether this really works and an empirical proof was offered that it does. Here is how it works. In a**T the first thing tried is to ask a.__pow__ if it can manage with T as an argument. The array says, hmm, T is an instance of a class that I never heard of, and it doesn't have an __array__ attribute to call in order to make it an array. I pass. Then T's class' __rpow__ is given a turn, and asked if it can do the job with a as an argument, which it can. This is how 2**a works, or 2 - a, too. The int type disavows any knowledge of what to do, so the other operand gets a chance to save the day. So this is a "trick" we use every day. |
From: Paul F D. <pa...@pf...> - 2002-03-08 16:24:41
|
To sum up my previous postings: I prefer the "constructor" notation, not the funny-attribute notation. However, I believe an efficient matrix class can be done in Python on top of a numeric array object. Shadow classing just doesn't seem to be an overhead problem in most cases. The coding for MA is very complicated because of its semantics are so different; for Matrix it would be much less complicated. When I designed Basis (1984) I was faced with the operator issue. The vast majority of the operations were going to be elementwise but some users would also want matrix multiply and the solution of linear systems. (Aside: a/b meaning the solution x of bx = a, is best calculated not as (b**-1) * a but by solving the system without forming the inverse.) My choice was: matrix multiply and divide were *!, /!. This was successful in two senses: the users found it easy to remember, and you could implement it in the tokenizer or just in the grammar. I chose to make it a token so as to forbid internal spaces, but for Python it could be done without touching the tokenizer, and it doesn't use up any new symbols. Symbols are precious; when I designed Basis I had a keyboard map and I would cross out the keys I had "used up". If I were Guido I would be very reluctant to give up anything valuable like @ for our purposes. One should not have any illusions: putting such operators on the array class is just expediency, a way to give the arrays a bit of a dual life. But a real matrix facility would have an abstract base class, be restricted to <= 2 dimensions, have realizations including symmetric, Hermitian, sparse, tridiagonal, yada yada yada. Another aside: There are mathematical operations whose output type depends not just on the input type but some of these other considerations. This led me to believe that the Numpy approach is essentially correct, that the type of the elements be variable rather than having separate classes for each type. |
From: David B. <df...@mr...> - 2002-03-08 16:04:40
|
On Fri, 8 Mar 2002, Konrad Hinsen wrote: > > regarding to consider arrays as a mathematical concept too. Just in > > mathematics I never (need to) use arrays in that way (my fields are > > mathematical analysis, integrable systems, and not computer science nor > > I meant "mathematical" as a school of thought (going from the abstract > to the concrete), not as a domain of research. I don't know any area > of mathematics either that uses the array concept, but it is > definitely common in computer science (as a structured collection of > similar data). Image data is a good example. Just my 2c worth: I count myself in the "mathematical" school despite being a physicist. I look at matrices as having a specific algebra which, for instance, cannot be easily made to apply to higher-dimensional arrays. Therefore they are not just arrays looked at in a different way. For object-oriented thinkers this means they are different objects. They may "inherit" a lot of attributes from arrays but are not arrays. Another point to note is that a specific complaint earlier in the thread was the computational inefficiency of using numpy arrays for matrix-intensive operations. It seems to me that it would be far easier to write an optimised set of code for matrices if they were known to be a separate class. An example (which is probably not useful, but serves for illustration) is that one could "cache" or delay transposes etc, knowing that a matrix-multiply was likely to be about to come up. This sort of thing would be more difficult if the result of the transpose would have to be sensible when followed by a generic array operation. David |
From: Pearu P. <pe...@ce...> - 2002-03-08 15:37:29
|
On Fri, 8 Mar 2002, Perry Greenfield wrote: > It's safer, but it isn't safe. Besides, one could still do this and raise > exceptions on mixed types. Is the issue that people strongly want to do > (if both a an b are arrays) > > a.M * b (or matrix(a) * b) > > instead > > a.M * b.M (or matrix(a) * matrix(b)) > > to get matrix behavior? Just to be clear, my suggestion consisted on the following points: 1) not to introduce any new type or concept such as matrix 2) to forget the current Matrix class 3) all objects are arrays, including a.M 4) a.M has a temporary bit set only for the following operation So, with this setup there is no issue with mixed types at all and it is easy to implement. But if there will be introduced a new type, matrix, then this setup does not work. The reason why I proposed the above setup was exactly because I didn't like that a.M would return a different object type in the middle of an expresssion. Pearu |
From: Perry G. <pe...@st...> - 2002-03-08 15:11:49
|
Pearu Peterson writes: [...] > > I find separating array and matrix instances (in a sense of raising > exception when doing <matrix> <op> <array>) not a very simple solution: > New concepts are introduced that actually do not solve the simplicity > problem of representing matrix operations. As I see it, they > only introduce restrictions and the main assumption behind the rationale > is that "users are dumb and they don't know what is best for them". > This is how I interpret the raised exception as behind the scenes matrix > and array are the same (in the sense of data representation). > I don't think the issue is whether users are "dumb" but rather, will it be more or less transparent to them what is supposed to happen. Remember, this particular proposal affects in no way the notational convenience when operands are of the same type. It doesn't even affect the notational convenience of most of the examples presented (e.g., a.M * b.M) as long as the resulting operands are of the same type. It only affects cases involving mixed types. Do we really want <matrix> * <array> (yes, it would be possible to have one dominate over <array> * <matrix> the other always regardless of order) to mean two different things for example? Will a user always be aware that a module function returns arrays rather than matrices. Yes, users ought to check the documentation, but they often don't or they misremember. The more I think about it the more I come to think it really is better to be safer in this case. It will not be hard for users to explicitly convert, nor should it be notationally cumbersome. E.g. (just to use one the proposed options) matrix(a) * b a * array(b) I don't see this as a big burden. I would rather do it this way myself for my own code. [...] > (there have been two implementation approaches proposed for this: (i) a.M > returns a Matrix instance, (ii) a.M returns the same array with a > temporarily set bit saying that the following operation is somehow > special). > To me, this looks like a safe solution. Though it is a hack, at least it > is simple and understandable in anywhere where it is used (having a * b > where b can be either matrix or array, it is not predictable from just > looking the code what the result will be -- not very pythonic indeed). > It's safer, but it isn't safe. Besides, one could still do this and raise exceptions on mixed types. Is the issue that people strongly want to do (if both a an b are arrays) a.M * b (or matrix(a) * b) instead a.M * b.M (or matrix(a) * matrix(b)) to get matrix behavior? Perry |
From: Konrad H. <hi...@cn...> - 2002-03-08 14:51:54
|
> regarding to consider arrays as a mathematical concept too. Just in > mathematics I never (need to) use arrays in that way (my fields are > mathematical analysis, integrable systems, and not computer science nor I meant "mathematical" as a school of thought (going from the abstract to the concrete), not as a domain of research. I don't know any area of mathematics either that uses the array concept, but it is definitely common in computer science (as a structured collection of similar data). Image data is a good example. > something can be accomplished efficiently. To be honest, I don't like > programming in Python because it has a nice style, but because I can > accomplish a lot with it in a very efficient way (and not only by using I want both :-) > > Moreover, I claim that Python *does* provide a good solution, it is > > merely a very different one. > > So, what is it? Separate matrix and array objects, with computationally efficient but explicit (verbose) interconversion. > Yes, it works: > >>> from Numeric import * > >>> class T: __rpow__ = lambda s,o: transpose(o) > ... > >>> print array([[1,2],[3,4]]) ** T() > [[1 3] > [2 4]] Right, it works as long as the left argument doesn't try to do the power operation itself. > And I don't understand why it is abusive (because it is a different > approach?). It's just an idea. For me, "power" is a shorthand for repeated multiplication, with certain properties attached to it. I have no problem with using the ** operator for something else, but then on different data types. The idea that a**b could be completely different operations for the same a as a function of b is not very appealing to me. In fact, the idea that an operand instead of the operator defines the operation is not very appealing to me. There's also a more pragmatic objection which is purely technical, I like to stay away from playing tricks with the binary operator type coercion system in Python. Sooner or later it always bites back. And the details have changed over Python releases, which is a compatibility nightmare. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hi...@cn... Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- |