RE: [Numpy-discussion] Re: Meta: too many numerical libraries doing the same thing?

[Paul F. D. <pa...@pf...>]

There is more to this issue than meets the eye, both technically and
historically.

For numerical algorithms to be available independent of language, they
would have to be packaged as components such as COM objects. While there
is research in this field, nobody knows whether it can be done is a way
that is efficient enough.

For a given language like C, C++, Eiffel or Fortran used as the
speed-demon base for wrapping up in Python, there are some difficult
technical issues. Reusable numerical software needs context to operate
and there is no decent way to supply the context in a
non-object-oriented language. Geoff Furnish wrote a good paper about the
issue for C++ showing the way to truly reusable libraries in that
language, and recent improvements in Eiffel make it easier to do there
now. In C or Fortran you simply can't do it. (Note that Eiffel or C++
versions of some NAG routines typically have methods with one or two
arguments while the C or Fortran ones have 15 or more; a routine is not
reusable if you have to understand that many arguments to try it. There
are also important issue with regard to error handling and memory).

The second issue is the algorithmic issue: most scientists do NOT know
the right algorithms to use, and the ones they do use are often
inferior. The good algorithms are for the most part in commercial
libraries, and the numerical analysis literature, where they were
written by numerical analysts. Often the coding from both sources is
unavailable for free use, in the wrong language, and/or wretched.

The commerical libraries also exist because some companies have
requirements for fiduciary responsibility; in effect, they need a
guarantor of the software to show that they have not carelessly depended
on software of unknown quality. 

In short, computer scientists are not going to be able to write such a
library without an army of numerical analysts familiar with the
literature, and the numerical analysts aren't going to write it unless
they are OO-experienced, which almost all of them aren't, so far.

Most people when they discuss mathematical software think of leaves on
the call tree. In fact the most useful mathematical software, in the
sense that it incorporates the most expertise, is middleware such as ODE
solvers, integrators, root finders, etc. The algorithm itself will have
many controls, optional outputs, etc. This requires a library-wide
design motif.

I thus feel there are perfectly good reasons not to expect such a
library soon. The Python community could do a good OO-design using what
is available (such as LAPACK) but we haven't -- all the contributions
are functional.