Re: [Numpy-discussion] Follow-up Numarray header PEP

[<ger...@gr...>]

Hi Todd,

Attached is a new version of numnum (including 'topbot', an alternative
implementation of numnum). The README contains some additional comments
with respect to numarray and Numeric (new comments are preceeded by '+',
old comments by '-').  There were still some other bugs in numnum, too.

On 23 Jul 2004 13:28:47 -0400, Todd Miller wrote
> I finally got to your numnum stuff today... awesome work!  You've got
> lots of good suggestions.  Here are some comments:
> 
> 1. Thanks for catching the early return problem with numarray's
> import_array().  It's not just bad, it's wrong.  It'll be fixed for 1.1.
> 
> 2. That said,  I think expanding the macros in-line in numnum is a
> mistake.  It seems to me that "import_array(); PyErr_Clear();" or
> something like it ought to be enough...  after numarray-1.1 anyway.
>
Indeed, but I am spoiled by C++ and was falling back on gcc -E for
debugging.
> 
> 3. I think there's a problem in numnum.toNP() because of numarray's
> array "behavior" issues.  A test needs to be done to ensure that the
> incoming array is not byteswapped or misaligned;  if it is, the easy 
> fix is to make a numarray copy of the array before copying it to Numeric.
>
Done, but what would be the best function to do this? And the documentation
could insist a little more on the possibility of ill-behaved arrays (see
README). 
>
> 4. Kudos for the LP64 stuff.  numconfig is a thorn in the side of the
> PEP,  so I'll put your techniques into numarray for 1.1. 
>  HAS_FLOAT128 is not currently used,  so it might be time to ditch 
> it.  Anyway, thanks!
>
There is a difference between the PEP header files and internal numarray
usage. I find in my CVS working copy:
[packer@slow numarray]$ grep HAS_FLOAT */*
Src/_ndarraymodule.c:#if HAS_FLOAT128
and
[packer@slow numarray]$ grep HAS_UINT64 */*
Src/buffer.ch:  #if HAS_UINT64
Src/buffer.ch:        #if HAS_UINT64
Src/buffer.ch:        #if HAS_UINT64
Src/buffer.ch:        #if HAS_UINT64
Src/buffer.ch:        #if HAS_UINT64
Src/libnumarraymodule.c:        #if HAS_UINT64
Src/libnumarraymodule.c:        #if HAS_UINT64
Src/libnumarraymodule.c:        #if HAS_UINT64
Src/libnumarraymodule.c:        #if HAS_UINT64
Src/libnumarraymodule.c:        #if HAS_UINT64

but that is not be true for the header files (more important for the PEP)
[packer@slow Include]$ grep HAS_UINT64 */*
[packer@slow Include]$ grep HAS_FLOAT128 */*
numarray/arraybase.h:#if HAS_FLOAT128

> 
> 5. PyArray_Present() and isArray() are superfluous *now*.  I was
> planning to add them to Numeric.
> 
> 6. The LGPL may be a problem for us and is probably an issue if we ever
> try to get numnum into the Python distribution.  It would be better 
> to release numnum under the modified BSD license,  same as numarray.
>
Done, with certain regrets because I believe in (L)GPL.
The minutes of the last board meeting of the PSF tipped the scale
( http://www.python.org/psf/records/board/minutes-2004-06-18.html )

What remains to be done is showing how to add numnum's functionality
to a 3rd party extension by linking numnum's object files to the
extension instead of importing numnum's C-API (numnum should not
become another dependency)

Gerard

> 
> 7. Your API struct was very clean.  Eventually I'll regenerate numarray
> like that.
> 
> 8. I logged your comments and bug reports on Source Forge and eventually
> they'll get fixed.
> 
> A to Z the numnum/pep code is beautiful.  Next stop, header PEP update.
> 
> Regards,
> Todd
>
> 
> On Sun, 2004-07-18 at 17:24, ger...@gr... wrote: 
> > Hi Todd,
> > 
> > This is a follow-up on the 'header pep' discussion.
> > 
> > The attachment numnum-0.1.tar.gz contains the sources for the
> > extension modules pep and numnum.  At least on my systems, both
> > modules behave as described in the 'numarray header PEP' when the
> > extension modules implementing the C-API are not present (a situation
> > not foreseen by the macros import_array() of Numeric and especially
> > numarray).  IMO, my solution is 'bona fide', but requires further
> > testing.
> > 
> > The pep module shows how to handle the colliding C-APIs of the Numeric
> > and numarray extension modules and how to implement automagical
> > conversion between Numeric and numarray arrays.
> > 
> > For a technical reason explained in the README, the hard work of doing
> > the conversion between Numeric and numarray arrays has been delegated
> > to the numnum module.  The numnum module is useful when one needs to
> > convert from one array type to the other to use an extension module
> > which only exists for the other type (eg. combining numarray's image
> > processing extensions with pygame's Numeric interface):
> > 
> > Python 2.3+ (#1, Jan  7 2004, 09:17:35)
> > [GCC 3.3.1 (SuSE Linux)] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> import numnum; import Numeric as np; import numarray as na
> > >>> np1 = np.array([[1, 2], [3, 4]]); na1 = numnum.toNA(np1)
> > >>> na2 = na.array([[1, 2, 3], [4, 5, 6]]); np2 = numnum.toNP(na2)
> > >>> print type(np1); np1; type(np2); np2
> > <type 'array'>
> > array([[1, 2],
> >        [3, 4]])
> > <type 'array'>
> > array([[1, 2, 3],
> >        [4, 5, 6]],'i')
> > >>> print type(na1); na1; type(na2); na2
> > <class 'numarray.numarraycore.NumArray'>
> > array([[1, 2],
> >        [3, 4]])
> > <class 'numarray.numarraycore.NumArray'>
> > array([[1, 2, 3],
> >        [4, 5, 6]])
> > >>>
> > 
> > The pep module shows how to implement array processing functions which
> > use the Numeric, numarray or Sequence C-API:
> > 
> > static PyObject *
> > wysiwyg(PyObject *dummy, PyObject *args)
> > {
> >     PyObject *seq1, *seq2;
> >     PyObject *result;
> > 
> >     if (!PyArg_ParseTuple(args, "OO", &seq1, &seq2))
> >         return NULL;
> > 
> >     switch(API) {
> >     case NumericAPI:
> >     {
> >         PyObject *np1 = NN_API->toNP(seq1);
> >         PyObject *np2 = NN_API->toNP(seq2);
> >         result = np_wysiwyg(np1, np2);
> >         Py_XDECREF(np1);
> >         Py_XDECREF(np2);
> >         break;
> >     }
> >     case NumarrayAPI:
> >     {
> >         PyObject *na1 = NN_API->toNA(seq1);
> >         PyObject *na2 = NN_API->toNA(seq2);
> >         result = na_wysiwyg(na1, na2);
> >         Py_XDECREF(na1);
> >         Py_XDECREF(na2);
> >         break;
> >     }
> >     case SequenceAPI:
> >         result = seq_wysiwyg(seq1, seq2);
> >         break;
> >     default:
> >         PyErr_SetString(PyExc_RuntimeError, "Should never happen");
> >         return 0;
> >     }
> > 
> >     return result;
> > }
> > 
> > See the README for an example session using the pep module showing that
> > it is possible pass a mix of Numeric and numarray arrays to pep.wysiwyg().
> > 
> > Notes:
> > 
> > - it is straightforward to adapt pep and numnum so that the conversion
> >   functions are linked into pep instead of imported.
> > 
> > - numnum is still 'proof of concept'.  I am thinking about methods to
> >   make those techniques safer if the numarray (and Numeric?) header
> >   files make it never into the Python headers (or make it safer to
> >   use those techniques with Python < 2.4).  In particular it would
> >   be helpful if the numerical C-APIs export an API version number,
> >   similar to the versioning scheme of shared libraries -- see the
> >   libtool->versioning info pages. 
> > 
> > I am considering three possibilities to release a more polished
> > version of numnum (3rd party extension writers may prefer to link
> > rather than import numnum's functionality):
> > 
> > 1. release it from PyQwt's project page
> > 2. register an independent numnum project at SourceForge
> > 3. hand numnum over to the Numerical Python project (frees me from
> >    worrying about API changes).
> > 
> > 
> > Regards -- Gerard Vermeulen
> 
> --


--
Open WebMail Project (http://openwebmail.org)