numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] Re: take for character arrays?

[Faheem M. <fa...@em...>]

On Thu, 15 Apr 2004, Todd Miller wrote:

> On Thu, 2004-04-15 at 16:17, Faheem Mitha wrote:
> > On Thu, 15 Apr 2004, Todd Miller wrote:
> >
> > > Try this for a workaround:
> > >
> > > s.swapaxes(0,1)
> > > s = s.copy()
> > > r = numarray.take(s, (1,2))
> > > r.swapaxes(0,1)
> >
> > I was trying
> >
> > In [50]: ts = numarray.transpose(s)
> >
> > In [51]: numarray.take(ts,(1,2),axis=0)
> > ...
> > ValueError: Invalid destination array: partial indices require
> > contiguous non-byteswapped destination
> >
> > In [52]: tscopy = numarray.transpose(s).copy()
> >
> > In [53]: numarray.take(tscopy,(1,2),axis=0)
> > Out[53]:
> > CharArray([['b', 'e', 'h'],
> >            ['c', 'f', 'i']])
> >
> > Is this the same or a related bug?
>
> Same bug.  Making a copy of the transposed array ensures that it (the
> copy) is contiguous.

I am wondering if this bug affects boost-python as well. I got the same
behaviour when using boost.python's version of take.

Can you point to where in the source code the bug is? Thanks.

                                                           Faheem.

Re: [Numpy-discussion] Re: take for character arrays?

[Todd M. <jm...@st...>]

On Thu, 2004-04-15 at 16:17, Faheem Mitha wrote:
> On Thu, 15 Apr 2004, Todd Miller wrote:
> 
> > Try this for a workaround:
> >
> > s.swapaxes(0,1)
> > s = s.copy()
> > r = numarray.take(s, (1,2))
> > r.swapaxes(0,1)
> 
> I was trying
> 
> In [50]: ts = numarray.transpose(s)
> 
> In [51]: numarray.take(ts,(1,2),axis=0)
> ...
> ValueError: Invalid destination array: partial indices require
> contiguous non-byteswapped destination
> 
> In [52]: tscopy = numarray.transpose(s).copy()
> 
> In [53]: numarray.take(tscopy,(1,2),axis=0)
> Out[53]:
> CharArray([['b', 'e', 'h'],
>            ['c', 'f', 'i']])
> 
> Is this the same or a related bug? 

Same bug.  Making a copy of the transposed array ensures that it (the
copy) is contiguous.

Regards,
Todd

-- 
Todd Miller <jm...@st...>

Re: [Numpy-discussion] Re: take for character arrays?

[Faheem M. <fa...@em...>]

On Thu, 15 Apr 2004, Todd Miller wrote:

> Try this for a workaround:
>
> s.swapaxes(0,1)
> s = s.copy()
> r = numarray.take(s, (1,2))
> r.swapaxes(0,1)

I was trying

In [50]: ts = numarray.transpose(s)

In [51]: numarray.take(ts,(1,2),axis=0)
...
ValueError: Invalid destination array: partial indices require
contiguous non-byteswapped destination

In [52]: tscopy = numarray.transpose(s).copy()

In [53]: numarray.take(tscopy,(1,2),axis=0)
Out[53]:
CharArray([['b', 'e', 'h'],
           ['c', 'f', 'i']])

Is this the same or a related bug? Thanks for the reply and info.

                                                        Faheem.

Re: [Numpy-discussion] Re: take for character arrays?

[Todd M. <jm...@st...>]

On Thu, 2004-04-15 at 13:06, Faheem Mitha wrote:
> On Thu, 15 Apr 2004 05:13:29 +0000 (UTC), Faheem Mitha
> <fa...@em...> wrote:
> 
> > Hi,
> >
> > I'm trying to extract subarrays of a character array using take(), but
> > getting mysterious errors, for example:
> >
> > ValueError: Invalid destination array: partial indices require
> > contiguous non-byteswapped destination
> >
> > I am thinking that perhaps this operation is not supported for
> > character arrays, and I should convert my character array to a numeric
> > array instead (the strings are all of length 1).

This is a bug.  I logged it on source forge.

> 
> I read this again and realised this does not give any specific
> information, and is not very coherent. Got to avoid those late night
> postings. So here are the details. If I set axis = 0 there is no
> problem. I also don't have this problem with numerical arrays.
> 
> I am sure there is a simple reason for this, and hopefully a simple
> workaround.

Try this for a workaround:

s.swapaxes(0,1)
s = s.copy()
r = numarray.take(s, (1,2))
r.swapaxes(0,1)

Regards,
Todd

-- 
Todd Miller <jm...@st...>

[Numpy-discussion] Re: take for character arrays?

[Faheem M. <fa...@em...>]

On Thu, 15 Apr 2004 05:13:29 +0000 (UTC), Faheem Mitha
<fa...@em...> wrote:

> Hi,
>
> I'm trying to extract subarrays of a character array using take(), but
> getting mysterious errors, for example:
>
> ValueError: Invalid destination array: partial indices require
> contiguous non-byteswapped destination
>
> I am thinking that perhaps this operation is not supported for
> character arrays, and I should convert my character array to a numeric
> array instead (the strings are all of length 1).

I read this again and realised this does not give any specific
information, and is not very coherent. Got to avoid those late night
postings. So here are the details. If I set axis = 0 there is no
problem. I also don't have this problem with numerical arrays.

I am sure there is a simple reason for this, and hopefully a simple
workaround.

Thanks in advance.

                                                          Faheem.

**************************************************************************
In [22]: import numarray.strings as str

In [23]: import numarray

In [24]: s = str.array(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h',
'i'],shape=(3,3))

In [25]: numarray.take(s,(1,2),axis=1)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call
last)

/home/faheem/wc/corrmodel/boost/<console>

/usr/lib/python2.3/site-packages/numarray/ufunc.py in take(array,
indices, axis, outarr, clipmode)
   1779             raise ValueError("indices must be a sequence")
   1780         work = _gen.swapaxes(array, 0, axis)
-> 1781         work = work._take((indices,), outarr=outarr,
clipmode=clipmode)
   1782         return _gen.swapaxes(work, 0, axis)
   1783     else:

/usr/lib/python2.3/site-packages/numarray/generic.py in _take(self,
indices, **keywds)
    795     def _take(self, indices, **keywds):
    796         indices = list(indices)
--> 797         impliedShape, N = _takeShape(self, indices)
    798         result = self._clone(shape=impliedShape)
    799         indices = self._fix_pt_indices(indices)

/usr/lib/python2.3/site-packages/numarray/generic.py in
_takeShape(scattered, indexArrays)
    133         nShape = scattered._shape[-leftOver:]
    134         if scattered.isbyteswapped() or not
scattered.iscontiguous():
--> 135             raise ValueError("Invalid destination array:
partial indices require contiguous non-byteswapped destination")
    136     else:
    137         N = scattered._itemsize

ValueError: Invalid destination array: partial indices require
contiguous non-byteswapped destination

[Numpy-discussion] take for character arrays?

[Faheem M. <fa...@em...>]

Hi,

I'm trying to extract subarrays of a character array using take(), but
getting mysterious errors, for example:

ValueError: Invalid destination array: partial indices require
contiguous non-byteswapped destination

I am thinking that perhaps this operation is not supported for
character arrays, and I should convert my character array to a numeric
array instead (the strings are all of length 1).

Advice, suggestions, enlightenment appreciated.

                                                          Faheem.

[Numpy-discussion] sa.ve on software?? this is for u orchard pterygium

[Audrey S. <jii...@we...>]

commando arthralgic precipitin endenizen 
vicksburg seawater pretence 
choler sickbay brassiere xerobates 

[Numpy-discussion] Europython: Registration open - Talk submission deadline is Apr 15th

[Nicolas C. <Nic...@lo...>]

Dear Scientific Pythonistas,

I'm forwarding this update about the EuroPython conference. Please
note that the talk submission deadline is April 15th but will probably
be extended a bit. I suppose most of you will be interested in the
Science Track at least.

For previous years' program, please refer to :

	http://www.europython.org/2002/sessions/talks
	http://www.europython.org/2003/sessions/talks

---------------------------------------------------------------------
Subject: [EuroPython] Europython Update: Registration open

Europython Update
=================

- Registration is now open. We apologise for the delay, but we have
had some technical problems.

- Due to this, we have decided to keep the submission of abstracts
for the refereed track open for one more day. Last submission time is
now on Sunday 11 April at 23.59 CET.

- We have a limited number of beds available in very affordable
accomodation near the conference venue. Book early before it runs out.

- We are still receiving submissions for regular talks and tutorials.
Closing date is 15 April.

- There is now a wiki at the Europython website for sprint organising. 
Start planning!

About the conference
====================

EuroPython 2004 will be held 7-9 June in Göteborg, Sweden.

The EuroPython conference will have tracks for Science, Business,
Education, Applications, Frameworks, Zope and the Python language
itself. Lightning talks, Open Space and BOF sessions are also planned.
There will be tutorials as well, both for newcomers to Python and
Python users interested in special subjects. In the days before and
after the conference, programming sprints will be arranged.


Important dates
===============

Refereed paper proposals: until 11 April.
Submission of talks: 1 March - 15 April.
Early Bird registration: 9 April - 1 May.
Accomodation booking: 9 April - 1 May (or until space runs out)

More information at http://www.europython.org.

---------------------------------------------------------------------

Hope to see you there.


-- 
Nicolas Chauvat

logilab.fr - services en informatique avancée et gestion de connaissances  

Re: [Numpy-discussion] Questions about C API

[Russell E O. <ow...@as...>]

At 2:32 PM -0400 2004-04-09, Todd Miller wrote:
>On Fri, 2004-04-09 at 13:25, Russell E Owen wrote:
>>  I'm trying to write a median function (surely one is there already,
>>  but if so I've not found it). The fastest technique seems to involve
>>  rearranging the data in place.
>
>You might want to try out numarray.image.median (or
>numarray.image.combine.median for versions < 0.9).

Thanks. Based on the source, it looks like that prefers a stack of 
identical images (instead of a single array, though perhaps one could 
make that case work).

John Hunter also kindly pointed out MLab (which it turns out is 
buried under linear_algebra in numarray: 
numarray.linear_algebra.mlab).

I had been working on a fast median instead of one that sorted the 
whole array, but after timing mlab's median I can probably get away 
with the simple implementation (and skip the C extension).

>>  Also, is there any simple sample code running around for handling
>>  arrays of any type (e.g. with C++ templates)?
>
>I think Peter Verveer did some code like this in the nd_image package.
>The general idea is to constrain the arrays with type tAny in the high
>level API and then switch off of a->descr->type_num to select type
>specific code.

Thank you very much for the explanation of the high level interface 
and the pointer to this sample code.

-- Russell

Re: [Numpy-discussion] Questions about C API

[Todd M. <jm...@st...>]

On Fri, 2004-04-09 at 13:25, Russell E Owen wrote:
> I'm trying to write a median function (surely one is there already, 
> but if so I've not found it). The fastest technique seems to involve 
> rearranging the data in place. 

You might want to try out numarray.image.median (or
numarray.image.combine.median for versions < 0.9).

> So I'd like to offer two calls:
> - A version that rearranges the input array if no local copy needed 
> to be made (e.g. the input data was already contiguous and of the 
> expected type). I know how to write that one.
> - A safer, slower version that always makes a copy of the input data. 
> I'm stuck on that. The high-level interface has flags for when to 
> make a local copy, but none of them seems to mean "always make a 
> local copy". Surely I am missing something there? I guess I could get 
> the data, see if a local copy was made and if not, make a copy of my 
> own, but that seems messy.

The high level API is set up to do the least amount of work to guarantee
a set of constraints.  You can independently constrain numarray
properties (contiguousness, byteswap, alignment) and the array type to
be what you know your code can process.  The high level API will create
or convert arrays as necessary, and then release them when the function
is done processing (provided you manage your ref counts correctly as
shown in the manual).  So, the gist of the high level API is that it
takes some mysterious object (a poorly behaved array, an array of the
wrong type, nested lists, etc.) and converts it into a nicely formed
proxy array on which you do your computing.  In the case where the
mysterious object actually was a well behaved properly typed array,
almost no work has to be done and the proxy is the original array.

If you need a copy because your code is going to destroy the input
array,  use NA_copy() or just do a .copy() method call yourself.  Making
this kind of copy could indeed be folded into the existing API and it
would result in simpler application code, but it isn't there now.

> 
> Also, is there any simple sample code running around for handling 
> arrays of any type (e.g. with C++ templates)? 

I think Peter Verveer did some code like this in the nd_image package. 
The general idea is to constrain the arrays with type tAny in the high
level API and then switch off of a->descr->type_num to select type
specific code.

Regards,
Todd

> -- 
> Todd Miller <jm...@st...>

Re: [Numpy-discussion] Questions about C API

[John H. <jdh...@ac...>]

>>>>> "Russell" == Russell E Owen <ow...@as...> writes:

    Russell> I'm trying to write a median function (surely one is
    Russell> there already, but if so I've not found it). 

from MLab import median

JDH

[Numpy-discussion] Questions about C API

[Russell E O. <ow...@as...>]

I'm trying to write a median function (surely one is there already, 
but if so I've not found it). The fastest technique seems to involve 
rearranging the data in place. So I'd like to offer two calls:
- A version that rearranges the input array if no local copy needed 
to be made (e.g. the input data was already contiguous and of the 
expected type). I know how to write that one.
- A safer, slower version that always makes a copy of the input data. 
I'm stuck on that. The high-level interface has flags for when to 
make a local copy, but none of them seems to mean "always make a 
local copy". Surely I am missing something there? I guess I could get 
the data, see if a local copy was made and if not, make a copy of my 
own, but that seems messy.

Also, is there any simple sample code running around for handling 
arrays of any type (e.g. with C++ templates)? Right now I'm writing 
to a specific type (since it's easy and I know what type to expect 
for my application), but I'd rather make my code more general and 
numarray-like. I figured the numarray code itself would be a good 
example, but, well, I didn't understand the parts I looked at (maybe 
I'm clueless, maybe I didn't find the right bits to examine).

-- Russell

Re: [Numpy-discussion] problems with compiling and loading C++ extension

[Faheem M. <fa...@em...>]

On Wed, 7 Apr 2004, Fernando Perez wrote:

> I hope this helps you.  Mail me if you want the full files (this is old code I
> don't use anymore, so it may not quite compile now anymore, but you can still
> look a it).

Yes, please do send me your files (preferably as a tar.gz). Thanks for the
info. I expect you are right. I'll take a more careful look. One problem
is that I don't understand the logic behind C linkage very well. For
example, does it suffice to wrap the declaration of the function or do I
need to wrap the definition too?

Can you suggest a good reference, preferably online? My C++ books are not
much help with this.

Thanks.

                                                                Faheem.

Re: [Numpy-discussion] problems with compiling and loading C++ extension

[Fernando P. <fp...@co...>]

Faheem Mitha wrote:
> Dear People,
> 
> I have been having an odd problem with compiling and loading a simple
> C++ extension to python (as a .so file in Linux). Unfortunately, I'll
> need to include my files in here so I apologize in advance for the
> length of this message.

Are you sure you have all the needed 'extern C' declarations in place? 
Remember, you'll be compiling this thing with a C++ compiler, but it will be 
called by Python's C runtime.  Hence, _all_ python-visible functions will need 
to be wrapped in extern C calls, including your module initializer (you often 
don't think of this one, because it's implicitly

Here's a typical snippet for a python-visible function in one of my C++ 
blitz-based extensions:

// mts_contract
extern "C" {
     static PyObject* mts_contract(PyObject *self, PyObject *args)
     {

         PyArrayObject *M_array, *T_array;
...

and here's what the module initializer looks like:


/* Module initializer -- called on first import.
    It *must* be called initMODULENAME */
extern "C" void initutils_()
{
     import_array();

     // Initialize the dimension dispatch tables
     init_mts_contract_funcs();
     init_mtt_contract2_funcs();
     init_mseqtt_contract2_funcs();

     // Create the module and add the functions
     PyObject* mod = Py_InitModule4("utils_",
                                    compiled_methods,
                                    module_doc,
                                    (PyObject*)NULL,
                                    PYTHON_API_VERSION);

     // Add some symbolic constants to the module
     PyObject* mod_dict = PyModule_GetDict(mod);

     // Make a module-level exception
     // The first argument is what the exception class will appear to be
     // in python. The return value is a class object.
     utils_Error = PyErr_NewException("utils_.utils_Error",
                                      NULL, NULL);

     // Add it to the module's dictionary. Second argument is the key
     PyDict_SetItemString(mod_dict, "utils_Error", utils_Error);

     // Add other constants here
     PyDict_SetItemString(mod_dict, "__all__",
                          Py_BuildValue("sss",
                                        "mts_contract2",
                                        "mtt_contract2",
                                        "mseqtt_contract2"
                                        ));

     // Check for errors
     if (PyErr_Occurred())
         Py_FatalError("can't initialize module utils_");
}


I hope this helps you.  Mail me if you want the full files (this is old code I 
don't use anymore, so it may not quite compile now anymore, but you can still 
look a it).

Cheers,

f

[Numpy-discussion] problems with compiling and loading C++ extension

[Faheem M. <fa...@em...>]

Dear People,

I have been having an odd problem with compiling and loading a simple
C++ extension to python (as a .so file in Linux). Unfortunately, I'll
need to include my files in here so I apologize in advance for the
length of this message.

You can download also the files below from
http://www.stat.unc.edu/students/faheem/python/python.tar.gz in case
that is more convenient.

This needs blitz++ and python's distutils and numarray to compile.

When I try to load arrmod.so into python I get

In [1]: import arrmod
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call
last)

/tmp/wc/corrmodel/ex/<console>

ImportError: ./arrmod.so: undefined symbol:
_Z11py_to_blitzIdLi2EEN5blitz5ArrayIT_XT0_EEEP13PyArrayObject

If I move the py_to_blitz routine from conv.cc back into arrmod.cc,
the error disappears, arrmod loads in python, and everything works
correctly. I'm not sure what is going on here. Can someone enlighten
me? At first I thought it might have to do with the need for C
linkage, but I tried that, and it does not appear to be the case.

I also got the following compilation warning, which I don't
understand. I don't know if that is relevant.

---------------------------------------------------------------------
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall
-Wstrict-prototypes -fPIC -I/usr/include/python2.3 -c conv.cc -o
/tmp/temp.linux-i686-2.3/conv.o -Wall
/usr/include/python2.3/Numeric/arrayobject.h:286: warning:
`void**PyArray_API'
   defined but not used
---------------------------------------------------------------------

Thanks in advance for any enlightenment.

                                                 Faheem.

**********************************************************************
Makefile
**********************************************************************
arrmod.so: arrmod.cc common.hh conv.cc
	python setup.py build --build-base=/tmp --build-lib=.
# may not do the right thing for everyone
clean:
	rm arrmod.so
	rm -r /tmp/temp.linux-i686-2.3
**********************************************************************

**********************************************************************
setup.py
**********************************************************************
from distutils.core import setup, Extension

module4 = Extension('arrmod',
                    sources = ['arrmod.cc','conv.cc'],libraries=["blitz","m"],
                    extra_compile_args = ['-Wall'])

setup (name = 'arrmod',
       version = '1.0',
       description = 'This module performs different operations on arrays',
       ext_modules = [module4]
       )
**********************************************************************

**********************************************************************
common.hh
**********************************************************************
#include "Python.h"
#include "Numeric/arrayobject.h"
#include <blitz/array.h>

using namespace std;
using namespace blitz;

template<class T, int N>
static Array<T,N> py_to_blitz(PyArrayObject* arr_obj);
**********************************************************************

**********************************************************************
conv.cc
**********************************************************************
#include "common.hh"

// Convert a Numpy array to a blitz one, using the original's data (no
// copy)
template<class T, int N>
static Array<T,N> py_to_blitz(PyArrayObject* arr_obj)
{
  int T_size = sizeof(T);
  TinyVector<int,N> shape(0);
  TinyVector<int,N> strides(0);
  int *arr_dimensions = arr_obj->dimensions;
  int *arr_strides = arr_obj->strides;

  for (int i=0;i<N;++i)
    {
      shape[i]   = arr_dimensions[i];
      strides[i] = arr_strides[i]/T_size;
    }
  return Array<T,N>((T*) arr_obj->data,shape,strides,neverDeleteData);
}
************************************************************************

************************************************************************
arrmod.cc
************************************************************************
#include "common.hh"

template<class T, int N>
static Array<T,N> py_to_blitz(PyArrayObject* arr_obj);

static PyObject * arrmod_elem(PyObject *self, PyObject *args);

static PyMethodDef arrmod_methods[] =
  {
    {"elem", (PyCFunction)arrmod_elem, METH_VARARGS, "Returns the trace of a two-dimensional array.\n"},
    {NULL, NULL, 0, NULL}
  };

PyMODINIT_FUNC initarrmod(void)
{
  (void) Py_InitModule3("arrmod", arrmod_methods, "Returns the Trace of a two-dimensional array.\n");
  import_array();
}

static PyObject * arrmod_elem(PyObject *self, PyObject *args)
{
  PyObject *input, *result;
  PyArrayObject *array;
  double el;
  int i, j;

  if (!PyArg_ParseTuple(args, "Oii", &input, &i, &j))
    return NULL;
  array = (PyArrayObject *)
    PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2);
  if (array == NULL)
    return NULL;

  Array<double,2> arr = py_to_blitz<double,2>(array);

  el = arr(i,j);
  result = Py_BuildValue("d",el);
  return result;
}
********************************************************************************

Re: [Numpy-discussion] question about nonzero behavior in numarray

[Joachim S. <li...@js...>]

* I claimed:
>     if a==1:
>
> might be understood as "if a is everywhere one" or "if the length
> of vector a is one", etc. The current meaning is "if a is anywhere
> one", [...]

This is of course wrong and would probably not be misunderstood as
that. What I meant to say is that

    >>> if a:
    ...

would be evaluated as "true" if a is anywhere non-zero. It was too
late in the night it seems...

Sorry for the confusion.
Joachim

Re: [Numpy-discussion] question about nonzero behavior in numarray

[Joachim S. <li...@js...>]

* Fernando Perez [2004-04-02 02:15]:
> Joachim Saul wrote:
>
> >How often does it happen that you have an empty array, which you
> >need to test for? Certainly not as frequently as in case of a
> >list.
>
> Well, just these days I'm working with code where I'm precisely using that
> kind of check, relying on Numeric's behavior.  But I know that with
> potentially nested structures, it's probably best to be explicit and do all
> checks via things like:
>
> if len(a)
>
> and
>
> if a==0

But what would a==0 return? A boolean? An array of booleans? An
array if integers? The way it is evaluated now in Numeric is
pretty clear, though there could be other evaluations.  For
instance,

    if a==1:

might be understood as "if a is everywhere one" or "if the length
of vector a is one", etc. The current meaning is "if a is anywhere
one", which may not be intuitive under all circumstances, but as
long as it is the well defined behaviour, I simply don't see the
need to change or forbid that.

Nested structures are an important argument against keeping
compatibility with Python list behaviour. For instance

    >>> if []:   print "True"
    >>> if [[]]: print "True"
    True

Which perfectly makes sense in Python, but an equivalent behaviour
of a numeric array would be complete humbug. Thus, I don't think
that it would make any sense to emulate Python list behaviour,
rather I would support keeping the *existing* Numeric behaviour.

Cheers,
Joachim

Re: FW: [Numpy-discussion] Questions about records

[Jin-chung H. <hs...@st...>]

> >--> arr1 = num.arange(4, shape=(1,2,2), type=num.Float64)
> >--> arr2 = num.arange(4, shape=(1,2,2), type=num.Float64)+10
> >--> a = rec.array([arr1, arr2], names="a,b")
> 
> But is there any advantage to that compared to just using named 
> arrays of the desired shape:
> a = num.arange(4, shape=(2,2), type=num.Float64)
> b = num.arange(4, shape=(2,2), type=num.Float64)+10
>

Not really, in this particular example.
 
> >I'd be interested in your application as to why a 2x2 table is necessary.
> 
> Here are two different uses I've come up with (both related to image 
> processing). 
[snip]

You need it because you need to pass it to the C-structure, I think.  In any 
case, you have found a way to get around the problem by using setshape.  I'll 
take a look of the module to get an idea of how much effort is needed to make 
the 2-D (and higher) record arrays to work more smoothly.

JC Hsu

Re: FW: [Numpy-discussion] Questions about records

[Russell E O. <ow...@as...>]

At 2:12 PM -0500 2004-04-02, Jin-chung Hsu wrote:
>First of all, the records module was developed mainly having the 1-D table in
>mind.  Even though it can have higher than one dimension, it is not thoroughly
>tested, as you have found out.  However, I'd argue that in many cases that the
>need to use a 2-D (or high) table can be substituted by having an 
>array in each
>cell(element).  In your example, instead of creating a 2x2 table 
>with each cell
>just having one number, you may be able to use a table with just one row and
>each cell is a 2x2 array.  You can create such a record like this:
>
>--> arr1 = num.arange(4, shape=(1,2,2), type=num.Float64)
>--> arr2 = num.arange(4, shape=(1,2,2), type=num.Float64)+10
>--> a = rec.array([arr1, arr2], names="a,b")

But is there any advantage to that compared to just using named 
arrays of the desired shape:
a = num.arange(4, shape=(2,2), type=num.Float64)
b = num.arange(4, shape=(2,2), type=num.Float64)+10

>I'd be interested in your application as to why a 2x2 table is necessary.

Here are two different uses I've come up with (both related to image 
processing). Both are beautifully served by a 2-d records array:

1) Find the centroid of a star. The algorithm I'm using (invented by 
Jim Gunn, I believe) is to walk across the image, looking for the 
point of maximum symmetry. At each point total pixels and a measure 
of asymmetry are measured in a 3x3 grid centered at that point. The 
minimum asymmetry in that 3x3 array is then used to determine where 
to walk next. (At the end a parabolic fit is done to the 3x3 
asymmetry data to find the true centroid; up until then it's only 
know to the nearest pixel).

In any case...right now I maintain two separate 3x3 arrays (total 
pixels and asymmetry). Whenever I take a step I shift the both arrays 
and then compute new data for the points which are missing data.

It would be cleaner and nicer to maintain one 3x3 records array with 
fields "totPix" and "asymm". Then the related data sticks together 
and I only have to shift the data once. (I meant to code it that way 
from the start, but my early attempts to use numeric.records were a 
disaster. I have a somewhat better handle on it now and may update my 
code).

2) Find all stars on an image. The algorithm I'm using (invented by 
Jeff Morgan, I believe) is to break an image up into blocks of, say, 
5x5 pixels. I then compute information about each "super pixel", such 
as center of mass, total counts, etc. My C++ code has 12 items of 
information for each super pixel (including 7 boolean flags) and is 
written to use a 2-dimensional array each element of which is a data 
structure with the appropriate fields. The obvious python equivalent 
is a numarray.records array. It sure sounds better than trying to 
keep track of 12 separate arrays!

-- Russell

Re: FW: [Numpy-discussion] Questions about records

[Jin-chung H. <hs...@st...>]

> I've been trying to generate a 2-dimensional numarray.records array 
> and am rather puzzled about some failures:
> 
> I tried a pair of 2x2 arrays for the buffer (one per field). I 
> thought the record would get its shape from those, but instead of a 
> 2x2 record I get a 2-length record, each of whose elements is a pair 
> of 2-length arrays.
> 
> >>>  arr1 = num.arange(4, shape=(2,2), type=num.Float64)
> >>>  arr2 = num.arange(4, shape=(2,2), type=num.Float64) + 10
> >>>  a = rec.array([arr1, arr2], names="a,b")
> >>>  a
> array(
> [(array([ 0.,  1.]), array([ 10.,  11.])),
> (array([ 2.,  3.]), array([ 12.,  13.]))],
> formats=['(2,)Float64', '(2,)Float64'],
> shape=2,
> names=['a', 'b'])
> 
> So I tried passing in 4-length arrays while specifying the shape 
> explicitly, but it failed. Is this a bug?:
> 
> >>>  arr1 = num.arange(4, type=num.Float64)
> >>>  arr2 = num.arange(4, type=num.Float64) + 10
> >>>  a = rec.array([arr1, arr2], shape=(2,2), names="a, b")
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
>    File "/usr/local/lib/python2.3/site-packages/numarray/records.py", 
> line 384, in array
>      byteorder=byteorder, aligned=aligned)
>    File "/usr/local/lib/python2.3/site-packages/numarray/records.py", 
> line 177, in fromarrays
>      raise ValueError, "array has different lengths"
> ValueError: array has different lengths
> 
> Generating a 4-length record and reshaping it does seem to work, 
> though there seems to be a bug in __str__ which I'll report:
> 
> >>>  arr1 = num.arange(4, type=num.Float64)
> >>>  arr2 = num.arange(4, type=num.Float64) + 10
> >>>  a = rec.array([arr1, arr2], names="a, b")
> >>>  a.setshape((2,2))
> >>>  a
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
>    File "/usr/local/lib/python2.3/site-packages/numarray/records.py", 
> line 718, in __repr__
>      outlist.append(Record.__str__(i))
> TypeError: unbound method __str__() must be called with Record 
> instance as first argument (got RecArray instance instead)
> >>>  a[1,1]
> <numarray.records.Record instance at 0x15d2ad0>
> >>>  str(a[1,1])
> '(1.0, 11.0)'
> 
> I see the str problem again if I don't specify any buffer:
> 
> >>>  import numarray as num
> >>>  import numarray.records as rec
> >>>  a = rec.array(formats="Float64,Float64", names="a,b", shape=(2,2))
> >>>  a
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
>    File "/usr/local/lib/python2.3/site-packages/numarray/records.py", 
> line 718, in __repr__
>      outlist.append(Record.__str__(i))
> TypeError: unbound method __str__() must be called with Record 
> instance as first argument (got RecArray instance instead)
> 
> So...comments? Should I report the shape issues as a bug?
> 
> -- Russell

First of all, the records module was developed mainly having the 1-D table in 
mind.  Even though it can have higher than one dimension, it is not thoroughly 
tested, as you have found out.  However, I'd argue that in many cases that the 
need to use a 2-D (or high) table can be substituted by having an array in each 
cell(element).  In your example, instead of creating a 2x2 table with each cell 
just having one number, you may be able to use a table with just one row and 
each cell is a 2x2 array.  You can create such a record like this:

--> arr1 = num.arange(4, shape=(1,2,2), type=num.Float64)
--> arr2 = num.arange(4, shape=(1,2,2), type=num.Float64)+10
--> a = rec.array([arr1, arr2], names="a,b")

I'd be interested in your application as to why a 2x2 table is necessary.

The __str__ method in RecArray is rather primitive but usually works for 1-D 
tables.  Eventually, we'll need a C-implementation of this method for both speed 
and flexibility.

When you use a list for the buffer in the array function, it is using a 
relatively intuitive (simplistic?) approach.  It tries to figure out the (one 
dimensional) length of each item in the list and use that as the record shape.

JC Hsu

[Numpy-discussion] Questions about records

[Russell E O. <ow...@as...>]

I've been trying to generate a 2-dimensional numarray.records array 
and am rather puzzled about some failures:

I tried a pair of 2x2 arrays for the buffer (one per field). I 
thought the record would get its shape from those, but instead of a 
2x2 record I get a 2-length record, each of whose elements is a pair 
of 2-length arrays.

>>>  arr1 = num.arange(4, shape=(2,2), type=num.Float64)
>>>  arr2 = num.arange(4, shape=(2,2), type=num.Float64) + 10
>>>  a = rec.array([arr1, arr2], names="a,b")
>>>  a
array(
[(array([ 0.,  1.]), array([ 10.,  11.])),
(array([ 2.,  3.]), array([ 12.,  13.]))],
formats=['(2,)Float64', '(2,)Float64'],
shape=2,
names=['a', 'b'])

So I tried passing in 4-length arrays while specifying the shape 
explicitly, but it failed. Is this a bug?:

>>>  arr1 = num.arange(4, type=num.Float64)
>>>  arr2 = num.arange(4, type=num.Float64) + 10
>>>  a = rec.array([arr1, arr2], shape=(2,2), names="a, b")
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "/usr/local/lib/python2.3/site-packages/numarray/records.py", 
line 384, in array
     byteorder=byteorder, aligned=aligned)
   File "/usr/local/lib/python2.3/site-packages/numarray/records.py", 
line 177, in fromarrays
     raise ValueError, "array has different lengths"
ValueError: array has different lengths

Generating a 4-length record and reshaping it does seem to work, 
though there seems to be a bug in __str__ which I'll report:

>>>  arr1 = num.arange(4, type=num.Float64)
>>>  arr2 = num.arange(4, type=num.Float64) + 10
>>>  a = rec.array([arr1, arr2], names="a, b")
>>>  a.setshape((2,2))
>>>  a
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "/usr/local/lib/python2.3/site-packages/numarray/records.py", 
line 718, in __repr__
     outlist.append(Record.__str__(i))
TypeError: unbound method __str__() must be called with Record 
instance as first argument (got RecArray instance instead)
>>>  a[1,1]
<numarray.records.Record instance at 0x15d2ad0>
>>>  str(a[1,1])
'(1.0, 11.0)'

I see the str problem again if I don't specify any buffer:

>>>  import numarray as num
>>>  import numarray.records as rec
>>>  a = rec.array(formats="Float64,Float64", names="a,b", shape=(2,2))
>>>  a
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
   File "/usr/local/lib/python2.3/site-packages/numarray/records.py", 
line 718, in __repr__
     outlist.append(Record.__str__(i))
TypeError: unbound method __str__() must be called with Record 
instance as first argument (got RecArray instance instead)

So...comments? Should I report the shape issues as a bug?

-- Russell

Re: [Numpy-discussion] question about nonzero behavior in numarray

[Fernando P. <Fer...@co...>]

Joachim Saul wrote:

> How often does it happen that you have an empty array, which you
> need to test for? Certainly not as frequently as in case of a
> list.

Well, just these days I'm working with code where I'm precisely using that 
kind of check, relying on Numeric's behavior.  But I know that with 
potentially nested structures, it's probably best to be explicit and do all 
checks via things like:

if len(a)

and

if a==0

etc.  So I could also be +1 on Perry's wish for turning it into an extension, 
at the expense of my current code breaking.

Cheers,

f.

Re: [Numpy-discussion] question about nonzero behavior in numarray

[Joachim S. <li...@js...>]

* Fernando Perez [2004-04-02 00:51]:
> Perry Greenfield wrote:
>
> >Personally, I prefer 1) since I think there is too much confusion
> >between whether false means an empty (len=0) array (like lists),
> >or an array that contains all zeros, and that it is worth breaking
> >backward compatibility on this (functions should be used to
> >distinguish which meaning is desired).
>
> I tend to prefer keeping compatibility with Python list behavior.  I think
> that in general, numarray arrays should behave different from lists only
> when truly justified (for example, at least in numeric all ufuncs can be
> applied to a list of floats as much as to an array, which is nice).

How often does it happen that you have an empty array, which you
need to test for? Certainly not as frequently as in case of a
list.

I think that if

  >>> a=ones(3)
  >>> a==1
  array([1, 1, 1])
  >>> a==0
  array([0, 0, 0])

then I would follow that

  >>> if a==0:
  ...
  >>> if array([0, 0, 0]):
  ...

should also _both_ evaluate as "false".

> Since anyone who knows python already knows that lists are true based on
> whether they contain _anything_:
>
> In [9]: a=[]
>
> In [10]: b=[0]
>
> In [11]: if a:
>    ....:     print 1
>    ....:
>
> In [12]: if b:
>    ....:     print 1
>    ....:
> 1
>
>
> then I think it's a fair, clean extension to make NumArray behave similarly.
>
> I think a good guiding principle is to keep the number of special cases for
> arrays at a strict minimum (there are obviously cases where it does make
> sense to have special cases).  Besides, this is an O(1) check, so it
> doesn't have the dangers of checking for all elements being zero.   Such a
> potentially expensive operation should _definitely_ happen only if
> explicitly requested, IMHO.

But if the behaviour is unambiguously defined, I see no problem
here, because I can as well check the length of a list using
len().

BTW, How about

    >>> a = [[]]
    >>> if a:
    ...

?

Cheers,
Joachim

Re: [Numpy-discussion] question about nonzero behavior in numarray

[Colin J. W. <cj...@sy...>]

Perry Greenfield wrote:

>Todd pointed out to me that he's gotten some complaints about
>the difference in how numarray nonzero behavior works.
>In version 0.9 the use of an array for a truth value was
>changed from a deprecation warning to an exception.
>
>We deprecated it because the meaning of an array as a truth
>value was ambiguous (or potentially confusing at best)
>and we felt it was better to force users to be explicit
>about what they wanted. Nevertheless, Numeric does allow that
>and removing the capability to use it in that context breaks
>some code. 
>
>We can give some thought to reversing that decision, but since
>we only hear from those that don't like the new behavior, we'd
>like to get some sense if there are many that do like the new
>behavior.
>
>To clarify, what happens when:
>
>x = zeros(2)
>if x:
>    print "true!"
>
>I see 3 possibilities:
>
>1) Keep it the way it is, no use of an array is permitted; an 
>   exception is raised.
>2) Print a warning message when it is used that way, and always
>   keep it at that, i.e., warning message only.
>3) Replicate Numeric behavior.
>
>Personally, I prefer 1) since I think there is too much confusion
>between whether false means an empty (len=0) array (like lists),
>or an array that contains all zeros, and that it is worth breaking
>backward compatibility on this (functions should be used to 
>distinguish which meaning is desired). 
>
>Perry
>
>[Option 2 on the previous question wins hands down]
>  
>
I favour option 1, otherwise there is a need to define just what it means.

Does it return a Boolean array indicating those cells which are nonzero, or
a single Boolean value indicating that some cell is nonzero, or a single
Boolean value indicating that all cells are nonzero?

Colin W.

>
>
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by: IBM Linux Tutorials
>Free Linux tutorial presented by Daniel Robbins, President and CEO of
>GenToo technologies. Learn everything from fundamentals to system
>administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
>_______________________________________________
>Numpy-discussion mailing list
>Num...@li...
>https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>  
>

Re: [Numpy-discussion] question about nonzero behavior in numarray

[Fernando P. <Fer...@co...>]

Perry Greenfield wrote:

> Personally, I prefer 1) since I think there is too much confusion
> between whether false means an empty (len=0) array (like lists),
> or an array that contains all zeros, and that it is worth breaking
> backward compatibility on this (functions should be used to 
> distinguish which meaning is desired). 

I tend to prefer keeping compatibility with Python list behavior.  I think 
that in general, numarray arrays should behave different from lists only when 
truly justified (for example, at least in numeric all ufuncs can be applied to 
a list of floats as much as to an array, which is nice).

Since anyone who knows python already knows that lists are true based on 
whether they contain _anything_:

In [9]: a=[]

In [10]: b=[0]

In [11]: if a:
    ....:     print 1
    ....:

In [12]: if b:
    ....:     print 1
    ....:
1


then I think it's a fair, clean extension to make NumArray behave similarly.

I think a good guiding principle is to keep the number of special cases for 
arrays at a strict minimum (there are obviously cases where it does make sense 
to have special cases).  Besides, this is an O(1) check, so it doesn't have 
the dangers of checking for all elements being zero.   Such a potentially 
expensive operation should _definitely_ happen only if explicitly requested, IMHO.

Regards,

f