Re: [Numpy-discussion] Defining custom types

[Jonathan W. <jon...@gm...>]

On 10/26/06, Travis Oliphant <oli...@ee...> wrote:
>
> But, what do you mean "inheriting" from NumPy's double for your scalar
> data-type.  This has significant implications.  To define a new
> data-type object (that doesn't build from the VOID data-type),  you need
> to flesh out the PyArray_Descr * structure and this can only be done in
> C.   Perhaps you are borrowing most entries in the structure builtin
> double type and then filling in a few differently like setitem and
> getitem?   Is that accurate?


Sorry, I should have been clearer. When I talk about inheritance, I mean of
the type underlying the array. For example, the built-in scalar double array
has an underlying type of PyDoubleArrType_Type. My underlying type is a
separate PyTypeObject. The interesting changes here are to tp_repr, tp_str,
and tp_as_number. The rest of the fields are inherited from
PyDoubleArrType_Type using the tp_base field.

The array itself has another statically defined type object of type
PyArray_Descr, which I'm creating with a PyObject_New call and filling in
with many of the entries from the descriptor returned by
PyArray_DescrFromType(NPY_DOUBLE), while overriding getitem and setitem to
handle PyObject* of type mxDateTime as you guessed.

> The interface used by Array_FromPyScalar does not conform with the
> > documentation's claim that a negative return value indicates an error.
>
> You must be talking about a different function.  Array_FromPyScalar is
> an internal function and not a C-API call.  It also returns a PyObject *
> not an integer. So, which function are you actually referring to?
>
> > The return code from setitem is not checked. Instead, the code depends
> > on a Python error being set.

This may be true, but how is it a problem?
>

It's just confusing as the documentation indicates that the setitem function
should return 0 for success and a negative number for failure. But within
Array_FromPyScalar, we have:

        ret->descr->f->setitem(op, ret->data, ret);

        if (PyErr_Occurred()) {
                Py_DECREF(ret);
                return NULL;
        } else {
                return (PyObject *)ret;
        }

So, someone reading the documentation could return -1 on failure without
setting the Python error flag, and the function would happily continue on
its way and fail to perform the proper casts.

>
> > I seem to be able to load values into the array, but I can't extract
> > anything out of the array, even to print it. In gdb I've verified that
> > loading DateTime.now() correctly puts a float representation of the
> > date into my array. However, if I try to get the value out, I get an
> > error:
> > >>> mxArr[0] = DateTime.now()
> > >>> mxArr[0]
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> >   File "/usr/lib/python2.4/site-packages/numpy/core/numeric.py", line
> > 391, in array_repr
> >     ', ', "array(")
> >   File "/usr/lib/python2.4/site-packages/numpy/core/arrayprint.py",
> > line 204, in array2string
> >     separator, prefix)
> >   File "/usr/lib/python2.4/site-packages/numpy/core/arrayprint.py",
> > line 160, in _array2string
> >     format = _floatFormat(data, precision, suppress_small)
> >   File "/usr/lib/python2.4/site-packages/numpy/core/arrayprint.py",
> > line 281, in _floatFormat
> >     non_zero = _uf.absolute(data.compress(_uf.not_equal(data, 0)))
> > TypeError: bad operand type for abs()
> >
> > I'm not sure why it's trying to call abs() on my object to print it.
>
> Because that's the implication of inheriting from a double.  It's just
> part of the code that tries to format your values into an array (notice
> the _floatFormat).  I actually borrowed this code from numarray so I
> can't speak to exactly what it's doing without more study.


Hmm, so does Numpy ignore the tp_repr and tp_str fields in the PyTypeObject
of the underlying type? I admittedly haven't had a chance to look at this
code closely yet.

> I have a separate PyNumberMethods attached to my object type, copied
> > from the float scalar type, and nb_absolute is set to 0. When I break
> > at the various functions I've registered, the last thing Numpy tries
> > to do is cast my custom data type to an object type (which it does so
> > successfully) via _broadcast_cast.
>
> Don't confuse the Python object  associated when an element of the array
> is extracted and the data-type of the array. Also don't confuse the
> PyNumberMethods of the scalar object with the ufuncs.  Defining
> PyNumberMethods won't usually give you the ability to calculate ufuncs.


Okay, is my understanding here correct? I am defining two type descriptors:
PyArray_Descr mxNumpyType - describes the Numpy array type.
PyTypeObject mxNumpyDataType - describes the data type of the contents of
the array (i.e. mxNumpyType->typeobj points to this), inherits from
PyDoubleArrType_Type and overrides some fields as mentioned above.

And the getitem and setitem functions are designed to only give/take
PyObject* of type mxDateTime.

I guess it's not clear to me whether the abs() referred to by the error is
an abs() ufunc or the nb_absolute pointer in the PyNumberMethods. Let me try
overriding ufuncs and get back to you...

Perhaps you just want to construct an "object" array of mxDateTime's.
> What is the reason you want to define an mxDateTime data-type?


Currently I am using an object array of mxDateTime's, but it's rather
frustrating that I can't treat them as normal floats internally since that's
really all they are.

Jonathan