numpy-discussion — Discussion list for all users of Numerical Python

[Numpy-discussion] Proposed record array behavior: the rest of the story

[Perry G. <pe...@st...>]

We now turn to the behavior of Records. We'll note that many of the current
proposals had been considered in the past but not implemented with more of a
'wait and see' attitude towards what was really necessary and a desire to
prevent too many ways of doing the same thing without seeing that there was
a real call for them.

This proposal deals with the behavior of record array 'items', i.e., what we
call Record objects now.

The primary issues that have been raised with regard to Record behavior are
summarized as follows:

1) Items should be tuples instead of Records
2) Items should be objects, but present tuple and/or dictionary consistent
behavior.
3) Field (or column) names should be accessible as Record (and record
array) attributes.

Issue 1: Should record array items be tuples instead of Records?

Francesc Alted made this suggestion recently. Essentially the argument is
that tuples are a natural way of representing records. Unfortunately, tuples
do not provide a means of accessing fields of a record by name, but only by
number. For this reason alone, tuples don't appear to be adequate. Francesc
proposed allowing dictionary-like indexing to record arrays to facilitate
the field access to tuple entries by name. However, it seems that if
rarr is a record array, that both rarr['column 1'][2] and rarr[2]['column
1'] should work, not just the former. So the short answer is "No".

It should be noted that using tuples will force another change in current
behavior. Note that the current Record objects are actually views into the
record array. Changing the value within a record object changes the record
array. Use of tuples won't allow that since tuples are not mutable. Whole
records must be changed in their entirety if single elements of record
arrays were set by and returned from tuples.

But his comments (and well as those of others) do point out a number of
problems with the current implementation that could be improved, and making
the Record object support tuple behaviors is quite reasonable. Hence:

Issue 2: Should record array items present tuple and/or dictionary
compatible behaviors?

The short answer is, yes, we do agree that they should. This includes many
of the proposals made including:

1) supporting all Tuple capabilities with the following differences:
    a) fields are mutable (unlike tuple items) so long as the assigned value
is coerceable to the expected type. For example the current methods of doing
so are:

>>> cell = oneRec.field(1)
>>> oneRec.setfield(1, newValue)

This proposal would allow:

>>> cell = oneRec[1]
>>> oneRec[1] = newValue

    b) slice assignments are permitted so long as it doesn't change the size
of the record (i.e., no insertion of extra items) and the items can be
assigned as permitted for a. E.g.,

OneCell[2:4] = (3, 'abc')

    c) __str__ will result in a display looking like that for tuples,
__repr__ will show a Record constructor

>>> print oneRec # as is currently implemented
(1.1, 2, 'abc', 3)
>>> oneRec
Record((1.1, 2, 'abc', 3), formats=['1Float32', '1Int16', '1a3', '1Int32'])
    names=['abc', 'c2', 'xyz', 'c4'])

(note that how best to handle formats is still being thought about)

2) supporting all Dictionary capabilities with the following differences:
    a) keys and items are ordered.
    b) keys are restricted to being integers or strings only
    c) new keys cannot be dynamically added or deleted as for dictionaries
    d) no support for any other dictionary capabilities that can change the
number or names of items
    e) __str__ will not show a result looking like a dictionary (see 1c)
    f) values must meet Record object required type (or be coerceable to it)
    
For example the current

>>> cell = onRec.field('c2')
>>> oneRec.setfield('c2', newValue)

And the proposed added indexing capability:

>>> cell = oneRec['c2']
>>> oneRec['c2'] = newValue

Issue 3: Field (or column) names should be accessible as Record (and record
array) attributes.

As much as the attribute approach has appeal for simple usage, the problems
of name collisions and mismatches between acceptable field names
and attribute names strikes us as it does Russell Owen as being very
problematic. The technique of using a special attribute as Francesc suggests
(in his case, cols) that contains the field name attributes solves the name
collision problem, but not the legality issue (particularly with regard to
illegal characters, it's hard to imagine easily remembered mappings between
legal attribute representations and the actual field name. We are inclined
to try to pass (for now anyway) on mapping fields to attributes in any way.
It seems to us that indexing by name should be convenient enough, as well as
fully flexible to really satisfy all needs (and is needed in any case since
attributes are a clumsy way to use field access when using a variable to
specify the field (yes, one can use  getattr(), but it's clumsy)

*******************************************

Record array behavior changes:

1) It will be possible to assign any sequence to a record array item so long
as the sequence contains the right number of fields, and each item of the
sequence can be coerced to what the record array expects for the
corresponding field of the record. (addressing numarray feature request
928473 by Russell Owen).

I.e.,

>>> recArr[1] = (2, 3.2, 'xyz', 3)

2) One may assign a record to a record array so long as the record matches
the format of the record format of the record array (current behavior).
3) Easier construction and initialization of recarrays with default field
values as requested in numarray bug report 928479)
4) Support for lists of field names and formats as detailed in numarray bug
report 928488.
5) Field name indexing for record arrays. It will be possible to index
record arrays with a field name, i.e., if the index is a string, then what
will be returned is a numarray/chararray for that column. (Note that it
won't be possible to index record arrays by field number for obvious
reasons).

I.e. Currently

>>> col = recArr.field('doc')

Can also be

>>> col = recArr['abc']

But the current

>>> col = recArr.field(1)

Cannot become

>>> col = recArr[1]

On the other hand, it will not be permitted to mix a field index with an
array index in the same brackets, e.g., rarr[10, 'column 2'] will not be
supported. Allowing indexing to have two different interpretations is a bit
worrying. But if record array items may be indexed in this manner, it seems
natural to permit the same indexing for the record array. Mixing the two
kinds of indexing in one index seems of limited usefulness in the first
place and it makes inheriting the existing indexing machinery for NDArrays
more complicated (any efficiency gains in avoiding the intermediate object
creation by using two separate index operations will likely be offset by the
slowness of handling much more complicated mixed indices). Perhaps someone
can argue for why mixing field indices with array indices is important, but
for now we will prohibit this mode of indexing.

This does point to a possible enhancement for the field indexing, namely
being able to provide the equivalent of index arrays (e.g., a list of field
names) to generate a new  record array with a subset of fields.

Are there any other issues that should be addressed for improving record
arrays?

Re: [Numpy-discussion] Follow-up Numarray header PEP

[Todd M. <jm...@st...>]

On Sun, 2004-07-18 at 17:24, ger...@gr... wrote:
> Hi Todd,
> 
> This is a follow-up on the 'header pep' discussion.

Great!  I was afraid you were going to disappear back into the ether.

Sorry I didn't respond to this yesterday...  I saw it but accidentally
marked it as "read" and then forgot about it as the day went on.

> The attachment numnum-0.1.tar.gz contains the sources for the
> extension modules pep and numnum.  At least on my systems, both
> modules behave as described in the 'numarray header PEP' when the
> extension modules implementing the C-API are not present (a situation
> not foreseen by the macros import_array() of Numeric and especially
> numarray).  

For numarray,  this was *definitely* foreseen at some point,  so I'm
wondering what doesn't work now...

> IMO, my solution is 'bona fide', but requires further
> testing.

I'll look it over today or tomorrow and comment more then.

> The pep module shows how to handle the colliding C-APIs of the Numeric
> and numarray extension modules and how to implement automagical
> conversion between Numeric and numarray arrays.

Nice;  the conversion code sounds like a good addition to me.  

> For a technical reason explained in the README, the hard work of doing
> the conversion between Numeric and numarray arrays has been delegated
> to the numnum module.  The numnum module is useful when one needs to
> convert from one array type to the other to use an extension module
> which only exists for the other type (eg. combining numarray's image
> processing extensions with pygame's Numeric interface):
> 
> Python 2.3+ (#1, Jan  7 2004, 09:17:35)
> [GCC 3.3.1 (SuSE Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numnum; import Numeric as np; import numarray as na
> >>> np1 = np.array([[1, 2], [3, 4]]); na1 = numnum.toNA(np1)
> >>> na2 = na.array([[1, 2, 3], [4, 5, 6]]); np2 = numnum.toNP(na2)
> >>> print type(np1); np1; type(np2); np2
> <type 'array'>
> array([[1, 2],
>        [3, 4]])
> <type 'array'>
> array([[1, 2, 3],
>        [4, 5, 6]],'i')
> >>> print type(na1); na1; type(na2); na2
> <class 'numarray.numarraycore.NumArray'>
> array([[1, 2],
>        [3, 4]])
> <class 'numarray.numarraycore.NumArray'>
> array([[1, 2, 3],
>        [4, 5, 6]])
> >>>
> 
> The pep module shows how to implement array processing functions which
> use the Numeric, numarray or Sequence C-API:
> 
> static PyObject *
> wysiwyg(PyObject *dummy, PyObject *args)
> {
>     PyObject *seq1, *seq2;
>     PyObject *result;
> 
>     if (!PyArg_ParseTuple(args, "OO", &seq1, &seq2))
>         return NULL;
> 
>     switch(API) {

We'll definitely need to cover API in the PEP.  There is a design choice
here which needs to be discussed some and any resulting consensus
documented.  I haven't looked at the attachment yet.

>     case NumericAPI:
>     {
>         PyObject *np1 = NN_API->toNP(seq1);
>         PyObject *np2 = NN_API->toNP(seq2);
>         result = np_wysiwyg(np1, np2);
>         Py_XDECREF(np1);
>         Py_XDECREF(np2);
>         break;
>     }
>     case NumarrayAPI:
>     {
>         PyObject *na1 = NN_API->toNA(seq1);
>         PyObject *na2 = NN_API->toNA(seq2);
>         result = na_wysiwyg(na1, na2);
>         Py_XDECREF(na1);
>         Py_XDECREF(na2);
>         break;
>     }
>     case SequenceAPI:
>         result = seq_wysiwyg(seq1, seq2);
>         break;
>     default:
>         PyErr_SetString(PyExc_RuntimeError, "Should never happen");
>         return 0;
>     }
> 
>     return result;
> }
> 
> See the README for an example session using the pep module showing that
> it is possible pass a mix of Numeric and numarray arrays to pep.wysiwyg().
> 
> Notes:
> 
> - it is straightforward to adapt pep and numnum so that the conversion
>   functions are linked into pep instead of imported.
> 
> - numnum is still 'proof of concept'.  I am thinking about methods to
>   make those techniques safer if the numarray (and Numeric?) header
>   files make it never into the Python headers (or make it safer to
>   use those techniques with Python < 2.4).  In particular it would
>   be helpful if the numerical C-APIs export an API version number,
>   similar to the versioning scheme of shared libraries -- see the
>   libtool->versioning info pages. 

I've thought about this a few times;  there's certainly a need for it in
numarray anyway... and I'm always one release too late.  Thanks for the
tip on libtool->versioning.

> I am considering three possibilities to release a more polished
> version of numnum (3rd party extension writers may prefer to link
> rather than import numnum's functionality):
> 
> 1. release it from PyQwt's project page
> 2. register an independent numnum project at SourceForge
> 3. hand numnum over to the Numerical Python project (frees me from
>    worrying about API changes).
> 
> Regards -- Gerard Vermeulen

(3) sounds best to me, for the same reason that numarray is a part of
the numpy project and because numnum is a Numeric/numarray tool.  There
is a small issue of sub-project organization (seperate bug tracking,
etc.),  but I figure if SF can handle Python,  it can handle Numeric,
numarray, and probably a number of other packages as well.  Something
like numnum should not be a problem and so to promote it, it would be
good to keep it where people can find it without having to look too
hard.

For now,  I'm again marking your post as "unread" and will revisit it
later this week.  In the meantime, thanks very much for your efforts
with numnum and the PEP.

Regards,
Todd

[Numpy-discussion] Follow-up Numarray header PEP

[<ger...@gr...>]

Hi Todd,

This is a follow-up on the 'header pep' discussion.

The attachment numnum-0.1.tar.gz contains the sources for the
extension modules pep and numnum.  At least on my systems, both
modules behave as described in the 'numarray header PEP' when the
extension modules implementing the C-API are not present (a situation
not foreseen by the macros import_array() of Numeric and especially
numarray).  IMO, my solution is 'bona fide', but requires further
testing.

The pep module shows how to handle the colliding C-APIs of the Numeric
and numarray extension modules and how to implement automagical
conversion between Numeric and numarray arrays.

For a technical reason explained in the README, the hard work of doing
the conversion between Numeric and numarray arrays has been delegated
to the numnum module.  The numnum module is useful when one needs to
convert from one array type to the other to use an extension module
which only exists for the other type (eg. combining numarray's image
processing extensions with pygame's Numeric interface):

Python 2.3+ (#1, Jan  7 2004, 09:17:35)
[GCC 3.3.1 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numnum; import Numeric as np; import numarray as na
>>> np1 = np.array([[1, 2], [3, 4]]); na1 = numnum.toNA(np1)
>>> na2 = na.array([[1, 2, 3], [4, 5, 6]]); np2 = numnum.toNP(na2)
>>> print type(np1); np1; type(np2); np2
<type 'array'>
array([[1, 2],
       [3, 4]])
<type 'array'>
array([[1, 2, 3],
       [4, 5, 6]],'i')
>>> print type(na1); na1; type(na2); na2
<class 'numarray.numarraycore.NumArray'>
array([[1, 2],
       [3, 4]])
<class 'numarray.numarraycore.NumArray'>
array([[1, 2, 3],
       [4, 5, 6]])
>>>

The pep module shows how to implement array processing functions which
use the Numeric, numarray or Sequence C-API:

static PyObject *
wysiwyg(PyObject *dummy, PyObject *args)
{
    PyObject *seq1, *seq2;
    PyObject *result;

    if (!PyArg_ParseTuple(args, "OO", &seq1, &seq2))
        return NULL;

    switch(API) {
    case NumericAPI:
    {
        PyObject *np1 = NN_API->toNP(seq1);
        PyObject *np2 = NN_API->toNP(seq2);
        result = np_wysiwyg(np1, np2);
        Py_XDECREF(np1);
        Py_XDECREF(np2);
        break;
    }
    case NumarrayAPI:
    {
        PyObject *na1 = NN_API->toNA(seq1);
        PyObject *na2 = NN_API->toNA(seq2);
        result = na_wysiwyg(na1, na2);
        Py_XDECREF(na1);
        Py_XDECREF(na2);
        break;
    }
    case SequenceAPI:
        result = seq_wysiwyg(seq1, seq2);
        break;
    default:
        PyErr_SetString(PyExc_RuntimeError, "Should never happen");
        return 0;
    }

    return result;
}

See the README for an example session using the pep module showing that
it is possible pass a mix of Numeric and numarray arrays to pep.wysiwyg().

Notes:

- it is straightforward to adapt pep and numnum so that the conversion
  functions are linked into pep instead of imported.

- numnum is still 'proof of concept'.  I am thinking about methods to
  make those techniques safer if the numarray (and Numeric?) header
  files make it never into the Python headers (or make it safer to
  use those techniques with Python < 2.4).  In particular it would
  be helpful if the numerical C-APIs export an API version number,
  similar to the versioning scheme of shared libraries -- see the
  libtool->versioning info pages. 

I am considering three possibilities to release a more polished
version of numnum (3rd party extension writers may prefer to link
rather than import numnum's functionality):

1. release it from PyQwt's project page
2. register an independent numnum project at SourceForge
3. hand numnum over to the Numerical Python project (frees me from
   worrying about API changes).


Regards -- Gerard Vermeulen

Re: [Numpy-discussion] RecArray.tolist() suggestion

[Colin J. W. <cj...@sy...>]

Paul Barrett wrote:

> Russell E Owen wrote:
>
>>> A Divendres 16 Juliol 2004 02:21, Colin J. Williams va escriure:
>>>
>>>>  To allow for multi-word column names, assignment could replace a 
>>>> space
>>>>  by an underscore
>>>>  and, in retrieval, the reverse could be done - ie. underscore 
>>>> would be
>>>>  banned for a column name.
>>>
>>>
>>>
>>> That's not so easy. What about other chars like '/&%@$()' that 
>>> cannot be
>>> part of python names? Finding a biunivocal map between them and allowed
>>> chars would be difficult (if possible at all). Besides, the resulting
>>> colnames might become a real mess.
>>
>>
>>
>> Personally, I think the idea of allowing access to fields via 
>> attributes is fatally flawed. The problems raised (non-obvious 
>> mapping between field names with special characters and allowable 
>> attribute names and also the collision with existing instance 
>> variable and method names) clearly show it would be forced and 
>> non-pythonic.
>
>
> +1

Paul,

Below, I've appended my response to Francesc's 08:36 message, it was 
copied to the list
but does not appear in the archive.

>
> It also make it difficult to do the following:
>
> a = item[:10, ('age', 'surname', 'firstname')]
>
> where field (or column) 1 is 'firstname, field 2 is 'surname', and 
> field 10 is 'age'.
>
> -- Paul

Could you clarify what you have in mind here please?  Is this a proposed
extension to records.py, as it exists in version 1.0?

Colin W.
------------------------------------------------------------------------
Yes, if the objective is to include special characters or facilitate 
multi-lingual columns names and
it probably should be, then my suggestion is quite inadequate.

Perhaps there could be a simple name -> column number mapping in place 
of _names.  References
to a column, or a field in a record, could then be through this dictionary.

Basic access to data in a record would be by position number, rather 
than name, but the dictionary
would facilitate access by name.

Data could be referenced either through the column name: r1.c2[1] or
through the record r1[1].c2, with the possibility that the index is 
multi-dimensional in either case.

Colin W.

[Numpy-discussion] multidimensional record arrays

[Jin-chung H. <hs...@st...>]

There have been a number of questions and suggestions about
how the record array facility in numarray could be improved.
We've been talking about these internally and thought it would
be useful to air some proposals along with discussions of the
rationale behind each proposal as well discussions of drawbacks,
and some remaining open questions. Rather than do this in one
long message, we will do this in pieces. The first addresses
how to improve handling multidimensional record arrays.

These will not discuss how or when we implement the proposed
enhancements or changes. We first want to come to some
consensus (or lacking that, decision) first about what the
target should be.

*********************************************************

Proposal for records module enhancement, to handle record arrays of
dimension (rank) higher than 1.

Background:

The current records module in numarray doesn't handle record arrays of
dimension higher than one well.  Even though most of the infrastructure
for higher dimensionality is already in place, the current implementation
for the record arrays was based on the implicit assumption that record
arrays are 1-D. This limitation is reflected in the areas of input user
interface, indexing, and output.

The indexing and output are more straightforward to modify, so I'll
discuss it first.

Although it is possible to create a multi-dimensional record array,
indexing does not work properly for 2 or more dimensions.  For example,
for a 2-D record array r, r[i,j] does not give correct result (but r[i][j]
does). This will be fixed.

At present, a user cannot print record arrays higher than 1-D.  This will
also be fixed as well as incorporating some numarray features (e.g.,
printing only the beginning and end of an array for large arrays--as is done
for numarrays now).

Input Interface:

There are currently several different ways to construct the record array
using the array() function These include setting the buffer argument to:

(1) None
(2) File object
(3) String object or appropriate buffer object (i.e., binary data)
(4) a list of records (in the form of sequences),
    for example:  [(1,'abc', 2.3), (2,'xyz', 2.4)]
(5) a list of numarrays/chararrays for each field (e.g., effectively
 'zipping' the arrays into records)

The first three types of input are very general and can be used to generate
multi-dimensional record arrays in the current implementation.  All these
options need to specify the "shape" argument.

The input options that do not work for multi-dimensional record arrays now
are the last two.

Option 4 (sequence of 'records')

If a user has a multi-dimensional record array and if one or more field is
also a multidimensional array, using this option is potentially confusing
since there can be ambiguity regarding what part of a nested sequence
structure is the structure of the record array and what should be considered
part of the record since record elements themselves may be arrays. (Some of
the same issues arise for object arrays)

As an example:

--> r=rec.array([([1,2],[3,4]),([11,12],[13,14])])

could be interpreted as a 1-D record array, where each cell is an
(num)array:

RecArray[
(array([1, 2]), array([3, 4])),
(array([11, 12]), array([13, 14]))
]

or a 2-D record array, where each cell is just a number:

RecArray(
             [[(1, 2),
              (3, 4)],

             [(11, 12),
              (13, 14)]])

Thus we propose a new argument "rank" (following the convention used in
object arrays) to specify the dimensionality of the output record array.  In
the first example above, rank is 1, and the second example rank=2.  If rank
is set to None, the highest possible rank will be assumed (in this example,
2).

We propose to eventually generalize that to accept any sequence object for
the array structure (though there will be the same requirement that exist
for
other arrays that the nested sequences be of the same type). As would be
expected, strings are not permitted as the enclosing sequence. In this
future implementation the record 'item' itself must either be:

1) A tuple
2) A subclass of tuple
3) A Record object (this may be taken care of by 2 if we make Record
   a subclass of tuple; this will be discussed in a subsequent proposal.

This requirement allows distinguishing the sequence of records from Option 5
below. For tuples (or tuple derived elements), the items of the tuple must
be one of the following: basic data types such as int, float, boolean, or
string; a numarray or chararray; or an object that can be converted to a
numarray or chararray.

Option 5 (List of Arrays)

Using a list of arrays to construct an N-D record array should be easier
Than using the previous option.  The input syntax is simply:

[array1, array2, array3,...]

The shape of the record array will be determined from the shape of the input
arrays as described below. All the user needs to do is to construct the
arrays in the list.  There is, similar to option 4, a possible ambiguity:
if all the arrays are of the shape, say, (2,3), then the user may intend a
1-D record array of 2 rows while each cell is an array of shape 3, or a 2-D
record array of shape (2,3) while each cell is a single number of string.
Thus, the user must either explicitly specify the "shape" or "rank".

We propose the following behavior via examples:

Example 1:

given:

array1.shape=(2,3,4,5)
array2.shape=(2,3,4)
array3.shape=(2,3)

Rank can only be specified as rank=1 (the record array's shape will then be
(2,)) or rank=2 (the record array's shape will then be (2,3)). For rank=None
the record shape will be (2,3), i.e. the "highest common denominator": each
cell in the first field will be an array of shape (4,5), each cell in the
second field will be an array of shape (4,), and each cell in the 3rd field
will be a single number or a string.  If "shape" is specified, it will take
precedence over "rank" and its allowed value in this example will be either
2, or (2,3).

Example 2:

array1.shape=(3,4,5)
array2.shape=(4,5)

this will raise exception because the 'slowest' axes do not match.

*********

For both the sequence of records and list-of-arrays input options, we
Propose the default value for "rank" be None (current default is 1).
This gives consistent behavior with object arrays but does change the
current behavior.

Also for both cases specifying a shape inconsistent with the supplied data
will raise an exception.

[Numpy-discussion] I move your "Bugs" reports...

[Todd M. <jm...@st...>]

Not infrequently even very experienced numarray contributors file bug
reports in the numpy "Bugs" tracker.  Because numpy is a shared SF
project with both Numeric and numarray,  numarray bugs are actually
tracked in the "Numarray Bugs" tracker, here:

http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse

"Numarray Bugs" can also be found through the "Tracker" link at the top
of any numpy SF web page.  

So, don't worry, your painstaking reports are not getting deleted,
they're getting relocated to a place where *only* numarray bugs live. 
There's probably a better way to do this,  but until I find it or
someone tells me about it,  I thought I should tell everyone what's
going on.  Thanks to everybody who takes the time to fill out bug
reports to make numarray better...

Regards,
Todd

Re: [Numpy-discussion] RecArray.tolist() suggestion

[Paul B. <ba...@st...>]

Russell E Owen wrote:

>> A Divendres 16 Juliol 2004 02:21, Colin J. Williams va escriure:
>>
>>>  To allow for multi-word column names, assignment could replace a space
>>>  by an underscore
>>>  and, in retrieval, the reverse could be done - ie. underscore would be
>>>  banned for a column name.
>>
>>
>> That's not so easy. What about other chars like '/&%@$()' that cannot be
>> part of python names? Finding a biunivocal map between them and allowed
>> chars would be difficult (if possible at all). Besides, the resulting
>> colnames might become a real mess.
>
>
> Personally, I think the idea of allowing access to fields via 
> attributes is fatally flawed. The problems raised (non-obvious mapping 
> between field names with special characters and allowable attribute 
> names and also the collision with existing instance variable and 
> method names) clearly show it would be forced and non-pythonic.

+1

It also make it difficult to do the following:

a = item[:10, ('age', 'surname', 'firstname')]

where field (or column) 1 is 'firstname, field 2 is 'surname', and field 
10 is 'age'.

 -- Paul

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Branch
FAX:   410-338-4767    Baltimore, MD 21218

Re: [Numpy-discussion] RecArray.tolist() suggestion

[Russell E O. <rowen@u.washington.edu>]

>A Divendres 16 Juliol 2004 02:21, Colin J. Williams va escriure:
>>  To allow for multi-word column names, assignment could replace a space
>>  by an underscore
>>  and, in retrieval, the reverse could be done - ie. underscore would be
>>  banned for a column name.
>
>That's not so easy. What about other chars like '/&%@$()' that cannot be
>part of python names? Finding a biunivocal map between them and allowed
>chars would be difficult (if possible at all). Besides, the resulting
>colnames might become a real mess.

Personally, I think the idea of allowing access to fields via 
attributes is fatally flawed. The problems raised (non-obvious 
mapping between field names with special characters and allowable 
attribute names and also the collision with existing instance 
variable and method names) clearly show it would be forced and 
non-pythonic.

The obvious solution seems to be some combination of the dict 
interface (an ordered dict that keeps its keys in original field 
order) and the list interface.

My personal leaning is:
- Offer most of the dict methods, including __get/setitem__, keys, 
values and all iterators but NOT set_default pop_item or anything 
else that adds or deletes a field.
- Offer the list version of __get/setitem__, as well, but NONE of 
list's methods.
- Make the default iterator iterate over values, not keys (field names),
   i.e have the item act like a list, not a dict when used as an iterator.

In other words, the following all work (where item is one element of 
a numarray.record array):
item[0] = 10  # set value of field 0 to 10
x = item[0:5]  # get value of fields 0 through 4
item[:] = list of replacement values
item["afield"] = 10
"%s(afield)" % item
the methods iterkeys, itervalues, iteritems, keys, values, has_key all work
the method update might work, but it's an error to add new fields

-- Russell

P.S. Folks are welcome to use my ordered dictionary implementation 
RO.Alg.OrderedDictionary, which is part of the RO package 
<http://www.astro.washington.edu/rowen/ROPython.html>. It is fully 
standalone (despite its location in my hierarchy) and is used in 
production code.

Re: [Numpy-discussion] RecArray.tolist() suggestion

[Colin J. W. <cj...@sy...>]

Francesc Alted wrote:

>A Divendres 16 Juliol 2004 02:21, Colin J. Williams va escriure:
>  
>
>>To allow for multi-word column names, assignment could replace a space 
>>by an underscore
>>and, in retrieval, the reverse could be done - ie. underscore would be 
>>banned for a column name.
>>    
>>
>
>That's not so easy. What about other chars like '/&%@$()' that cannot be
>part of python names? Finding a biunivocal map between them and allowed
>chars would be difficult (if possible at all). Besides, the resulting
>colnames might become a real mess.
>
>Regards,
>  
>
Yes, if the objective is to include special characters or facilitate 
multi-lingual columns names and
it probably should be, then my suggestion is quite inadequate.

Perhaps there could be a simple name -> column number mapping in place 
of _names.  References
to a column, or a field in a record, could then be through this dictionary.

Basic access to data in a record would be by position number, rather 
than name, but the dictionary
would facilitate access by name.

Data could be referenced either through the column name: r1.c2[1] or
through the record r1[1].c2, with the possibility that the index is 
multi-dimensional in either case.

Colin W.

Re: [Numpy-discussion] RecArray.tolist() suggestion

[Francesc A. <fa...@py...>]

A Divendres 16 Juliol 2004 02:21, Colin J. Williams va escriure:
> To allow for multi-word column names, assignment could replace a space 
> by an underscore
> and, in retrieval, the reverse could be done - ie. underscore would be 
> banned for a column name.

That's not so easy. What about other chars like '/&%@$()' that cannot be
part of python names? Finding a biunivocal map between them and allowed
chars would be difficult (if possible at all). Besides, the resulting
colnames might become a real mess.

Regards,

-- 
Francesc Alted

Re: [Numpy-discussion] RecArray.tolist() suggestion

[Colin J. W. <cj...@sy...>]

Perry Greenfield wrote:

>Francesc Alted wrote:
>  
>
>>A Dijous 15 Juliol 2004 17:21, Colin J. Williams va escriure:
>>    
>>
>>>>What I propose is to be able to say:
>>>>        
>>>>
>>>>>>>r["c1"][1]
>>>>>>>              
>>>>>>>
>>>I would suggest going a step beyond this, so that one can have r.c1[1],
>>>see the script below.
>>>      
>>>
>>Yeah. I've implemented something similar to access column elements for
>>pytables Table objects. However, the problem in this case is that
>>there are
>>already attributes that "pollute" the column namespace, so that a column
>>named "size" collides with the size() method.
>>
>>    
>>
>The idea of mapping field names to attributes occurs to everyone
>quickly, but for the reasons Francesc gives (as well as another I'll
>mention) we were reluctant to implement it. The other reason is that
>it would be nice to allow field names that are not legal attributes
>(e.g., that include spaces or other illegal attribute characters).
>There are potentially people with data in databases or other similar
>formats that would like to map field name exactly. Well certainly
>one can still use the attribute approach and not support all field
>names (or column, or col...) it does introduce another glitch in
>the user interface when it works only for a subset of legal names.
>  
>
It would, I suggest, not be unduly restrictive to bar the existing 
attribute names but, if that's not
acceptable, Francesc has suggested the.col workaround, although I would 
prefer to avoid the
added clutter.

Incidentally, there is no current protection against wiping out an 
existing method:
[Dbg]>>> r1.size= 0
[Dbg]>>> r1.size
0
[Dbg]>>>

>  
>
>>I came up with a solution by adding a new "cols" attribute to the Table
>>object that is an instance of a simple class named Cols with no attributes
>>that can pollute the namespace (except some starting by "__" or "_v_").
>>Then, it is just a matter of provide functionality to access the different
>>columns. In that case, when a reference of a column is made,
>>another object
>>(instance of Column class) is returned. This Column object is basically an
>>accessor to column values with a __getitem__() and __setitem__() methods.
>>That might sound complicated, but it is not. I'm attaching part of the
>>relevant code below.
>>
>>I personally like that solution in the context of pytables because it
>>extends the "natural naming" convention quite naturally. A
>>similar approach
>>could be applied to RecArray objects as well, although numarray might (and
>>probably do) have other usage conventions.
>>
>>    
>>
>>>I have not explored the assignment of a value to r.c1.[1], but it seems
>>>to be achievable.
>>>      
>>>
>>in the schema I've just proposed the next should be feasible:
>>
>>value = r.cols.c1[1]
>>r.cols.c1[1] = value
>>
>>    
>>
>This solution avoids name collisions but doesn't handle the other
>problem. This is worth considering, but I thought I'd hear comments
>about the other issue before deciding it (there is also the
>"more than one way" issue as well; but this guideline seems to bend
>quite often to pragmatic concerns).
>
To allow for multi-word column names, assignment could replace a space 
by an underscore
and, in retrieval, the reverse could be done - ie. underscore would be 
banned for a column name.

Colin W.

>
>We're still chewing on all the other issues and plan to start floating
>some proposals, rationales and questions before long.
>
>Perry
>
>
>  
>

Re: [Numpy-discussion] RecArray.tolist() suggestion

[Francesc A. <fa...@py...>]

A Dijous 15 Juliol 2004 19:37, Perry Greenfield va escriure:
> formats that would like to map field name exactly. Well certainly
> one can still use the attribute approach and not support all field
> names (or column, or col...) it does introduce another glitch in
> the user interface when it works only for a subset of legal names.

Yep. I forgot that issue. My particular workaround on that was to provide an
optional trMap dictionary during Table (in our case, RecArray) creation time
to map those original names that are not valid python names by valid ones. 

That would read something like:

>>> r=records.array([(1,"as")], "1i4,1a2",
                    names=["c 1", "c2"], trMap={"c1": "c 1"})

that would indicate that the "c 1" column which is not a valid python name
(it has an space in the middle) can be accessed by using "c1" string, which
is a valid python id. That way, r.cols.c1 would access column "c 1".

And although I must admit that this solution is not very elegant, it allows
to cope with those situations where the columns are not valid python names.

-- 
Francesc Alted

RE: [Numpy-discussion] RecArray.tolist() suggestion

[Perry G. <pe...@st...>]

Francesc Alted wrote:
> A Dijous 15 Juliol 2004 17:21, Colin J. Williams va escriure:
> > >What I propose is to be able to say:
> > >>>>r["c1"][1]
> > I would suggest going a step beyond this, so that one can have r.c1[1],
> > see the script below.
>
> Yeah. I've implemented something similar to access column elements for
> pytables Table objects. However, the problem in this case is that
> there are
> already attributes that "pollute" the column namespace, so that a column
> named "size" collides with the size() method.
>
The idea of mapping field names to attributes occurs to everyone
quickly, but for the reasons Francesc gives (as well as another I'll
mention) we were reluctant to implement it. The other reason is that
it would be nice to allow field names that are not legal attributes
(e.g., that include spaces or other illegal attribute characters).
There are potentially people with data in databases or other similar
formats that would like to map field name exactly. Well certainly
one can still use the attribute approach and not support all field
names (or column, or col...) it does introduce another glitch in
the user interface when it works only for a subset of legal names.

> I came up with a solution by adding a new "cols" attribute to the Table
> object that is an instance of a simple class named Cols with no attributes
> that can pollute the namespace (except some starting by "__" or "_v_").
> Then, it is just a matter of provide functionality to access the different
> columns. In that case, when a reference of a column is made,
> another object
> (instance of Column class) is returned. This Column object is basically an
> accessor to column values with a __getitem__() and __setitem__() methods.
> That might sound complicated, but it is not. I'm attaching part of the
> relevant code below.
>
> I personally like that solution in the context of pytables because it
> extends the "natural naming" convention quite naturally. A
> similar approach
> could be applied to RecArray objects as well, although numarray might (and
> probably do) have other usage conventions.
>
> > I have not explored the assignment of a value to r.c1.[1], but it seems
> > to be achievable.
>
> in the schema I've just proposed the next should be feasible:
>
> value = r.cols.c1[1]
> r.cols.c1[1] = value
>
This solution avoids name collisions but doesn't handle the other
problem. This is worth considering, but I thought I'd hear comments
about the other issue before deciding it (there is also the
"more than one way" issue as well; but this guideline seems to bend
quite often to pragmatic concerns).

We're still chewing on all the other issues and plan to start floating
some proposals, rationales and questions before long.

Perry

Re: [Numpy-discussion] RecArray.tolist() suggestion

[Francesc A. <fa...@py...>]

A Dijous 15 Juliol 2004 17:21, Colin J. Williams va escriure:
> >What I propose is to be able to say:
> >>>>r["c1"][1]
> I would suggest going a step beyond this, so that one can have r.c1[1], 
> see the script below.

Yeah. I've implemented something similar to access column elements for
pytables Table objects. However, the problem in this case is that there are
already attributes that "pollute" the column namespace, so that a column
named "size" collides with the size() method.

I came up with a solution by adding a new "cols" attribute to the Table
object that is an instance of a simple class named Cols with no attributes
that can pollute the namespace (except some starting by "__" or "_v_").
Then, it is just a matter of provide functionality to access the different
columns. In that case, when a reference of a column is made, another object
(instance of Column class) is returned. This Column object is basically an
accessor to column values with a __getitem__() and __setitem__() methods.
That might sound complicated, but it is not. I'm attaching part of the
relevant code below.

I personally like that solution in the context of pytables because it
extends the "natural naming" convention quite naturally. A similar approach
could be applied to RecArray objects as well, although numarray might (and
probably do) have other usage conventions.

> I have not explored the assignment of a value to r.c1.[1], but it seems 
> to be achievable.

in the schema I've just proposed the next should be feasible:

value = r.cols.c1[1]
r.cols.c1[1] = value


-- 
Francesc Alted


-----------------------------------------------------------------
class Cols(object):
    """This is a container for columns in a table

    It provides methods to get Column objects that gives access to the
    data in the column.

    Like with Group instances and AttributeSet instances, the natural
    naming is used, i.e. you can access the columns on a table like if
    they were normal Cols attributes.
    
    Instance variables:

        _v_table -- The parent table instance
        _v_colnames -- List with all column names

    Methods:
    
        __getitem__(colname)
        
    """

    def __init__(self, table):
        """Create the container to keep the column information.

        table -- The parent table
        
        """
        self.__dict__["_v_table"] = table
        self.__dict__["_v_colnames"] = table.colnames
        # Put the column in the local dictionary
        for name in table.colnames:
            self.__dict__[name] = Column(table, name)

    def __len__(self):
        return self._v_table.nrows

    def __getitem__(self, name):
        """Get the column named "name" as an item."""

        if not isinstance(name, types.StringType):
            raise TypeError, \
"Only strings are allowed as keys of a Cols instance. You passed object: %s" % name
        # If attribute does not exist, return None
        if not name in self._v_colnames:
            raise AttributeError, \
"Column name '%s' does not exist in table:\n'%s'" % (name, str(self._v_table))

        return self.__dict__[name]

    def __str__(self):
        """The string representation for this object."""
        # The pathname
        pathname = self._v_table._v_pathname
        # Get this class name
        classname = self.__class__.__name__
        # The number of columns
        ncols = len(self._v_colnames)
        return "%s.cols (%s), %s columns" % (pathname, classname, ncols)

    def __repr__(self):
        """A detailed string representation for this object."""

        out = str(self) + "\n"
        for name in self._v_colnames:
            # Get this class name
            classname = getattr(self, name).__class__.__name__
            # The shape for this column
            shape = self._v_table.colshapes[name]
            # The type
            tcol = self._v_table.coltypes[name]
            if shape == 1:
                shape = (1,)
            out += "  %s (%s%s, %s)" % (name, classname, shape, tcol) + "\n"
        return out

               
class Column(object):
    """This is an accessor for the actual data in a table column

    Instance variables:

        table -- The parent table instance
        name -- The name of the associated column

    Methods:
    
        __getitem__(key)
        
    """

    def __init__(self, table, name):
        """Create the container to keep the column information.

        table -- The parent table instance
        name -- The name of the column that is associated with this object
        
        """
        self.table = table
        self.name = name
        # Check whether an index exists or not
        iname = "_i_"+table.name+"_"+name
        self.index = None
        if iname in table._v_parent._v_indices:
            self.index = Index(where=self, name=iname,
                               expectedrows=table._v_expectedrows)
        else:
            self.index = None

    def __getitem__(self, key):
        """Returns a column element or slice

        It takes different actions depending on the type of the
        "key" parameter:

        If "key" is an integer, the corresponding element in the
        column is returned as a NumArray/CharArray, or a scalar
        object, depending on its shape. If "key" is a slice, the row
        slice determined by this slice is returned as a NumArray or
        CharArray object (whatever is appropriate).

        """
        
        if isinstance(key, types.IntType):
            if key < 0:
                # To support negative values
                key += self.table.nrows
            (start, stop, step) = processRange(self.table.nrows, key, key+1, 1)
            return self.table._read(start, stop, step, self.name, None)[0]
        elif isinstance(key, types.SliceType):
            (start, stop, step) = processRange(self.table.nrows, key.start,
                                               key.stop, key.step)
            return self.table._read(start, stop, step, self.name, None)
        else:
            raise TypeError, "'%s' key type is not valid in this context" % \
                  (key)

    def __str__(self):
        """The string representation for this object."""
        # The pathname
        pathname = self.table._v_pathname
        # Get this class name
        classname = self.__class__.__name__
        # The shape for this column
        shape = self.table.colshapes[self.name]
        if shape == 1:
            shape = (1,)
        # The type
        tcol = self.table.coltypes[self.name]
        return "%s.cols.%s (%s%s, %s)" % (pathname, self.name,
                                          classname, shape, tcol)

    def __repr__(self):
        """A detailed string representation for this object."""
        return str(self)
               

Re: [Numpy-discussion] RecArray.tolist() suggestion

[Colin J. W. <cj...@sy...>]

Francesc Alted wrote:

>A Dimarts 13 Juliol 2004 10:28, Francesc Alted va escriure:
>  
>
>>A Dilluns 12 Juliol 2004 23:14, Perry Greenfield va escriure:
>>    
>>
>>>What I'm wondering about is what a single element of a record array
>>>should be. Returning a tuple has an undeniable simplicity to it.
>>>      
>>>
>>Yeah, this why I'm strongly biased toward this possibility.
>>
>>    
>>
>>>On the other hand, we've been using recarrays that allow naming the
>>>various columns (which we refer to as "fields").  If one can refer
>>>to fields of a recarray, shouldn't one be able to refer to a field
>>>(by name) of one of it's elements? Or are you proposing that basic
>>>recarrays not have that sort of capability (something added by a
>>>subclass)?
>>>      
>>>
>>Well, I'm not sure about that. But just in case most of people would like to
>>access records by field as well as by index, I would advocate for the
>>possibility that the Record instances would behave as similar as possible as
>>a tuple (or dictionary?). That include creating appropriate __str__() *and*
>>__repr__() methods as well as __getitem__() that supports both name fields
>>and indices. I'm not sure about whether providing an __getattr__() method
>>would ok, but for the sake of simplicity and in order to have (preferably)
>>only one way to do things, I would say no.
>>    
>>
>
>I've been thinking that one can made compatible to return a tuple on a
>single element of a RecArray and still being able to retrieve a field by
>name is to play with the RecArray.__getitem__ and let it to suport key names
>in addition to indices. This would be better seen as an example:
>
>Right now, one can say:
>
>  
>
>>>>r=records.array([(1,"asds", 24.),(2,"pwdw", 48.)], "1i4,1a4,1f8")
>>>>r._fields["c1"]
>>>>        
>>>>
>array([1, 2])
>  
>
>>>>r._fields["c1"][1]
>>>>        
>>>>
>2
>
>What I propose is to be able to say:
>
>  
>
>>>>r["c1"]
>>>>        
>>>>
>array([1, 2])
>  
>
>>>>r["c1"][1]
>>>>
I would suggest going a step beyond this, so that one can have r.c1[1], 
see the script below.
I have not explored the assignment of a value to r.c1.[1], but it seems 
to be achievable.
If changes along this line are acceptable, it is suggested that fields 
be renamed cols, or some
such, to indicate its wider impact.

Colin W.

>>>>        
>>>>
>2
>
>Which would replace the notation:
>
>  
>
>>>>r[1]["c1"]
>>>>        
>>>>
>2
>
>which was recently suggested.
>
>I.e. the suggestion is to realize RecArrays as a collection of columns,
>as well as a collection of rows.
>  
>
# tRecord.py to explore RecArray

import numarray.records as _rec
import sys
#
class Rec1(_rec.RecArray):
  def __new__(cls, buffer, formats, shape=0, names=None, byteoffset=0,
                 bytestride=None, byteorder=sys.byteorder, aligned=0):
    # This calls RecArray.__init__ - reason unclear.
    # Why can't the instance be fully created by RecArray.__init__?
    return _rec.RecArray.__new__(cls, buffer, formats=formats, 
shape=shape, names=names,
                         byteorder=byteorder, aligned=aligned)

  def __init__(self, buffer, formats, shape=0, names=None, byteoffset=0,
               bytestride=None, byteorder=sys.byteorder, aligned=0):
 
    arr= _rec.array(buffer, formats=formats, shape=shape, names=names,
                   byteorder=byteorder, aligned=aligned)

    self.__setstate__(arr.__getstate__())

  def __getattr__(self, name):
    # We reach here if the attribute does not belong to the basic Rec1 set
    return self._fields[name]
       
  def __getattribute__(self, name):
    return _rec.RecArray.__getattribute__(self, name)
 
  def __repr__(self):
    return self.__class__.__name__ + _rec.RecArray.__repr__(self)[8:]

  def __setattr__(self, name, value):
    return _rec.RecArray.__setattr__(self, name, value)
 
  def __str__(self):
    return self.__class__.__name__ + _rec.RecArray.__str__(self)[8:]  
   
if __name__ == '__main__':   
  # Frances Alted 13-Jul-04 05:06
  r= _rec.array([(1,"asds", 24.),(2,"pwdw", 48.)], "1i4,1a4,1f8")
  print r._fields["c1"]
  print r._fields["c1"][1]
  r1= Rec1([(1,"asds", 24.),(2,"pwdw", 48.)], "1i4,1a4,1f8")
  print r1._fields["c1"]
  print r1._fields["c1"][1]
  #
  r1.zz= 99                       #  acceptable
  print r1.c1
  print r1.c1[1]
  try:
    x= r1.ugh
  except:
    print 'ugh not recognized as an attribute'
'''
  The above delivers:
[1 2]
2
[1 2]
2
[1 2]
2
ugh not recognized as an attribute
'''
 

Re: [Numpy-discussion] NA.dot transposing in place

[Todd M. <jm...@st...>]

On Thu, 2004-07-15 at 10:50, Curzio Basso wrote:
> Todd Miller wrote:
> 
> > It's a bug introduced in numarray-1.0.  It'll be fixed for 1.1 in a
> > couple weeks.
> 
> Ah, ok. Is it related with the bug announced a couple of days ago?

Only peripherally.  The Numeric compatibility layer problem was
discovered as a result of porting a bunch of Numeric functions to
numarray... ports done to try to get better small array speed. 
Similarly, the  setup for matrixmultiply was moved into C for
numarray-1.0... to try to get better small array speed.

numarray-1.0 is disappointingly buggy,  but the interest generated by
the 1.0 moniker is making the open source model work well so I think 1.1
will be much more solid as a result of strong user feedback.  So, thanks
for the report.

Regards,
Todd

Re: [Numpy-discussion] NA.dot transposing in place

[Curzio B. <cur...@un...>]

Todd Miller wrote:

> It's a bug introduced in numarray-1.0.  It'll be fixed for 1.1 in a
> couple weeks.

Ah, ok. Is it related with the bug announced a couple of days ago?

Re: [Numpy-discussion] sum() and mean() broken?

[Todd M. <jm...@st...>]

On Thu, 2004-07-15 at 10:35, Gary Ruben wrote:
> Thanks Todd,
> It's under Win98 as you suspected and the selftest definitely doesn't pass.
> Are you planning on supporting Win98? 

I'm planning to debug this particular problem because I'm concerned that
it's just latent in the newer windows variants.  To the degree that
Win98 is "free" under the umbrella of win32, it will continue to be
supported.  An ongoing issue will likely be that Win98 testing doesn't
get done on a regular basis... just as problems are reported.

Regards,
Todd 

Re: [Numpy-discussion] NA.dot transposing in place

[Todd M. <jm...@st...>]

On Thu, 2004-07-15 at 10:21, Curzio Basso wrote:
> Hi all.
> 
> I wonder if anyone noticed the following behaviour (new in 1.0) of the 
> dot/matrixmultiply functions:
> 
>  >>> alpha = NA.arange(10, shape = (10,1))
> 
>  >>> beta = NA.arange(10, shape = (10,1))
> 
>  >>> NA.dot(alpha, alpha)
> array([[285]])
> 
>  >>> alpha.shape # here it looks like it's doing the transpose in place
> (1, 10)
> 
>  >>> NA.dot(beta, alpha)
> array([[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
>         [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],
>         [ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18],
>         [ 0,  3,  6,  9, 12, 15, 18, 21, 24, 27],
>         [ 0,  4,  8, 12, 16, 20, 24, 28, 32, 36],
>         [ 0,  5, 10, 15, 20, 25, 30, 35, 40, 45],
>         [ 0,  6, 12, 18, 24, 30, 36, 42, 48, 54],
>         [ 0,  7, 14, 21, 28, 35, 42, 49, 56, 63],
>         [ 0,  8, 16, 24, 32, 40, 48, 56, 64, 72],
>         [ 0,  9, 18, 27, 36, 45, 54, 63, 72, 81]])
> 
>  >>> alpha.shape, beta.shape # but not the second time
> ((1, 10), (10, 1))
> 
> -------------------------------------------------
> 
> Can someone explain me what's going on?

It's a bug introduced in numarray-1.0.  It'll be fixed for 1.1 in a
couple weeks.

Regards,
Todd

Re: [Numpy-discussion] sum() and mean() broken?

[Gary R. <ga...@em...>]

Thanks Todd,
It's under Win98 as you suspected and the selftest definitely doesn't pass.
Are you planning on supporting Win98? If so, I'll revert to numarray 0.9. Otherwise, I'll just use Numeric for this task and restrict playing with numarray 1.0 to my Win2k laptop.
thanks,
Gary
-- 
_______________________________________________
Talk More, Pay Less with Net2Phone Direct(R), up to 1500 minutes free! 
http://www.net2phone.com/cgi-bin/link.cgi?143 

Re: [Numpy-discussion] static void** libnumarray_API

[Todd M. <jm...@st...>]

On Thu, 2004-07-15 at 09:28, Mathieu Gontier wrote:
> Hello, 
> 
> I am developping FEM bendings from a C++ code to Python with Numarray.
> So, I have the following problem.
> 
> In the distribution file 'libnumarray.h', the variable 'libnumarray_API' is 
> defined as a static variable (because of the symbol NO_IMPORT is not 
> defined).
> 
> Then, I understand that all the examples are implemented in a unique file.
> 
> But, in my project, I must edit header files and source files in order to 
> solve other problems (like cycle includes). So, I have two different source 
> files which use numarray :
> 	- the file containing the 'init' function which call the function 
> 'import_libnumarray()' (which initialize 'libnumarray_API')
> 	- a file containing implementations, more precisely an implementation calling 
> numarray functionnalities: with is 'static' state, this 'libnumarray_API' is 
> NULL...
> 
> I tried to compile NumArray with the symbol 'NO_IMPORT' (see libnumarray.h) in 
> order to have an extern variable. But this symbol doesn't allow to import 
> numarray in the python environment.
> 
> So, does someone have a solution allowing to use NumArray API with 
> header/source files ? 

The good news is that the 1.0 headers, at least, work.

I intended to capture this form of multi-compilation-unit module in the
numpy_compat example... but didn't.  I think there's two "tricks"
missing in the example.  In *a* module of the several modules you're
linking together, do the following:

#define NO_IMPORT 1	   /* This prevents the definition of the static
			      version of the API var.  The extern won't
			      conflict with the real definition below. 			   */

#include "libnumarray.h"

void **libnumarray_API;    /* This defines the missing API var for *all*
				your compilation units */

This variable will be assigned the API pointer by the
import_libnumarray() call.

I fixed the numpy_compat example to demonstrate this in CVS but they
have a Numeric flavor.  The same principles apply to libnumarray.  Note
that for numarray-1.0 you must include/import both the Numeric
compatible and native numarray APIs separately if you use both.

Regards,
Todd

[Numpy-discussion] NA.dot transposing in place

[Curzio B. <cur...@un...>]

Hi all.

I wonder if anyone noticed the following behaviour (new in 1.0) of the 
dot/matrixmultiply functions:

 >>> alpha = NA.arange(10, shape = (10,1))

 >>> beta = NA.arange(10, shape = (10,1))

 >>> NA.dot(alpha, alpha)
array([[285]])

 >>> alpha.shape # here it looks like it's doing the transpose in place
(1, 10)

 >>> NA.dot(beta, alpha)
array([[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
        [ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],
        [ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18],
        [ 0,  3,  6,  9, 12, 15, 18, 21, 24, 27],
        [ 0,  4,  8, 12, 16, 20, 24, 28, 32, 36],
        [ 0,  5, 10, 15, 20, 25, 30, 35, 40, 45],
        [ 0,  6, 12, 18, 24, 30, 36, 42, 48, 54],
        [ 0,  7, 14, 21, 28, 35, 42, 49, 56, 63],
        [ 0,  8, 16, 24, 32, 40, 48, 56, 64, 72],
        [ 0,  9, 18, 27, 36, 45, 54, 63, 72, 81]])

 >>> alpha.shape, beta.shape # but not the second time
((1, 10), (10, 1))

-------------------------------------------------

Can someone explain me what's going on?

thanks, curzio

[Numpy-discussion] static void** libnumarray_API

[Mathieu G. <mat...@ff...>]

Hello, 

I am developping FEM bendings from a C++ code to Python with Numarray.
So, I have the following problem.

In the distribution file 'libnumarray.h', the variable 'libnumarray_API' is 
defined as a static variable (because of the symbol NO_IMPORT is not 
defined).

Then, I understand that all the examples are implemented in a unique file.

But, in my project, I must edit header files and source files in order to 
solve other problems (like cycle includes). So, I have two different source 
files which use numarray :
	- the file containing the 'init' function which call the function 
'import_libnumarray()' (which initialize 'libnumarray_API')
	- a file containing implementations, more precisely an implementation calling 
numarray functionnalities: with is 'static' state, this 'libnumarray_API' is 
NULL...

I tried to compile NumArray with the symbol 'NO_IMPORT' (see libnumarray.h) in 
order to have an extern variable. But this symbol doesn't allow to import 
numarray in the python environment.

So, does someone have a solution allowing to use NumArray API with 
header/source files ? 

Thanks, 
Mathieu Gontier

Re: [Numpy-discussion] sum() and mean() broken?

[Todd M. <jm...@st...>]

numarray-1.0 is known to have problems with Windows-98, etc. (My guess
is any Pre-NT windows).  I haven't seen any problems with Windows XP or
Windows 2000 Pro.  

Which windows variant are you running?  

Does the numarray selftest pass?  It should look something like:

>>> import nuamrray.testall as testall
>>> testall.test()
numarray:                               ((0, 1178), (0, 1178))
numarray.records:                       (0, 48)
numarray.strings:                       (0, 176)
numarray.memmap:                        (0, 82)
numarray.objects:                       (0, 105)
numarray.memorytest:                    (0, 16)
numarray.examples.convolve:             ((0, 20), (0, 20), (0, 20), (0,
20))
numarray.convolve:                      (0, 52)
numarray.fft:                           (0, 75)
numarray.linear_algebra:                ((0, 46), (0, 51))
numarray.image:                         (0, 27)
numarray.nd_image:                      (0, 390)
numarray.random_array:                  (0, 53)
numarray.ma:                            (0, 671)


On Wed, 2004-07-14 at 23:50, Gary Ruben wrote:
> I'm getting tracebacks on even the most basic sum() and mean() calls in numarray 1.0 under Windows. Apologies if this has already been reported.
> Gary
> 
> >>> from numarray import *
> >>> arange(10).sum()
> 
> Traceback (most recent call last):
>   File "<pyshell#4>", line 1, in -toplevel-
>     arange(10).sum()
>   File "C:\APPS\PYTHON23\Lib\site-packages\numarray\numarraycore.py", line 1106, in sum
>     return ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
> error: Int32asInt64: buffer not aligned on 8 byte boundary.
-- 

[Numpy-discussion] sum() and mean() broken?

[Gary R. <ga...@em...>]

I'm getting tracebacks on even the most basic sum() and mean() calls in numarray 1.0 under Windows. Apologies if this has already been reported.
Gary

>>> from numarray import *
>>> arange(10).sum()

Traceback (most recent call last):
  File "<pyshell#4>", line 1, in -toplevel-
    arange(10).sum()
  File "C:\APPS\PYTHON23\Lib\site-packages\numarray\numarraycore.py", line 1106, in sum
    return ufunc.add.reduce(ufunc.add.areduce(self, type=type).flat, type=type)
error: Int32asInt64: buffer not aligned on 8 byte boundary.

-- 
_______________________________________________
Talk More, Pay Less with Net2Phone Direct(R), up to 1500 minutes free! 
http://www.net2phone.com/cgi-bin/link.cgi?143