numpy-discussion — Discussion list for all users of Numerical Python

[Numpy-discussion] ACML and dotblas

[Arnd B. <arn...@we...>]

Hi,

has anyone figured out how to use dotblas
with the AMD Core Math Library (ACML, http://developer.amd.com/acml.aspx)?
Darren Dale  asked this a month ago here
http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2886285
but I could not find any answer.

Many thanx,

Arnd

Re[2]: [Numpy-discussion] Indices of min/max element of an array?

[Alan G I. <ai...@am...>]

On Tue, 6 Dec 2005, Christos Siopis apparently wrote: 
> As a side note, i am wondering if there is a semantic 
> asymmetry in using min() and max() to signify the min/max 
> element in the entire array while argmin() and argmax() 
> signify the min/max element along each axis. 

SciPy arrays function as "expected" in this sense:
>>> import scipy
>>> x=scipy.array([[1,2],[3,4]])
>>> x.max()
4
>>> x.argmax()
3

Note that, as I understand, argmax gives the index from x.flat

Also, the scipy ufuncs max and argmax have the symmetry you 
seek, if I understand correctly.

hth,
Alan Isaac

Re: [Numpy-discussion] Indices of min/max element of an array?

[Travis O. <oli...@ee...>]

Christos Siopis wrote:

>On Tue, Dec 06, 2005 at 12:17:14PM -0500, Perry Greenfield wrote:
>
>  
>
>>Why not:
>>
>>ind = where(arr == arr.max())
>>
>>and use the first element of the index array(s)
>>    
>>
>
>
>Would it be safe to assume that the numarray definition of where() will be the one 
>propagated into scipy.core?
>  
>
Yes, it is already there.

-Travis

Re: [Numpy-discussion] Indices of min/max element of an array?

[Christos S. <chr...@ul...>]

On Tue, Dec 06, 2005 at 12:17:14PM -0500, Perry Greenfield wrote:

> Why not:
> 
> ind = where(arr == arr.max())
> 
> and use the first element of the index array(s)

Thanks, that worked nice! I was stuck with the old Numeric definition of where(). 
Could it perhaps be made more obvious in numarray's where() documentation that it 
can also be called with one parameter only? (I.e., put this information in the 
section header, not mid-way in the description). I'm looking at the April 20, 2005 
documentation release.

Would it be safe to assume that the numarray definition of where() will be the one 
propagated into scipy.core?

As a side note, i am wondering if there is a semantic asymmetry in using min() and 
max() to signify the min/max element in the entire array while argmin() and 
argmax() signify the min/max element along each axis. At the same time, and as far 
as i can tell, there is no min()/max() method to provide the min/max element along 
each axis, and there is no method to do the equivalent of "argmin/max_all()", as 
implemented by where(arr == arr.min/max()). Apologies if this has been discussed 
before.

Thank you,
Christos

[Numpy-discussion] ANN: numarray-1.5.0

[Todd M. <jm...@st...>]

Numarray is an array processing package designed to efficiently
manipulate large multi-dimensional arrays.  Numarray is modelled after
Numeric and features c-code generated from python template scripts,
the capacity to operate directly on arrays in files, arrays of
heterogeneous records, string arrays, and in-place operation on
memory mapped files.

=========================================================================

Release Notes for numarray-1.5.0

I. ENHANCEMENTS

1. Implementation of scipy newcore array interface and __array_struct__
support (supplying and consuming) for numarray.  This should facilitate
better interoperability with scipy newcore and Numeric.  This protocol
is documented here: http://numeric.scipy.org/array_interface.html
but also in numarray's source code in Include/numarray/arrayif.h.

Thanks to everyone who gave feedback on 1.4.1 and particularly to
Francesc Altet for his diligent testing (and doctest) of
Numeric<->numarray data interchange.

II. BUGS FIXED / CLOSED

1365121         Definition of PyObject* operator conflicts with C++
1350954         nummacro.h:27: error: expected type-specifier before ';' tok
1346470         Please document more of what byteswap and byteswapped do
1364215         Cannot combine array and masked array (e.g. via divide)
1363723         my_array = +my_other_array uncovers a bug
1364811         Infinite loop converting empty CharArrays to lists
1364815         Strange behaviour when creating empty Int32 arrays
1340983         Fix __get_array_data__
1346480         byteswapped doesn't match doc string
1346425         Please document if copy "fixes" byte order
1346426         Please document what array does with numarray arrays

See
http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse
for more details.

III. CAUTIONS

This release should be binary compatible with numarray-1.3.x.  New
extensions compiled against 1.5.0 may not work with prior versions of
numarray.

WHERE
-----------

Numarray-1.5.0 windows executable installers, source code, and manual is
here:

http://sourceforge.net/project/showfiles.php?group_id=1369

Numarray is hosted by Source Forge in the same project which hosts
Numeric:

http://sourceforge.net/projects/numpy/

The web page for Numarray information is at:

http://www.stsci.edu/resources/software_hardware/numarray

Trackers for Numarray Bugs, Feature Requests, Support, and Patches are
at the Source Forge project for NumPy at:

http://sourceforge.net/tracker/?group_id=1369

REQUIREMENTS
------------------------------

numarray-1.5.0 requires Python 2.2.3 or greater.


AUTHORS, LICENSE
------------------------------

Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC
Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science
Institute.  We'd like to acknowledge the assitance of Francesc Alted,
Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav
Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant,
Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and
everyone else who has contributed with comments and feedback.

Numarray is made available under a BSD-style License.  See
LICENSE.txt in the source distribution for details.

--
Todd Miller             jm...@st...

Re: [Numpy-discussion] performance of nd_image affine_transform

[Peter V. <ve...@em...>]

On Dec 6, 2005, at 18:24, Sebastian Haase wrote:

> (Peter, I hope you meant to reply to the mailing list...)

Yes, I did :-)

> Hi Peter,
> Does that mean that the performance hit happens "at the lowest  
> level", i.e.
> there is lot's of extra stuff in the in the innermost c loop that  
> iterates
> over the output array ?

Yes, I would say so.

> Or is there just so much setup and cleanup code that
> makes the overall function slow ?

Some of that may be a problem too, for instance, the function does  
malloc and free some temporary data, so that would add if you apply  
the function repeatedly.

> How could I profile C extensions anyway - pointers/hints are  
> appreciated !

Not sure how you would go about that. I guess optimizing would  
require a rewrite for the specific cases that you like to optimize.

>
> Thanks for your nd_image (In case that wasn't clear - I like it a  
> lot !)

Glad to hear that!

>
> Sebastian Haase
>
>
>
> On Tuesday 06 December 2005 01:32, you wrote:
>> Hi Sebastian,
>>
>> The interpolation routines are not really optimized very well. The
>> underlying code is quite generic for spline order, data type, and
>> arbritatry array dimension and data layout. Your fortran code is
>> probably much more optimized. It would be nice to have more optimized
>> code in nd_image, but I do not really have the resources to do that.
>>
>>> Hi,
>>> Thanks Peter for checking on the problem I reported in my last
>>> posting...
>>>
>>> Now I'm looking into using nd_image.affine_transform inplace of a
>>> fortran
>>> routine that I have been using to do this.
>>> a) I need to run this on Windows - where I don't have Fortran
>>> b) My Fortran routine does only do linear interpolation and I like
>>> the idea of
>>> experimenting with splines.
>>>
>>> A and B would of course be good reasons to use nd_image,
>>> BUT c)
>>> for a 512x512 float32 image my fortran takes about 14ms
>>> nd.affine_transform with given output array, prefilter=0 and order=1
>>> takes about 132ms !
>>> With  prefilter=1 it takes 138ms; with prefilter=1 and order=3 it
>>> takes
>>> 279ms !!  (order=2,prefilter=1  takes 226ms ; order=3,prefilter=0
>>> 222ms)
>>> All these are averaged over 10 runs on Linux (P4 2.8GHz)
>>>
>>> Why is nd_image 10x slower ? (spline order 1 does the same as linear
>>> (non-spline) interpolation, right ?)  I would call this many (100,
>>> 1000 ?)
>>> times inside a simplex algorithm which takes already many seconds
>>> to complete
>>> using the Fortran routine...
>>>
>>> Thanks,
>>> Sebastian Haase
>>>
>>> On Monday 05 December 2005 14:31, Peter Verveer wrote:
>>>> Works for me with the latest CVS version.
>>>>
>>>> On 5 Dec, 2005, at 21:13, Sebastian Haase wrote:
>>>>> Hi,
>>>>> When I call nd_image.rotate with reshape=False I always get
>>>>> "output shape not correct"
>>>>>
>>>>>>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1],
>>>>>>>> order=1,
>>>>>
>>>>> mode="constant", cval=0.0, prefilter=0)
>>>>> Traceback (most recent call last):
>>>>>   File "<input>", line 1, in ?
>>>>>   File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py",
>>>>> line 351, in
>>>>> rotate
>>>>>     output, order, mode, cval, prefilter)
>>>>>   File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py",
>>>>> line 205, in
>>>>> affine_transform
>>>>>     output_type)
>>>>>   File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line
>>>>> 73, in
>>>>> _get_output
>>>>>     raise RuntimeError, "output shape not correct"
>>>>> RuntimeError: output shape not correct
>>>>>
>>>>> I tracked the problem down to  "inputShape != outputShape" one
>>>>> being a tuple
>>>>> the output shape being a list.
>>>>> (Pdb) p shape
>>>>> [128, 528]
>>>>> (Pdb) p output.shape
>>>>> (128, 528)
>>>>> (Pdb) p shape != output.shape
>>>>> 1
>>>>> (Pdb) p shape , output.shape
>>>>> ([128, 528], (128, 528))
>>>>> (Pdb)
>>>>>
>>>>> I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri
>>>>> Apr 22
>>>>> 20:35:27 2005//THEAD)
>>>>> but I took a look at the current CVS and it seems to still be a
>>>>> problem
>>>>>
>>>>> Looks like I'm the only one who doesn't want the reshape ;-)
>>>>>
>>>>> Thanks,
>>>>> Sebastian Haase
>>>>>
>>>>>
>>>>> -------------------------------------------------------
>>>>> This SF.net email is sponsored by: Splunk Inc. Do you grep through
>>>>> log files
>>>>> for problems?  Stop!  Download the new AJAX search engine that  
>>>>> makes
>>>>> searching your log files as easy as surfing the  web.  DOWNLOAD
>>>>> SPLUNK!
>>>>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
>>>>> _______________________________________________
>>>>> Numpy-discussion mailing list
>>>>> Num...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>>
>>>> -------------------------------------------------------
>>>> This SF.net email is sponsored by: Splunk Inc. Do you grep through
>>>> log
>>>> files for problems?  Stop!  Download the new AJAX search engine
>>>> that makes
>>>> searching your log files as easy as surfing the  web.  DOWNLOAD
>>>> SPLUNK!
>>>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
>>>> _______________________________________________
>>>> Numpy-discussion mailing list
>>>> Num...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>
>>> -------------------------------------------------------
>>> This SF.net email is sponsored by: Splunk Inc. Do you grep through
>>> log files
>>> for problems?  Stop!  Download the new AJAX search engine that makes
>>> searching your log files as easy as surfing the  web.  DOWNLOAD
>>> SPLUNK!
>>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Num...@li...
>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through  
> log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD  
> SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] performance of nd_image affine_transform

[Sebastian H. <ha...@ms...>]

(Peter, I hope you meant to reply to the mailing list...)
Hi Peter,
Does that mean that the performance hit happens "at the lowest level", i.e. 
there is lot's of extra stuff in the in the innermost c loop that iterates 
over the output array ?  Or is there just so much setup and cleanup code that 
makes the overall function slow ?
How could I profile C extensions anyway - pointers/hints are appreciated !

Thanks for your nd_image (In case that wasn't clear - I like it a lot !)

Sebastian Haase



On Tuesday 06 December 2005 01:32, you wrote:
> Hi Sebastian,
>
> The interpolation routines are not really optimized very well. The
> underlying code is quite generic for spline order, data type, and
> arbritatry array dimension and data layout. Your fortran code is
> probably much more optimized. It would be nice to have more optimized
> code in nd_image, but I do not really have the resources to do that.
>
> > Hi,
> > Thanks Peter for checking on the problem I reported in my last
> > posting...
> >
> > Now I'm looking into using nd_image.affine_transform inplace of a
> > fortran
> > routine that I have been using to do this.
> > a) I need to run this on Windows - where I don't have Fortran
> > b) My Fortran routine does only do linear interpolation and I like
> > the idea of
> > experimenting with splines.
> >
> > A and B would of course be good reasons to use nd_image,
> > BUT c)
> > for a 512x512 float32 image my fortran takes about 14ms
> > nd.affine_transform with given output array, prefilter=0 and order=1
> > takes about 132ms !
> > With  prefilter=1 it takes 138ms; with prefilter=1 and order=3 it
> > takes
> > 279ms !!  (order=2,prefilter=1  takes 226ms ; order=3,prefilter=0
> > 222ms)
> > All these are averaged over 10 runs on Linux (P4 2.8GHz)
> >
> > Why is nd_image 10x slower ? (spline order 1 does the same as linear
> > (non-spline) interpolation, right ?)  I would call this many (100,
> > 1000 ?)
> > times inside a simplex algorithm which takes already many seconds
> > to complete
> > using the Fortran routine...
> >
> > Thanks,
> > Sebastian Haase
> >
> > On Monday 05 December 2005 14:31, Peter Verveer wrote:
> >> Works for me with the latest CVS version.
> >>
> >> On 5 Dec, 2005, at 21:13, Sebastian Haase wrote:
> >>> Hi,
> >>> When I call nd_image.rotate with reshape=False I always get
> >>> "output shape not correct"
> >>>
> >>>>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1],
> >>>>>> order=1,
> >>>
> >>> mode="constant", cval=0.0, prefilter=0)
> >>> Traceback (most recent call last):
> >>>   File "<input>", line 1, in ?
> >>>   File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py",
> >>> line 351, in
> >>> rotate
> >>>     output, order, mode, cval, prefilter)
> >>>   File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py",
> >>> line 205, in
> >>> affine_transform
> >>>     output_type)
> >>>   File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line
> >>> 73, in
> >>> _get_output
> >>>     raise RuntimeError, "output shape not correct"
> >>> RuntimeError: output shape not correct
> >>>
> >>> I tracked the problem down to  "inputShape != outputShape" one
> >>> being a tuple
> >>> the output shape being a list.
> >>> (Pdb) p shape
> >>> [128, 528]
> >>> (Pdb) p output.shape
> >>> (128, 528)
> >>> (Pdb) p shape != output.shape
> >>> 1
> >>> (Pdb) p shape , output.shape
> >>> ([128, 528], (128, 528))
> >>> (Pdb)
> >>>
> >>> I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri
> >>> Apr 22
> >>> 20:35:27 2005//THEAD)
> >>> but I took a look at the current CVS and it seems to still be a
> >>> problem
> >>>
> >>> Looks like I'm the only one who doesn't want the reshape ;-)
> >>>
> >>> Thanks,
> >>> Sebastian Haase
> >>>
> >>>
> >>> -------------------------------------------------------
> >>> This SF.net email is sponsored by: Splunk Inc. Do you grep through
> >>> log files
> >>> for problems?  Stop!  Download the new AJAX search engine that makes
> >>> searching your log files as easy as surfing the  web.  DOWNLOAD
> >>> SPLUNK!
> >>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> >>> _______________________________________________
> >>> Numpy-discussion mailing list
> >>> Num...@li...
> >>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> >>
> >> -------------------------------------------------------
> >> This SF.net email is sponsored by: Splunk Inc. Do you grep through
> >> log
> >> files for problems?  Stop!  Download the new AJAX search engine
> >> that makes
> >> searching your log files as easy as surfing the  web.  DOWNLOAD
> >> SPLUNK!
> >> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> >> _______________________________________________
> >> Numpy-discussion mailing list
> >> Num...@li...
> >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc. Do you grep through
> > log files
> > for problems?  Stop!  Download the new AJAX search engine that makes
> > searching your log files as easy as surfing the  web.  DOWNLOAD
> > SPLUNK!
> > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> > _______________________________________________
> > Numpy-discussion mailing list
> > Num...@li...
> > https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Indices of min/max element of an array?

[Perry G. <pe...@st...>]

Why not:

ind = where(arr == arr.max())

and use the first element of the index array(s)

Perry Greenfield

On Dec 6, 2005, at 12:06 PM, Christos Siopis wrote:

> Hi all,
>
> I was wondering if i'm missing a numpy/numarray function that would 
> return the
> indices of the minimum/maximum element of an array. Argmin/max can 
> only do this
> one axis at a time. Alternatively, one can find the index for the 
> flattened array
> and then play with modulo arithmetic, which gets quickly annoying for 
> > 2
> dimensions. Since this is a rather frequent operation, i would think 
> there's
> already a function for it?...
>
> Thanks,
> Christos
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion

[Numpy-discussion] Indices of min/max element of an array?

[Christos S. <chr...@ul...>]

Hi all,

I was wondering if i'm missing a numpy/numarray function that would return the 
indices of the minimum/maximum element of an array. Argmin/max can only do this 
one axis at a time. Alternatively, one can find the index for the flattened array 
and then play with modulo arithmetic, which gets quickly annoying for > 2 
dimensions. Since this is a rather frequent operation, i would think there's 
already a function for it?...

Thanks,
Christos

[Numpy-discussion] Data type change completed

[Colin J. W. <cj...@sy...>]

Travis Oliphant wrote:

> Colin J. Williams wrote:
>
>> Travis Oliphant wrote:
>>
>>>
>>> I've committed the data-type change discussed at the end of last 
>>> week to the SVN repository.  Now the concept of a data type for an 
>>> array has been replaced with a "data-descriptor".  This 
>>> data-descriptor is flexible enough to handle an arbitrary record 
>>> specification with fields that include records and arrays or arrays 
>>> of records.  While nesting may not be the best data-layout for a new 
>>> design, when memory-mapping an arbitrary fixed-record-length file, 
>>> this capability allows you to handle even the most obsure record file.
>>>
>> Does this mean that the dtype parameter is changed?  obscure??
>
>
> No, it's not changed.  The dtype parameter is still used and it is 
> still called the same thing.   It's just that what constitutes a 
> data-type has changed significantly.
>
> For example now tuples and dictionaries can be used to describe a 
> data-type.  These definitions are recursive so that whenever data-type 
> is used it means anything that can be interpreted as a data-type.  And 
> I really mean data-descriptor, but data-type is in such common usage 
> that I still use it.

This would appear to be a good step forward but with all of the 
immutable types (int8, FloatType, TupleType, etc.) the data is stored in 
the ArrayType instance (array_data?) whereas, with a dictionary, it 
would appear to be necessary to store the items outside the array.  Is 
that desirable?

Even the tuple can have its content modified, as the example below shows:

 >>> a= []
 >>> b= (a, [2, 3])
 >>> b[0]
[]
 >>> b[0]=99
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: object does not support item 
assignment                             <<<    GOOD
 >>> b[1][0]
2
 >>> b[1][0]=99
 >>> b
([], [99, 
3])                                                                                          
<<< HERE WE CHANGE THE VALUE OF THE
                                                                                                                   
TUPLE
 >>>

>
> Tuple:
> ========
> (fixed-size-data-type, shape)
> (generic-size-data-type, itemsize)
> (base-type-data-type, new-type-data-type)
>
> Examples:
>
> dtype=(int32, (5,5))   ---  a 5x5 array of int32 is the description of 
> this item.
> dtype=(str, 10) --- a length-10 string

So dtype now contains both the data type of each element and the shape 
of the array?  This seems a significant change from numarray or Numeric.

> dtype=(int16, {'real':(int8,0),'imag':(int8,4)}  --- a descriptor that 
> acts
>                                                                               
> like an int16 array mathematically
>                                                                               
> (in ufuncs) but has real and imag
>                                                                        
>      
> fields.                                                                              
>
>
This adds complexity, is there a compensating benefit?  Do all of the 
complex operations apply?

>
> Dictionary (defaults to a dtypechar == 'V')
> ==========

Why no clean things up by dropping typechar?  These seemed to be one of 
the warts in numarray, only carried forward for
compatibility reasons.  Could the compatibility objectives of the 
project not be achieved, outside the ArrayType object, with a wrapper of 
some sort?

> format1:
>
> {"names": list-of-field-names,
>  "formats":  list of data-types,
>
> <optionally>
>  "offsets" : list of  start-of-the-field
>  "titles" : extra field names
> }
>
Couldn't the use of records avoid the cumbersome use of keys?

> format2 (and how it's stored internally)
>
> {key1 : (data-type1, offset1 [, title1]),
>  key2 : (data-type2, offset2 [, title2]),
>   ...
>  keyn : (data-typen, offsetn [, titlen])
> }
>
This is cleaner, but couldn't this inormation be contained within the 
Record instance?

>
> Other objects not already covered:
> =====================
> ????
> Right now, it just passes the tp_dict of the typeobject to the 
> dictionary-conversion routine.
> I'm open for ideas here and will probably have better ideas once the 
> actual record data-type (not data-descriptor but actual subclass of 
> the scipy.void data type) looks like.
>
> All of these can be used as the dtype parameter wherever it is taken 
> (of course you can't
> always do something useful with every data-descriptor).
> When an ndarray has an associated type descriptor with fields (that's 
> where the field information is
> stored),  then those fields can be accessed using string or unicode 
> keys to the getitem call.
>
I've used ArrayType in place of ndarray (or maybe it should have been 
ndbigarray?) above as it appear to be more descriptive and fits with the 
Python convention on class naming.

> Thus, you can do something like this:
>
> >>> a = ones((4,3), dtype=(int16, {'real':(int8, 0), 'imag':(int8, 1)}))
> >>> a['imag'] = 2
> >>> a['real'] = 1
> >>> a.tostring()
> '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' 
>
>
Or, one could have something like:
class SmallComplex(Record):
..''' This class typically has no instances in user code. '''
..real= (int8, )
..imag= (int8)
..def __init__(self):
....
..def __new__(self):
....

 >>> a = ones((4,3), dtype= SmallComplex)
 >>> a.imag = 2
 >>> a.real = 1
 >>> a.tostring()
'\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' 


>
> Note that there are now three distinct but interacting Python objects:
>
> 1) the N-dimensional array of a fixed itemsize.
> 2) a Python object representing one element of the array.
> 3) the data-descriptor object describing the data-type of the array.

This looks cleaner.  Perehaps 2) and 3) could be phrased a little 
differently:

2) a Python object which is one element of the array.
3) the data-descriptor object describing the data-type of the array 
element.

>
> These three things were always there under the covers (the 
> PyArray_Descr* has been there since Numeric), and the standard Python 
> types were always filling in for number 2.  Now we are just being more 
> explicit about it.
>
> Now, all three things are present and accounted for.  I'm really quite 
> happy with the resulting infrastructure. I think it will allow some 
> really neat possibilities.
>
> I'm thinking the record array subclass will allow attribute-based 
> look-up and register a nice record type for the actual "element" in of 
> the record array.
>
This is good but the major structure is the array which can have 
elements of various types such as ComplexType, NoneType, int8 or a 
variety of user defined immutable records.

Colin W.

PS  My Record sketch above needs a lot more thinking through

>
> -Travis
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log 
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>

Re: ***[Possible UCE]*** Re: [Numpy-discussion] Data type change completed

[Travis O. <oli...@ie...>]

Colin J. Williams wrote:

> Travis Oliphant wrote:
>
>>
>> I've committed the data-type change discussed at the end of last week 
>> to the SVN repository.  Now the concept of a data type for an array 
>> has been replaced with a "data-descriptor".  This data-descriptor is 
>> flexible enough to handle an arbitrary record specification with 
>> fields that include records and arrays or arrays of records.  While 
>> nesting may not be the best data-layout for a new design, when 
>> memory-mapping an arbitrary fixed-record-length file, this capability 
>> allows you to handle even the most obsure record file.
>>
> Does this mean that the dtype parameter is changed?  obscure??

No, it's not changed.  The dtype parameter is still used and it is still 
called the same thing.   It's just that what constitutes a data-type has 
changed significantly.

For example now tuples and dictionaries can be used to describe a 
data-type.  These definitions are recursive so that whenever data-type 
is used it means anything that can be interpreted as a data-type.  And I 
really mean data-descriptor, but data-type is in such common usage that 
I still use it.

Tuple:
========
(fixed-size-data-type, shape)
(generic-size-data-type, itemsize)
(base-type-data-type, new-type-data-type)

Examples:

dtype=(int32, (5,5))   ---  a 5x5 array of int32 is the description of 
this item.
dtype=(str, 10) --- a length-10 string
dtype=(int16, {'real':(int8,0),'imag':(int8,4)}  --- a descriptor that acts
                                                                               
like an int16 array mathematically
                                                                               
(in ufuncs) but has real and imag
                                                                        
      
fields.                                                                              



Dictionary (defaults to a dtypechar == 'V')
==========
format1:

{"names": list-of-field-names,
  "formats":  list of data-types,

<optionally>
  "offsets" : list of  start-of-the-field
  "titles" : extra field names
}

format2 (and how it's stored internally)

{key1 : (data-type1, offset1 [, title1]),
  key2 : (data-type2, offset2 [, title2]),
   ...
  keyn : (data-typen, offsetn [, titlen])
}


Other objects not already covered:
=====================
????
Right now, it just passes the tp_dict of the typeobject to the 
dictionary-conversion routine.
I'm open for ideas here and will probably have better ideas once the 
actual record data-type (not data-descriptor but actual subclass of the 
scipy.void data type) looks like.

All of these can be used as the dtype parameter wherever it is taken (of 
course you can't
always do something useful with every data-descriptor). 

When an ndarray has an associated type descriptor with fields (that's 
where the field information is
stored),  then those fields can be accessed using string or unicode keys 
to the getitem call.

Thus, you can do something like this:

 >>> a = ones((4,3), dtype=(int16, {'real':(int8, 0), 'imag':(int8, 1)}))
 >>> a['imag'] = 2
 >>> a['real'] = 1
 >>> a.tostring()
'\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02'


Note that there are now three distinct but interacting Python objects:

1) the N-dimensional array of a fixed itemsize.
2) a Python object representing one element of the array.
3) the data-descriptor object describing the data-type of the array.

These three things were always there under the covers (the 
PyArray_Descr* has been there since Numeric), and the standard Python 
types were always filling in for number 2.  Now we are just being more 
explicit about it.

Now, all three things are present and accounted for.  I'm really quite 
happy with the resulting infrastructure. I think it will allow some 
really neat possibilities.

I'm thinking the record array subclass will allow attribute-based 
look-up and register a nice record type for the actual "element" in of 
the record array.


-Travis

Re: [Numpy-discussion] Data type change completed

[Colin J. W. <cj...@sy...>]

Travis Oliphant wrote:

>
> I've committed the data-type change discussed at the end of last week 
> to the SVN repository.  Now the concept of a data type for an array 
> has been replaced with a "data-descriptor".  This data-descriptor is 
> flexible enough to handle an arbitrary record specification with 
> fields that include records and arrays or arrays of records.  While 
> nesting may not be the best data-layout for a new design, when 
> memory-mapping an arbitrary fixed-record-length file, this capability 
> allows you to handle even the most obsure record file.
>
Does this mean that the dtype parameter is changed?  obscure??

> While the basic core tests pass for me, there may be lurking problems 
> and so testing of the SVN trunk of scipy core will be appreciated.  
> I've bumped up the version number because the C-API has changed (a few 
> new functions and some functions becoming macros).
> I'd like to make a release of the new version by the end of the week 
> (as soon as Chris Hanley at STSCI and I get records.py working 
> better), so please test.
>
> Recently some intel c-compiler tests were failing on a 64-bit 
> platform.  It would be nice to figure out why that is happening as 
> well, but I will probably not have time for that this week.
>
> Thanks,
>
> -Travis

[Numpy-discussion] performance of nd_image affine_transform

[Sebastian H. <ha...@ms...>]

Hi,
Thanks Peter for checking on the problem I reported in my last posting...

Now I'm looking into using nd_image.affine_transform inplace of a fortran 
routine that I have been using to do this.
a) I need to run this on Windows - where I don't have Fortran 
b) My Fortran routine does only do linear interpolation and I like the idea of 
experimenting with splines.

A and B would of course be good reasons to use nd_image,
BUT c)  
for a 512x512 float32 image my fortran takes about 14ms
nd.affine_transform with given output array, prefilter=0 and order=1 
takes about 132ms !
With  prefilter=1 it takes 138ms; with prefilter=1 and order=3 it takes 
279ms !!  (order=2,prefilter=1  takes 226ms ; order=3,prefilter=0 222ms)
All these are averaged over 10 runs on Linux (P4 2.8GHz)

Why is nd_image 10x slower ? (spline order 1 does the same as linear 
(non-spline) interpolation, right ?)  I would call this many (100, 1000 ?) 
times inside a simplex algorithm which takes already many seconds to complete 
using the Fortran routine...

Thanks,
Sebastian Haase




On Monday 05 December 2005 14:31, Peter Verveer wrote:
> Works for me with the latest CVS version.
>













> On 5 Dec, 2005, at 21:13, Sebastian Haase wrote:
> > Hi,
> > When I call nd_image.rotate with reshape=False I always get
> > "output shape not correct"
> >
> >>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1],
> >>>> order=1,
> >
> > mode="constant", cval=0.0, prefilter=0)
> > Traceback (most recent call last):
> >   File "<input>", line 1, in ?
> >   File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py",
> > line 351, in
> > rotate
> >     output, order, mode, cval, prefilter)
> >   File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py",
> > line 205, in
> > affine_transform
> >     output_type)
> >   File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line
> > 73, in
> > _get_output
> >     raise RuntimeError, "output shape not correct"
> > RuntimeError: output shape not correct
> >
> > I tracked the problem down to  "inputShape != outputShape" one
> > being a tuple
> > the output shape being a list.
> > (Pdb) p shape
> > [128, 528]
> > (Pdb) p output.shape
> > (128, 528)
> > (Pdb) p shape != output.shape
> > 1
> > (Pdb) p shape , output.shape
> > ([128, 528], (128, 528))
> > (Pdb)
> >
> > I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri
> > Apr 22
> > 20:35:27 2005//THEAD)
> > but I took a look at the current CVS and it seems to still be a
> > problem
> >
> > Looks like I'm the only one who doesn't want the reshape ;-)
> >
> > Thanks,
> > Sebastian Haase
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc. Do you grep through
> > log files
> > for problems?  Stop!  Download the new AJAX search engine that makes
> > searching your log files as easy as surfing the  web.  DOWNLOAD
> > SPLUNK!
> > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> > _______________________________________________
> > Numpy-discussion mailing list
> > Num...@li...
> > https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log
> files for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion

[Numpy-discussion] numarray: how to make sinc(x) function - "divide by zero" !

[Sebastian H. <ha...@ms...>]

Hi,
The best I could come up with was:

def sinc(r):
    na.Error.pushMode(all="ignore")
    a = na.where(r, na.divide(na.sin(r),r), 1)
    na.Error.popMode()
    return a

but I still seem to get  a warning...
>>> F.sinc(0)
Warning: Encountered invalid numeric result(s)  in divide
1.0

Thanks,
Sebastian Haase

Re: [Numpy-discussion] BUG in nd_image

[Peter V. <ve...@em...>]

Works for me with the latest CVS version.

On 5 Dec, 2005, at 21:13, Sebastian Haase wrote:

> Hi,
> When I call nd_image.rotate with reshape=False I always get
> "output shape not correct"
>>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1],  
>>>> order=1,
> mode="constant", cval=0.0, prefilter=0)
> Traceback (most recent call last):
>   File "<input>", line 1, in ?
>   File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py",  
> line 351, in
> rotate
>     output, order, mode, cval, prefilter)
>   File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py",  
> line 205, in
> affine_transform
>     output_type)
>   File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line  
> 73, in
> _get_output
>     raise RuntimeError, "output shape not correct"
> RuntimeError: output shape not correct
>
> I tracked the problem down to  "inputShape != outputShape" one  
> being a tuple
> the output shape being a list.
> (Pdb) p shape
> [128, 528]
> (Pdb) p output.shape
> (128, 528)
> (Pdb) p shape != output.shape
> 1
> (Pdb) p shape , output.shape
> ([128, 528], (128, 528))
> (Pdb)
>
> I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri  
> Apr 22
> 20:35:27 2005//THEAD)
> but I took a look at the current CVS and it seems to still be a  
> problem
>
> Looks like I'm the only one who doesn't want the reshape ;-)
>
> Thanks,
> Sebastian Haase
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through  
> log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD  
> SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion

[Numpy-discussion] Data type change completed

[Travis O. <oli...@ie...>]

I've committed the data-type change discussed at the end of last week to 
the SVN repository.  Now the concept of a data type for an array has 
been replaced with a "data-descriptor".  This data-descriptor is 
flexible enough to handle an arbitrary record specification with fields 
that include records and arrays or arrays of records.  While nesting may 
not be the best data-layout for a new design, when memory-mapping an 
arbitrary fixed-record-length file, this capability allows you to handle 
even the most obsure record file.

While the basic core tests pass for me, there may be lurking problems 
and so testing of the SVN trunk of scipy core will be appreciated.  I've 
bumped up the version number because the C-API has changed (a few new 
functions and some functions becoming macros). 

I'd like to make a release of the new version by the end of the week (as 
soon as Chris Hanley at STSCI and I get records.py working better), so 
please test.

Recently some intel c-compiler tests were failing on a 64-bit platform.  
It would be nice to figure out why that is happening as well, but I will 
probably not have time for that this week.

Thanks,

-Travis

[Numpy-discussion] BUG in nd_image

[Sebastian H. <ha...@ms...>]

Hi,
When I call nd_image.rotate with reshape=False I always get
"output shape not correct"
>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], order=1, 
mode="constant", cval=0.0, prefilter=0)
Traceback (most recent call last):
  File "<input>", line 1, in ?
  File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", line 351, in 
rotate
    output, order, mode, cval, prefilter)
  File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", line 205, in 
affine_transform
    output_type)
  File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line 73, in 
_get_output
    raise RuntimeError, "output shape not correct"
RuntimeError: output shape not correct

I tracked the problem down to  "inputShape != outputShape" one being a tuple 
the output shape being a list.
(Pdb) p shape
[128, 528]
(Pdb) p output.shape
(128, 528)
(Pdb) p shape != output.shape
1
(Pdb) p shape , output.shape
([128, 528], (128, 528))
(Pdb) 

I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri Apr 22 
20:35:27 2005//THEAD)
but I took a look at the current CVS and it seems to still be a problem

Looks like I'm the only one who doesn't want the reshape ;-)

Thanks,
Sebastian Haase

[Numpy-discussion] Re: scipy cygwin compilation error

[Vincent R. <vtr...@ya...>]

Jordan and others,

I have the same problem installing scipy in cygwin.  Has anyone found a solution?

Specifically "python setup.py install" fails with messages like:
"scipy/base/src/ufuncobject.c:475: undefined reference"

Vincent Randal
Longmont, CO

Re: [Numpy-discussion] Numeric3.0 - Currently available under scipy/base as version 0.6.1 (Windows)

[Colin J. W. <cj...@sy...>]

Travis Oliphant wrote:

> Colin J. Williams wrote:
>
>>
>> I have a package based on subclassing numarray, which is working 
>> satisfactorily, and am looking at the possibility of transferring the 
>> package to work under the revised Numeric.
>>
>> My feeling is that the transfer is probably feasible but that it 
>> would be premature to work on it at this time.
>
>
> That's unfortunate.  The more feedback we get early on about 
> subclassing, the better.

The publication of  Python style defs, together with doc strings
describing the acceptable arguments for the various __new__
methods would help here.  I don't find info on ndarray.__new__.
Incidentally, ArrayType seems a better, more meaningful name for the
class/type.

>>
>> One of the problems is the cluttered namespace, through the use of 
>> "from X import *".  This is a style which is deprecated, see page 401 
>> of Alex Martelli's /Python in a Nutshell/.
>
>
> You will have to be more specific about what you think is wrong.  What 
> namespace is "cluttered" exactly.   Just because use is made of from X 
> import * in one module does not mean everything is "cluttered".  SciPy 
> Core makes use of the __all__ variables to control what gets imported 
> and usually only specific functions are imported as necessary. 
>
>> Another problem, at this stage, is that many doc strings are missing 
>> and that some which exist are a little cryptic.
>
>
> I would submit there are more docstrings then Numeric had.   Jump in 
> and help.   The water is fine.
>
>>
>> I would like to take another look when the next win32 binaries are 
>> available.
>
>
> There has been much improvement since the last beta.  I'm trying to 
> track down some remaining memory leaks before releasing new windows 
> binaries.    The SVN code is always available for check out and it is 
> quite easy to build.   We could always use more build testers to make 
> sure building is going as smoothly as we believe it is.
>
I now have 0.6.1, thanks.  Now, some documentation on ndarray/ArrayType
would be helpful, with the fact that __new__ alone is used instead of
the usual __new__ + __init__.  Among the parameters is buffer, it seems
that this accepts buffers or strings, anything else? Iinformation on
call backs (from ufuncs?) would help.

For many cases, it would be better to create a Sub instance with
Sub(aList) but, if the subclass must respond to callbacks then this may
be inhibited.

>>
>> Some further thoughts on the present state of Numeric3.0 are 
>> available here <http://www3.sympatico.ca/cjw/scipy1/>.
>
>
>
> Most of your comments have more to do with differences between builtin 
> types and Python classes than anything about scipy.    The type-class 
> chasm has shrunken of late, but there are still remaining issues.     
> These are Python issues that I believe are of little concern.
>
> I will comment on your issues that are not related to the above comment:
>
>
>      Use of package __init__.py to create namespace.
>
> If the epydoc and pydoc tools are not respecting the __init__.py code 
> then I would say they are broken.    Using the __init__.py this way 
> frees the user from having to understand every little detail of the 
> package structure (which could also change as better organization is 
> obtained in the future).
>
>
>      Use of the from X import Y style
>
> Please give more support here.  Just because one Python user advocates 
> against it is not sufficient argument.  There is an argument to be 
> made for avoiding attribute lookups by importing the names directly in 
> this fashion.
>
>
>      *Methods vs functions*
>
> I agree that methods and functions are somewhat redundant.  However, 
> the functions are still needed both to support lists and especially 
> for back-wards compatibility. Your example using take is odd (perhaps 
> it's a bug in an old release).  I get consistent behavior.   One 
> problem is you define a 1-d array in one case and a 2-d array in 
> another.  Of course the answers will be different.  
> One difference is that the default axis argument is different.  The 
> old functions have exactly the same default axis argument as they 
> always did, while the methods have a default of None for the axis 
> (which means treat the array as flat).
>
>
>      Lack of basic information in the doc strings
>
> Your examples are all class constructors.  First of all, there is no 
> special __init__  method for the ndarray because __new__ takes care of 
> it.    Second of all, the new method does need better documentation.  
> I'm not sure where to put it, though.   The array_new function is 
> placed in the TypeObject of the array type.   The __new__ attribute is 
> pasted on by PyTypeReady.  I'm not sure how to give a docstring as well.
> I suspect the proper thing to do is place the information in the 
> docstring for the Type Object.
>
>
> -Travis
>
I was wrong in my earlier comment about take but there are some problems
as, I hope, the script below illustrates.

Colin W.

# tTake.py

'''
In those cases where there is a duplication, it is suggested that only
the method is needed unless the function handles a wider range of data
than a method.

The function take does more than the method:

>>> S.ndarray((1,2)).take([0])

array([[ 1.05176460e-305, 5.63390393e-311]])

>>> S.take([[1, 2]], [0])

array([[1, 2]])

>>> S.take.__doc__

>>>

'''

import numarray.numarraycore as N
import numarray.numerictypes as NT
import scipy as S

a= S.ndarray((2, ), dtype= S.int8, buffer= '12')
print 'a:', `a`
b= S.arange(12, dtype= S.int8).reshape(3, 4)
print 'b:', `b`
print 'b.take(((0, 0), (2, 2)), 1):', `b.take(((0, 0), (2, 2)), 1)`
c= N.arange(12, dtype=NT.Int8, shape=(3, 4))    # this is a bit easier
to use
print 'c:', `c`
print 'c.take(((0, 0), (2, 2)), 1):', `c.take(((0, 0), (2, 2)), 1)`  #
not quite the same as above
print 'b-c:', `b-c`                             # this needs to raise an
exception

[Numpy-discussion] py2exe compliance

[Antony K. <ant...@ya...>]

 Hi!
  
  I have used a trick to make py2exe recognize all of numarray's modules,  which I suggest be incorporated into the numarray CVS. It can be a real  pain to solve this problem the first time you're py2exe-ing a script  using numarray and find out that it simply doesn't work. I just added  this to numarray's __init__.py:
  
  # this is needed for py2exe to be able to include numarray
  def _give_py2exe_hints():
      # pyd's
      import numarray.libnumarray
      import numarray.libnumeric
      import numarray.memory
      import numarray._bytes
      import numarray._chararray
      import numarray._conv
      import numarray._converter
      import numarray._ndarray
      import numarray._numarray
      import numarray._objectarray
      import numarray._operator
      import numarray._sort
      import numarray._ufunc
      import numarray._ufuncBool
      import numarray._ufuncComplex32
      import numarray._ufuncComplex64
      import numarray._ufuncFloat32
      import numarray._ufuncFloat64
      import numarray._ufuncInt16
      import numarray._ufuncInt32
      import numarray._ufuncInt64
      import numarray._ufuncInt8
      import numarray._ufuncUInt16
      import numarray._ufuncUInt32
      import numarray._ufuncUInt8
      # py's
      import numarray.arrayprint
      import numarray.array_persist
      import numarray.dotblas
      import numarray.generic
      import numarray.ieeespecial
      import numarray.memmap
      import numarray.memorytest
      import numarray.numarrayall
      import numarray.numarraycore
      import numarray.numarrayext
      import numarray.numeric
      import numarray.numerictypes
      import numarray.numinclude
      import numarray.numtest
      import numarray.objects
      import numarray.readonly
      import numarray.records
      import numarray.safethread
      import numarray.strings
      import numarray.teacup
      import numarray.testall
      import numarray.testdata
      import numarray.typeconv
      import numarray.ufunc
      import numarray._ufuncall
      # subpackages
      import numarray.convolve
      import numarray.convolve.lineshape
      import numarray.fft
      import numarray.image
      import numarray.linear_algebra
      import numarray.ma
      import numarray.matrix
      import numarray.mlab
      import numarray.nd_image
      import numarray.random_array
  
  

		
---------------------------------
 Yahoo! Personals
 Single? There's someone we'd like you to meet.
 Lots of someones, actually. Yahoo! Personals

Re: [Numpy-discussion] Records in scipy core

[Stephen W. <go...@co...>]

Colin J. Williams wrote:
> Stephen Waterbury wrote:
>> I'm doing that in an application I'm developing.
>> My objects have an attribute called '_schema' that is an instance
>> of Zope InterfaceClass.  An object (read "record" ;) is assigned a 
>> _schema
>> when it is instantiated, and all information about its attributes (a.k.a.
>> "fields") is contained in the _schema's Properties (my 'Property' 
>> subtypes
>> the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes
>> like 'domain', 'range', 'id', 'name', etc. -- which could easily be
>> extended to include things like 'title', but I use another mechanism
>> for display characteristics, called 'DisplayMap', which can be used to
>> specify the order in which you want the object's properties to appear
>> in a grid, what you want their "display names" to be, etc. ... which are
>> also customizable by the end-user.
> 
> This is goes further than my suggestion.  For arrays, it seems to me 
> that an additional pointer to _schema is not needed as there is a 
> pointer to the data type and the data type can contain the meta data

In effect, the _schema *is* the data type in my scenario.
It contains all information necessary to describe the type
of the object and references to the types of all the
object's attributes (which are called "fields" in record
parlance, "Properties" in the world of ontologies, and
"Attributes" in Zope Interfaces and UML terminology).

Steve

Re: [Numpy-discussion] Records in scipy core

[Stephen W. <go...@co...>]

Colin J. Williams wrote:
> Stephen Waterbury wrote:
>> I'm doing that in an application I'm developing.
>> My objects have an attribute called '_schema' that is an instance
>> of Zope InterfaceClass.  An object (read "record" ;) is assigned a 
>> _schema
>> when it is instantiated, and all information about its attributes (a.k.a.
>> "fields") is contained in the _schema's Properties (my 'Property' 
>> subtypes
>> the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes
>> like 'domain', 'range', 'id', 'name', etc. -- which could easily be
>> extended to include things like 'title', but I use another mechanism
>> for display characteristics, called 'DisplayMap', which can be used to
>> specify the order in which you want the object's properties to appear
>> in a grid, what you want their "display names" to be, etc. ... which are
>> also customizable by the end-user.
> 
> This is goes further than my suggestion.  For arrays, it seems to me 
> that an additional pointer to _schema is not needed as there is a 
> pointer to the data type and the data type can contain the meta data

In effect, the _schema *is* the data type in my scenario.
It contains all information necessary to describe the type
of the object and has references to the types of all the
object's attributes (which are called "fields" in record
parlance, "Properties" in the world of ontologies, and
"Attributes" in Zope Interfaces and UML terminology).

Steve

Re: [Numpy-discussion] Records in scipy core

[Colin J. W. <cj...@sy...>]

Stephen Waterbury wrote:

> Travis Oliphant wrote:
>
>> So, I've been re-thinking the notion of "registering a data-type".  
>> It seems to me that while it's O.K. to have a set of pre-defined data 
>> types.  The notion of data-type ought to be flexible enough to allow 
>> the user to define one "on-the-fly".
>> I'm thinking of ways to do this right now.  Any suggestions are welcome.
>
>
> I'm doing that in an application I'm developing.
> My objects have an attribute called '_schema' that is an instance
> of Zope InterfaceClass.  An object (read "record" ;) is assigned a 
> _schema
> when it is instantiated, and all information about its attributes (a.k.a.
> "fields") is contained in the _schema's Properties (my 'Property' 
> subtypes
> the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes
> like 'domain', 'range', 'id', 'name', etc. -- which could easily be
> extended to include things like 'title', but I use another mechanism
> for display characteristics, called 'DisplayMap', which can be used to
> specify the order in which you want the object's properties to appear
> in a grid, what you want their "display names" to be, etc. ... which are
> also customizable by the end-user.
>
> Let me know if this sounds interesting.
>
> Cheers,
> Steve

This is goes further than my suggestion.  For arrays, it seems to me 
that an additional pointer to _schema is not needed as there is a 
pointer to the data type and the data type can contain the meta data

Colin W.

Re: [Numpy-discussion] Records in scipy core

[Colin J. W. <cj...@sy...>]

Travis Oliphant wrote:

> Perry Greenfield wrote:
>
>>  
>>
>> For us, probably not critical since we have to do some rewriting anyway.
>> (But it would be nice to retain for a while as deprecated).
>>  
>>
> Easy enough to do by defining an actual record array (however, see 
> below).   I've been retaining backwards compatibility in other ways 
> while not documenting it.  For example, you can actually now pass in 
> strings like 'Int32' for types.
>
>> But what about field names that don't map well to attributes?
>> I haven't had a chance to reread the past emails but I seem to
>> recall this was a significant issue. That would imply that .field()
>> would be needed for those cases anyway.
>>  
>>
> What I'm referring to as the solution here is a slight modification to 
> what Perry described.  In other words, all arrays have the attribute
>
> .fields

What I suggested in my posting was that there is no need and no benefit 
from the .fields attribute.

The base class Record could be organized so that certain attributes 
which are used in arrays are not acceptable.  For example, one would 
probably wish to avoid shape, size and the other attributes of the basic 
array but attributes associated with arrays with numeric types would 
probably not need to be barred.

>
> You can set this attribute to a dictionary which will automagically 
> gives field names to any array (this dictionary has ordered lists of 
> 'names', (optionally) 'titles', and "(data-descr, [offset])" lists 
> which defines the mapping.  If offset is not given, then the 
> "next-available" offset is assumed.  The data-descr is either 1) a 
> data-type or 2) a tuple of (data-type, shape).   The data-type is 
> either a defined data-type or alias, or an object with a .fields 
> attribute that provides the same dictionary and an .itemsize attribute 
> that computes the total size of the data-type.
>
I wonder about the need for explicit dictionary operations.  Can't this 
be handled through the class structure?

>
> You can get this attribute which returns a special fields object 
> (written in Python initially like the flags attribute) that can look 
> up field names like a dictionary, or with attribute access for names 
> that are either 1) acceptable or 2) have a user-provided "python-name" 
> associated with them. 
> Thus,
>
> .fields['home address']
>
> would always work
>
> but
>
> .fields.hmaddr
>
> would only work if the user had previously made the association hmaddr 
> -> 'home address' for the data type of this array.   Thus 'home 
> address' would be a title but hmaddr would be the name.
>
> The records module would simply provide functions for making record 
> arrays and a record data type.
> Driving my thinking is the concept that the notion of a record array 
> is really a description of the data type of the array (not the array 
> itself).  Thus, all the fields information should really just be part 
> of the data type itself.  Now, I don't really want to create and 
> register a new data type every time somebody has a new record layout.
>
A record array is an array which has a record as its data element, in 
the same way that an integer array has an integer as its element.

I don't understand the notion of registring a data type.  Presumably an 
integer array has a pointer to the appropriate type of integer.  Could 
the record array not have a pointer to the appropriate record type?

> So, I've been re-thinking the notion of "registering a data-type".  It 
> seems to me that while it's O.K. to have a set of pre-defined data 
> types.  The notion of data-type ought to be flexible enough to allow 
> the user to define one "on-the-fly".
> I'm thinking of ways to do this right now.  Any suggestions are welcome.
>
The record types would be created "on-the-fly" as the class is 
instatiated.  The array, through the dtype parameter would point to the 
record type.

>
> -Travis

Colin W.

Re: [Numpy-discussion] Records in scipy core

[Stephen W. <go...@co...>]

Travis Oliphant wrote:
> So, I've been re-thinking the notion of "registering a data-type".  It 
> seems to me that while it's O.K. to have a set of pre-defined data 
> types.  The notion of data-type ought to be flexible enough to allow the 
> user to define one "on-the-fly".
> I'm thinking of ways to do this right now.  Any suggestions are welcome.

I'm doing that in an application I'm developing.
My objects have an attribute called '_schema' that is an instance
of Zope InterfaceClass.  An object (read "record" ;) is assigned a _schema
when it is instantiated, and all information about its attributes (a.k.a.
"fields") is contained in the _schema's Properties (my 'Property' subtypes
the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes
like 'domain', 'range', 'id', 'name', etc. -- which could easily be
extended to include things like 'title', but I use another mechanism
for display characteristics, called 'DisplayMap', which can be used to
specify the order in which you want the object's properties to appear
in a grid, what you want their "display names" to be, etc. ... which are
also customizable by the end-user.

Let me know if this sounds interesting.

Cheers,
Steve