numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] numarray types and PIL modes, revisited

[Todd M. <jm...@st...>]

Edward C. Jones wrote:

> So now all I can say is:
>
> 1. UInt8 == 'X' should not raise an exception. It should return False.

OK.   I'll change numarray to return False.

>
> 3. There needs to be a function that returns True iff arg is a numarry 
> type (UInt8, "UInt8", "b", ...).
>
> def IsType(rep):
>     from numerictypes import typeDict
>     return isinstance(rep, NumericType) or typeDict.has_key(rep)

Sounds good too.  I'll add this to numerictypes.

>
>
Thanks,
Todd

Re: [Numpy-discussion] End of Holidays small comments

[Todd M. <jm...@st...>]

Wow!   This is great feedback.  Thanks Edward.

Edward C. Jones wrote:

> node35.html:
>
>     >>> print x.type(), x.real.type()
>     D d
>
> should be
>
>     >>> print x.type(), x.real.type()
>     numarray type: Complex64 numarray type: Float64

I taked this over with Perry,  and think it should behave and be 
documented more like:
 >>> print x.type(), x.real.type()
Complex64  Float64

>
> ------------------------------------------------
>
> Why use both NUM_C_ARRAY and C_ARRAY?

In the context of the defining enumeration,  NUM_C_ARRAY looks correct. 
  Anywhere else,  C_ARRAY is about all I can stand.   C_ARRAY is so 
common that I thought a little irregularity would be tolerable.  Chock 
it up to tastelessness.

>
> ------------------------------------------------
>
> in  _ndarraymodule.c:
>
>         {"_byteoffset",
>          (getter)_ndarray_byteoffset_get,
>          (setter)_ndarray_byteoffset_set,
>          "shortest seperation between elements in bytes"},
>         {"_bytestride",
>          (getter)_ndarray_bytestride_get,
>          (setter)_ndarray_bytestride_set,
>          "shortest seperation between elements in bytes"},
>
> One of the comments is wrong. Also "separation".

Noted.

>
> ------------------------------------------------
>
> libnumarraymodule.c:
>
>     /* Create an empty array. */
>     static PyArrayObject *
>     NA_Empty(int ndim, int *shape, NumarrayType type)
>
> node42.html:
>
>     static PyObject* NA_Empty( NumarrayType type, int ndim, ...)
>
Noted.

>
> ------------------------------------------------
>
> I think NA_New should be
>
>     NA_New(int ndim, int* shape, NumarrayType type, void* buffer)
>
> The current NA_New is useful only when ndim is known at code-writing 
> time.

NA_New is a  "convenience wrapper" around NA_NewAll,  but I see your point.

How about NA_vNew(),  in the spirit of vprintf?

>
> ------------------------------------------------
>
> node39.html:
>
>     Note: the type parameter for a macro is one of the Numarray Numeric
>     Data Types, not a NumarrayType enumeration value.
>
> There should be an example of one of the GET/SET macros. How about
>
>     unsigned char n;
>     int i;
>     ...
>     n = NA_GET1(arr, UInt8, i);

OK.

>
> ------------------------------------------------
>
> It seems that the parameters "aligned" and "writeable" are ignored in 
> the source code for NA_NewAll and class NumArray.

"aligned" is used.

"writeable" should probably be dropped since it is no longer used.   
Since doing that would break an interface someone might be using,  I'd 
rather not.

>
> ------------------------------------------------
>
> I would like to see an "int* strides" parameter added to NA_NewAll, so a
> non-contiguous "buffer" can be used. 

OK.   How about NA_NewAllWithStrides (or insert a better name here)?

>
> ------------------------------------------------
>
> I suggest NA_Copy(PyObject* arr) which is something like
>
> static PyObject* NA_Copy(PyObject* arr)
> {
>     PyArrayObject* arr1 = arr;
>     return NA_NewAll(arr1->nd, (long*) arr1->dimensions, 

This  ((long *)) doesn't work portably, so I would recommend avoiding it.

>
>        arr1->descr->type_num, arr1->data, arr1->byteoffset,
>        arr1->bytestride, arr1->byteorder, 1, 1);
> }
>
I'll add NA_Copy().

Re: [Numpy-discussion] Slicing API?

[Todd M. <jm...@st...>]

Edward C. Jones wrote:

> Todd Miller wrote:
>
>> Edward C. Jones wrote:
>>
>>> Both in Numeric and now in numarray I have found a need for API 
>>> functions for slicing. Has anyone thought about this?
>>>
>> Speaking for myself and the numarray C-API, the answer is no.   What 
>> API do you want?   Can you suggest function prototypes? 
>
>
> An API version of  arrout[slices] = arrin[slices]:
>
> static int
> NA_CopySlice(PyArrayObject* arrin, PyArrayObject* arrout,
>     int* startin, int* stepin, int* stopin, int* startout, int* stepout);
>
>
I would suggest something more like the following then:

typedef struct {
    int start, stop, step;
} NumSlice;

static int
NA_CopySlice(PyArrayObject* arrin, int indim, NumSlice *slicein,
    PyArrayObject* arrout,  int outdim, NumSlice *sliceout);

The differences are:

1.  A slice dimension count is added for both input and output arrays. 
 This enables use of partial indices.

2.  Slice values are expressed using the NumSlice typedef/struct rather 
than 3 independent int arrays.

3. The parameter order is shuffled so that input array parameters are 
kept together, and output array parameters are kept together.

But,  I still have these comments:

1.  It looks like it will be cumbersome to use.

2.  We should probably implement it as a callback to Python to avoid 
introducing another set of assignment semantics.  Thus, the 
implementation would really just be building up and executing the calls 
for:  outarr.__setitem__(outslices, inarr.__getitem__(inslices)).

3. The slicing implementation for numarray objects should be optimized 
to C this quarter, if not this month.  So in terms of efficiency, not to 
mention comment 2, this won't buy much.

4. Since Numeric doesn't have this already,  we're probably missing 
something obvious.  

Comments?  Still interested?

Todd

Re: [Numpy-discussion] Slicing API?

[Todd M. <jm...@st...>]

Edward C. Jones wrote:

> Both in Numeric and now in numarray I have found a need for API 
> functions for slicing. Has anyone thought about this?
>
Speaking for myself and the numarray C-API, the answer is no.   What API 
do you want?   Can you suggest function prototypes?

Todd

[Numpy-discussion] Slicing API?

[Edward C. J. <edc...@er...>]

Both in Numeric and now in numarray I have found a need for API 
functions for slicing. Has anyone thought about this?

[Numpy-discussion] End of Holidays small comments

[Edward C. J. <edc...@er...>]

node35.html:

     >>> print x.type(), x.real.type()
     D d

should be

     >>> print x.type(), x.real.type()
     numarray type: Complex64 numarray type: Float64

------------------------------------------------

Why use both NUM_C_ARRAY and C_ARRAY?

------------------------------------------------

in  _ndarraymodule.c:

         {"_byteoffset",
          (getter)_ndarray_byteoffset_get,
          (setter)_ndarray_byteoffset_set,
          "shortest seperation between elements in bytes"},
         {"_bytestride",
          (getter)_ndarray_bytestride_get,
          (setter)_ndarray_bytestride_set,
          "shortest seperation between elements in bytes"},

One of the comments is wrong. Also "separation".

------------------------------------------------

libnumarraymodule.c:

     /* Create an empty array. */
     static PyArrayObject *
     NA_Empty(int ndim, int *shape, NumarrayType type)

node42.html:

     static PyObject* NA_Empty( NumarrayType type, int ndim, ...)

Serious documentation error.

------------------------------------------------

I think NA_New should be

     NA_New(int ndim, int* shape, NumarrayType type, void* buffer)

The current NA_New is useful only when ndim is known at code-writing time.

------------------------------------------------

node39.html:

     Note: the type parameter for a macro is one of the Numarray Numeric
     Data Types, not a NumarrayType enumeration value.

There should be an example of one of the GET/SET macros. How about

     unsigned char n;
     int i;
     ...
     n = NA_GET1(arr, UInt8, i);

------------------------------------------------

It seems that the parameters "aligned" and "writeable" are ignored in 
the source code for NA_NewAll and class NumArray.

------------------------------------------------

I would like to see an "int* strides" parameter added to NA_NewAll, so a
non-contiguous "buffer" can be used.

------------------------------------------------

I suggest NA_Copy(PyObject* arr) which is something like

static PyObject* NA_Copy(PyObject* arr)
{
     PyArrayObject* arr1 = arr;
     return NA_NewAll(arr1->nd, (long*) arr1->dimensions,
        arr1->descr->type_num, arr1->data, arr1->byteoffset,
        arr1->bytestride, arr1->byteorder, 1, 1);
}

[Numpy-discussion] numarray types and PIL modes, revisited

[Edward C. J. <edc...@er...>]

Perry Greenfield wrote:
 > Edward Jones writes:

 > > I write code using both PIL and numarray. PIL uses strings for
 > > modes and numarray uses (optionally) strings as typecodes. This
 > > causes problems.  One fix is to emit a DeprecationWarning when
 > > string typecodes are used.  Two functions are needed:
 > > StringTypeWarningOn and StringTypeWarningOff.  The default
 > > should be to ignore this warning.
 >
 > I'm not sure I understand. Can you give me an example of problem
 > code or usage? It sounds like you are trying to test the types of
 > PIL and numarray objects in a generic sense. But I'd understand
 > better if you could show an example.

That's what I was thinking (incorrectly). But I don't need to directly 
compare PIL modes with numarray types.

My code never tries to deduce whether an array is a numarray or a PIL 
image from just the natype_or_mode. A module name (MODULE.NUMARRY, 
MODULE.PIL) must also be given. I do things this way because I might 
want to include other array/image systems. In an earlier version, I had 
a MODULE.IPL for the Intel Image Processing Library.

The code also implements a policy of forbidding string types.

So now all I can say is:

1. UInt8 == 'X' should not raise an exception. It should return False.

3. There needs to be a function that returns True iff arg is a numarry 
type (UInt8, "UInt8", "b", ...).

def IsType(rep):
     from numerictypes import typeDict
     return isinstance(rep, NumericType) or typeDict.has_key(rep)


Here is a typical piece of code. "module" can be MODULE.PIL or
MODULE.NUMARRAY.

----
"""General image casting function. Changes the C type of the pixels. 
Information can be lost. The "Convert" functions call C casting 
functions that clip the values, For example, if the input is a UInt16 
and the output is a Int16, any input value greater than 32767 becomes 32767.
"""
def ArrayToArrayCast(arrin, module, natype_or_mode):
     """Converts one array into another. Results are clipped."""
     pars = Parameters(arrin)
     if pars.module == module == MODULE.PIL and \
           pars.mode == natype_or_mode:
         return arrin
     if pars.module == module == MODULE.NUMARRAY and \
                      NA_SameType(pars.natype, natype_or_mode):
         return arrin
     if pars.module == MODULE.NUMARRAY and module == MODULE.NUMARRAY:
         return NA_To_NA_Convert(arrin, natype_or_mode)
     if pars.module == MODULE.PIL and module == MODULE.PIL:
         return PIL_To_PIL_Convert(arrin, natype_or_mode)
     if pars.module == MODULE.NUMARRAY and module == MODULE.PIL:
         return NA_To_PIL_Convert(arrin, natype_or_mode)
     if pars.module == MODULE.PIL and module == MODULE.NUMARRAY:
         return PIL_To_NA_Convert(arrin, natype_or_mode)
----

Re: [Numpy-discussion] argsort on CharArrays?

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>Todd Miller <jm...@st...>:
>  
>
>>Magnus Lie Hetland wrote:
>>
>>    
>>
>>>Is there any way to make argsort work on CharArrays?
>>>
>>>      
>>>
>>Not yet.
>>    
>>
>
>OK, thanks.
>
>(I don't mean to nag -- I'm just eager about numarray :)
>  
>
Don't worry about nagging.    All this feedback is good.  Results, 
however,  may take a little time.

>>Todd
>>    
>>
Todd

Re: [Numpy-discussion] Lists in fromfile shape?

[Magnus L. H. <ma...@he...>]

Magnus Lie Hetland <ma...@he...>:
>
> I just noticed that the shape argument in fromfile cannot be an
> unhashable sequence. I assume tuples are the norm, but still -- it
> would be nice to accept other sequences?

Sorry about that -- it worked after all. My mistake.

-- 
Magnus Lie Hetland
http://hetland.org

[Numpy-discussion] Lists in fromfile shape?

[Magnus L. H. <ma...@he...>]

I just noticed that the shape argument in fromfile cannot be an
unhashable sequence. I assume tuples are the norm, but still -- it
would be nice to accept other sequences?

And... I also saw an explicit type check in fromfile, comparing
_type(file) with _type("") -- what if I supplied some other
string-like object? Perhaps a try/except with file+"" would be
appropriate, to check for "string-like-ness" instead of an explicit
type check? (I've just grown suspicious of typechecking in general...)

Sorry if I'm being a pest, nagging about all these details, but I
expect it may be useful to get them resolved? :]

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] argsort on CharArrays?

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>Is there any way to make argsort work on CharArrays?
>
>  
>
Not yet.

Todd

[Numpy-discussion] argsort on CharArrays?

[Magnus L. H. <ma...@he...>]

Is there any way to make argsort work on CharArrays?

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] Bin selection

[Magnus L. H. <ma...@he...>]

Todd Miller <jm...@st...>:
[snip]
> Try searchsorted().

Ah! <slaps forehead> Thanks! That's exactly what I was looking for; I
think I've seen it before, but somehow I missed this when looking for
it...

Thanks again,

Magnus

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] Bin selection

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>I have a set of limits, e.g. array([0, 35, 45, 55, 75]) and I want to
>use these to "classify" a set of numbers (another one-dimensional
>array). The "class" is the number of the first limit that is lower
>than or equal to the number I want to classify. E.g. I'd classify 17
>as 0 and 42 as 1.
>
>My current approach is:
>
>  sum(nums[:,NewAxis] >= lims, dim=-1)
>
>It seems a bit unnecessary to compare each number with all the limits
>when O(log(n)) would suffice (the limits are ordered); or even with
>O(n) running time, a smarter implementation could get an average of
>n/2 comparisons...
>
>Suggestions?
>
>  
>
Try searchsorted().  Searchsorted returns the index of the first bin >= 
the number being classified and has O(log(n)) running time.

 >>> a=numarray.array([0, 35, 45, 55, 75])
 >>> numarray.searchsorted(a, [1,42,35])
array([1, 2, 1])

Todd

[Numpy-discussion] Possible NetCDF bug...

[Chris B. <Chr...@no...>]

Hi all,

Does anyone here use Konrad Hinsen's NetCDF module? I'm having a problem
with it.


I opened an existing NetCDF file and examine one of it's variables:

file = NetCDF.NetCDFFile("test.cdf")

>>> file.variables['water_u'].typecode()
'f'

The variable appears to contain float data (which I think it does).
However, when I try to get a slice of that data:

array([3199453293L, 3198428385L], 'u')

Numeric appears to think it's an unsigned 32 bit integer variable. I get
the same result if I use file.getValue() as well.

Could this be a Numeric version mismatch? I couldn't find a recommended
version of Numeric on the Scientific site.

RedHat Linux 7.2
Python2.2.1
Numeric 22.0

I've enclosed my test.cdf file, if you want to test it yourself.

Anyone have any ideas?

-Chris






-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

[Numpy-discussion] Bin selection

[Magnus L. H. <ma...@he...>]

I have a set of limits, e.g. array([0, 35, 45, 55, 75]) and I want to
use these to "classify" a set of numbers (another one-dimensional
array). The "class" is the number of the first limit that is lower
than or equal to the number I want to classify. E.g. I'd classify 17
as 0 and 42 as 1.

My current approach is:

  sum(nums[:,NewAxis] >= lims, dim=-1)

It seems a bit unnecessary to compare each number with all the limits
when O(log(n)) would suffice (the limits are ordered); or even with
O(n) running time, a smarter implementation could get an average of
n/2 comparisons...

Suggestions?

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] Yet another bug

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>Hi, there... I just found another bug :]
>
>  
>
numarray doesn't support "true division" yet.

>from numarray import arange
>from __future__ import division
>
>arange(10) / 10
>
>... This will give an error.
>
>I really think this "real" division should be supported for arrays...
>  
>
It will be.  It just hasn't been implemented yet.

>  
>

[Numpy-discussion] Yet another bug

[Magnus L. H. <ma...@he...>]

Hi, there... I just found another bug :]

from numarray import arange
from __future__ import division

arange(10) / 10

... This will give an error.

I really think this "real" division should be supported for arrays...

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] Compiling from CVS?

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>I guess I'm just missing something obvious here: How do I compile
>numarray from the cvs dist? It seems like files such as numconfig.h
>
Not really.  People have asked for modifications to setup.py to support 
a two step build/install process.  CVS has the first cut,  which works, 
but is harder to use than it has to be.

>etc. are missing -- it seems to have been generated by setup.py in the
>normal distributions, but I'm not sure how to do that... I just
>expected "python setup.py build" to work :]
>  
>
Try:

python setup.py build --gencode
python setup.py install

for now.   I plan to revisit this later and eliminate the need for the 
switch.

Todd

[Numpy-discussion] Compiling from CVS?

[Magnus L. H. <ma...@he...>]

I guess I'm just missing something obvious here: How do I compile
numarray from the cvs dist? It seems like files such as numconfig.h
etc. are missing -- it seems to have been generated by setup.py in the
normal distributions, but I'm not sure how to do that... I just
expected "python setup.py build" to work :]

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] indices with list arg?

[Magnus L. H. <ma...@he...>]

Todd Miller <jm...@st...>:
>
[snip]
> Neither.   It  looks like a limitation of the indices function.   
> Inserting "shape = tuple(shape)" into indices() removed the limitation.

Ah.

But it's still impossible to use foo.setshape([bar]), then, I guess,
even though it works with reshape()?

> Thanks,
> Todd

-- 
Magnus Lie Hetland
http://hetland.org

Re: [Numpy-discussion] indices with list arg?

[Todd M. <jm...@st...>]

Magnus Lie Hetland wrote:

>Why isn't indices([42]) (or foo.setshape([42]), for that matter)
>allowed? Overzealous type checking, or a necessity of the C code?
>
>  
>
Neither.   It  looks like a limitation of the indices function.   
Inserting "shape = tuple(shape)" into indices() removed the limitation.

Thanks,
Todd

[Numpy-discussion] indices with list arg?

[Magnus L. H. <ma...@he...>]

Why isn't indices([42]) (or foo.setshape([42]), for that matter)
allowed? Overzealous type checking, or a necessity of the C code?

-- 
Magnus Lie Hetland
http://hetland.org

[Numpy-discussion] average()...?

[Magnus L. H. <ma...@he...>]

Just a quick question (which I believe I have asked before sometime,
but I don't recall the outcome ;) -- why is the average() function
from MaskedArray not available in numarray? It seems like a very
useful function?

It seems that looking through the list of functions in MaskedArray
might be useful in finding candidates for numarray, in so far as they
differ. (Wouldn't be logical if they supported the same functions,
more or less?) For example, MaskedArray has the function count(),
which counts the number of elements in an array -- exactly what was
asked for yesterday, IIRC. (The use may be more obvious for masked
arrays, where it counts non-masked elements, but it works on plain
arrays too.)

If nothing else, perhaps it would be possible to put

from MA import count, average

in numarray.py? (And similarly for other functions that may be missing
in numarray -- I didn't check that closely.) Of course this wouldn't
work if MA isn't available... :/

-- 
Magnus Lie Hetland
http://hetland.org

RE: [Numpy-discussion] Non-numerical info associated with sub-arrays

[Francesc A. <fa...@op...>]

Mensaje citado por: Perry Greenfield <pe...@st...>:

> Are you asking for an option to create record arrays with
> aligned fields (in the sense that the addresses of all values
> are consistent with their type)?

Yes, I'm advocating for that

> Or are you arguing that
> non-aligned columns must be prohibited? The former is certainly
> possible (not not very difficult to implement; basically it requires
> that record sizes must be a multiple of the largest numerical type,
> and that padding is placed within records to ensure that all fields
> have offsets that are a multiple of their size).

Well, for the sake of keeping the size of dataset to a minimum, I think it's 
not necessary to adjust all the record field sizes to the largest data type 
because depending on the type of the field, the padding can be shorter or 
larger.

For example, short ints only needs to be aligned in two-byte basis, while 
doubles need 4 bytes (or 8, I don't remember well). In any case, this depends 
on the architecture. But it is still possible to figure out safely what is the 
required minimum alignments for the different types. Look at Python's struct 
module for a good example on how you can reduce the padding to a minimum, 
without sacrificing performance.

Francesc Alted