numpy-discussion — Discussion list for all users of Numerical Python

[Numpy-discussion] numarray 0.4 on osX/darwin

[Sebastian H. <ha...@ms...>]

Hi everybody,
I tried a 'python2.2 setup.py install'
of numarray  on a Mac running os-X (10.1; I have also Fink installed)
I starts crunching until:
/usr/bin/ld: Undefined symbols:
_fclearexcept
_fetestexcept

Anyone out there, who uses numarray on osX ?

I'm thankful for any pointer...

Sebastian Haase

[Numpy-discussion] mlab functions: psd, csd, cohere, corrcoef

[John H. <jdh...@ac...>]

I needed some spectral analysis functions, and finding none available,
wrote my own.  I use matlab a lot, so I wrote them to be matlab
compatible.  If you all think these look OK, I'm happy to submit them
for inclusion into MLab.  

-------------------------------------------------------------------

"""

Spectral analysis functions for Numerical python written for
compatability with matlab commands with the same names.

  psd - Power spectral density uing Welch's average periodogram
  csd - Cross spectral density uing Welch's average periodogram
  cohere - Coherence (normalized cross spectral density)
  corrcoef - The matrix of correlation coefficients

The functions are designed to work for real and complex valued Numeric
arrays.

One of the major differences between this code and matlab's is that I
use functions for 'detrend' and 'window', and matlab uses vectors.
This can be easily changed, but I think the functional approach is a
bit more elegant.

Please send comments, questions and bugs to:

Author: John D. Hunter <jdh...@ac...>

"""

from __future__ import division
from MLab import mean, hanning, cov
from Numeric import zeros, ones, diagonal, transpose, matrixmultiply, \
     resize, sqrt, divide, array, Float, Complex, concatenate, \
     convolve, dot, conjugate, absolute, arange, reshape
from FFT import fft


def norm(x):
    return sqrt(dot(x,x))

def window_hanning(x):
    return hanning(len(x))*x

def window_none(x):
    return x

def detrend_mean(x):
    return x - mean(x)

def detrend_none(x):
    return x

def detrend_linear(x):
    """Remove the best fit line from x"""
    # I'm going to regress x on xx=range(len(x)) and return
    # x - (b*xx+a)
    xx = arange(len(x), typecode=x.typecode())
    X = transpose(array([xx]+[x]))
    C = cov(X)
    b = C[0,1]/C[0,0]
    a = mean(x) - b*mean(xx)
    return x-(b*xx+a)


def psd(x, NFFT=256, Fs=2, detrend=detrend_none,
        window=window_hanning, noverlap=0):
    """
    The power spectral density by Welches average periodogram method.
    The vector x is divided into NFFT length segments.  Each segment
    is detrended by function detrend and windowed by function window.
    noperlap gives the length of the overlap between segments.  The
    absolute(fft(segment))**2 of each segment are averaged to compute Pxx,
    with a scaling to correct for power loss due to windowing.  Fs is
    the sampling frequency.

    -- NFFT must be a power of 2
    -- detrend and window are functions, unlike in matlab where they are
       vectors.
    -- if length x < NFFT, it will be zero padded to NFFT
    

    Refs:
      Bendat & Piersol -- Random Data: Analysis and Measurement
        Procedures, John Wiley & Sons (1986)

    """

    if NFFT % 2:
        raise ValueError, 'NFFT must be a power of 2'

    # zero pad x up to NFFT if it is shorter than NFFT
    if len(x)<NFFT:
        n = len(x)
        x = resize(x, (NFFT,))
        x[n:] = 0
    

    # for real x, ignore the negative frequencies
    if x.typecode()==Complex: numFreqs = NFFT
    else: numFreqs = NFFT//2+1
        
    windowVals = window(ones((NFFT,),x.typecode()))
    step = NFFT-noverlap
    ind = range(0,len(x)-NFFT+1,step)
    n = len(ind)
    Pxx = zeros((numFreqs,n), Float)

    # do the ffts of the slices
    for i in range(n):
        thisX = x[ind[i]:ind[i]+NFFT]
        thisX = windowVals*detrend(thisX)
        fx = absolute(fft(thisX))**2
        Pxx[:,i] = fx[:numFreqs]

    # Scale the spectrum by the norm of the window to compensate for
    # windowing loss; see Bendat & Piersol Sec 11.5.2
    if n>1: Pxx = mean(Pxx,1)
    Pxx = divide(Pxx, norm(windowVals)**2)
    freqs = Fs/NFFT*arange(0,numFreqs)
    return Pxx, freqs



def csd(x, y, NFFT=256, Fs=2, detrend=detrend_none,
        window=window_hanning, noverlap=0):
    """
    The cross spectral density Pxy by Welches average periodogram
    method.  The vectors x and y are divided into NFFT length
    segments.  Each segment is detrended by function detrend and
    windowed by function window.  noverlap gives the length of the
    overlap between segments.  The product of the direct FFTs of x and
    y are averaged over each segment to compute Pxy, with a scaling to
    correct for power loss due to windowing.  Fs is the sampling
    frequency.

    NFFT must be a power of 2

    Refs:
      Bendat & Piersol -- Random Data: Analysis and Measurement
        Procedures, John Wiley & Sons (1986)

    """

    if NFFT % 2:
        raise ValueError, 'NFFT must be a power of 2'

    # zero pad x and y up to NFFT if they are shorter than NFFT
    if len(x)<NFFT:
        n = len(x)
        x = resize(x, (NFFT,))
        x[n:] = 0
    if len(y)<NFFT:
        n = len(y)
        y = resize(y, (NFFT,))
        y[n:] = 0

    # for real x, ignore the negative frequencies
    if x.typecode()==Complex: numFreqs = NFFT
    else: numFreqs = NFFT//2+1
        
    windowVals = window(ones((NFFT,),x.typecode()))
    step = NFFT-noverlap
    ind = range(0,len(x)-NFFT+1,step)
    n = len(ind)
    Pxy = zeros((numFreqs,n), Complex)

    # do the ffts of the slices
    for i in range(n):
        thisX = x[ind[i]:ind[i]+NFFT]
        thisX = windowVals*detrend(thisX)
        thisY = y[ind[i]:ind[i]+NFFT]
        thisY = windowVals*detrend(thisY)
        fx = fft(thisX)
        fy = fft(thisY)
        Pxy[:,i] = fy[:numFreqs]*conjugate(fx[:numFreqs])

    # Scale the spectrum by the norm of the window to compensate for
    # windowing loss; see Bendat & Piersol Sec 11.5.2
    if n>1: Pxy = mean(Pxy,1)
    Pxy = divide(Pxy, norm(windowVals)**2)
    freqs = Fs/NFFT*arange(0,numFreqs)
    return Pxy, freqs

def cohere(x, y, NFFT=256, Fs=2, detrend=detrend_none,
           window=window_hanning, noverlap=0):
    """
    cohere the coherence between x and y.  Coherence is the normalized
    cross spectral density

    Cxy = |Pxy|^2/(Pxx*Pyy)

    The return value is (Cxy, f), where f are the frequencies of the
    coherence vector.  See the docs for psd and csd for information
    about the function arguments NFFT, detrend, windowm noverlap, as
    well as the methods used to compute Pxy, Pxx and Pyy.

    """

    
    Pxx,f = psd(x, NFFT=NFFT, Fs=Fs, detrend=detrend,
              window=window, noverlap=noverlap)
    Pyy,f = psd(y, NFFT=NFFT, Fs=Fs, detrend=detrend,
              window=window, noverlap=noverlap)
    Pxy,f = csd(x, y, NFFT=NFFT, Fs=Fs, detrend=detrend,
              window=window, noverlap=noverlap)

    Cxy = divide(absolute(Pxy)**2, Pxx*Pyy)
    return Cxy, f

def corrcoef(*args):
    """
    
    corrcoef(X) where X is a matrix returns a matrix of correlation
    coefficients for each row of X.
    
    corrcoef(x,y) where x and y are vectors returns the matrix or
    correlation coefficients for x and y.

    Numeric arrays can be real or complex

    The correlation matrix is defined from the covariance matrix C as

    r(i,j) = C[i,j] / (C[i,i]*C[j,j])
    """

    if len(args)==2:
        X = transpose(array([args[0]]+[args[1]]))
    elif len(args==1):
        X = args[0]
    else:
        raise RuntimeError, 'Only expecting 1 or 2 arguments'

    
    C = cov(X)
    d = resize(diagonal(C), (2,1))
    r = divide(C,sqrt(matrixmultiply(d,transpose(d))))[0,1]
    try: return r.real
    except AttributeError: return r


-------------------------------------------------------------------

I wrote a little test code comparing the output of matlab's equivalent
functions.  Basically, I compute the psd or cohere in matlab and
python and do the rms difference on the resultant vectors

  RMS cohere python/matlab difference 0.000854587104587
  RMS psd python/matlab difference 0.00210783306638

I am not sure where these differences are arising, but they are quite
small.  I'm going to keep trying to track them down.

For corrcoef, the answers are the same past 8 significant digits.

Hope this helps!
John Hunter

Re: [Numpy-discussion] fastest way to make two vectors into an array

[John H. <jdh...@ac...>]

>>>>> "John" == John Hunter <jdh...@ac...> writes:

    John> I have two equal length 1D arrays of 256-4096 complex or
    John> floating point numbers which I need to put into a
    John> shape=(len(x),2) array.

    John> I need to do this a lot, so I would like to use the most
    John> efficient means.  Currently I am doing:

I tested all the suggested methods and the transpose with [x] and [y]
was the clear winner, with an 8 fold speed up over my original code.
The concatenate method was between 2-3 times faster.

Thanks to all who responded,
John Hunter

cruncher2:~/python/test> python test.py test_naive
test_naive 0.480427026749
cruncher2:~/python/test> python test.py test_concat
test_concat 0.189149975777
cruncher2:~/python/test> python test.py test_transpose
test_transpose 0.0698409080505


from Numeric import transpose, concatenate, reshape, array, zeros
from RandomArray import normal
import time, sys

def test_naive(x,y):
    "Naive approach"
    X = zeros( (len(x),2), typecode=x.typecode())
    X[:,0] = x
    X[:,1] = y

def test_concat(x,y):
    "Thanks to Chris Barker and Bryan Cole"
    X = concatenate( ( reshape(x,(-1,1)), reshape(y,(-1,1)) ), 1)


def test_transpose(x,y):
    "Thanks to Joachim Saul"
    X = transpose(array([x]+[y]))


m = {'test_naive' : test_naive,
     'test_concat' : test_concat,
     'test_transpose' : test_transpose}

nse1 = normal(0.0, 1.0, (4096,))
nse2 = normal(0.0, 1.0, nse1.shape)

N = 1000

trials = range(N)

func = m[sys.argv[1]]
t1 = time.time()
for i in trials:
    func(nse1,nse2)
t2 = time.time()
print sys.argv[1], t2-t1

Re: [Numpy-discussion] fastest way to make two vectors into an array

[Pearu P. <pe...@ce...>]

On Wed, 29 Jan 2003, John Hunter wrote:

> 
> I have two equal length 1D arrays of 256-4096 complex or floating
> point numbers which I need to put into a shape=(len(x),2) array.
> 
> I need to do this a lot, so I would like to use the most efficient
> means.  Currently I am doing:
> 
> def somefunc(x,y):
>     X = zeros( (len(x),2), typecode=x.typecode())
>     X[:,0] = x
>     X[:,1] = y
>     do_something_with(X)
> 
> Is this the fastest way?

May be you could arange your algorithm so that you first create
X and then reference its columns by x,y without copying:

# Allocate memory
X = zeros( (n,2), typecode=.. )

# Get references to columns
x = X[:,0]
y = X[:,1]

while 1:
  do_something_inplace_with(x,y)
  do_something_with(X)

Pearu

[Numpy-discussion] Object too deep for desired array

[Karthikesh R. <ka...@ja...>]

Hi, 

i was tring out something like this 
import Numeric
import LinearAlgebra
import cmath
import RandomArray
import copy


def sMatrix(pd, code, window):
    if window == 0:
        nprime = 1
    else:
        nprime = window
    
    K, C = Numeric.shape(code)
    K1, L = Numeric.shape(pd)
    # check if K == K1 and raise an exception here
    sCode = Numeric.zeros([nprime*C,K*L*(window+1)],'d')

    for k in range(K):
        for l in range(L):
            code1 = copy.deepcopy(Numeric.array(code[k,0:C-pd[k,l]]))
            code1.shape = (C-pd[k,l],1)
            sCode1=
Numeric.concatenate((Numeric.zeros([pd[k,l],1]),Numeric.zeros([C*window,1]),code1))
            sCode[:, (window+1)*l+window*L*k] = copy.deepcopy(sCode1)
    
    return sCode

if __name__ == "__main__":
    pd = Numeric.array([[2]])
    code = Numeric.array([[-1,1,-1,1,1]])
    np = sMatrix(pd,code,0)
    print np
    print "--"*30
    np = sMatrix(pd,code,1)
    print Numeric.shape(np)
    print np
    print "--"*30
    np = sMatrix(pd,code,2)
    print Numeric.shape(np)
    print np
    print "--"*30


------------------------------
And i get struck with the following error message::

Traceback (most recent call last):
  File "sMatrix.py", line 31, in ?
    np = sMatrix(pd,code,0)
  File "sMatrix.py", line 24, in sMatrix
    sCode[:, (window+1)*l+window*L*k] = copy.deepcopy(sCode1)
ValueError: Object too deep for desired array


------------

i think it is due to the many deep copy operations taht i am performing. i
want to be in a position where slices of matrices should not be
references, but should be copies itself and i should be able to move these
copies around. (May be it is inefficient, but that is what i did in
Matlab and want some compatibility, till i learn more of python and till
i migrate to python completely).

Is there a way out? Why is this an problem? Am i missing something.

Best regards,

karthik








































-----------------------------------------------------------------------
Karthikesh Raju,		    email: ka...@ja...		
Researcher,			    http://www.cis.hut.fi/karthik
Helsinki University of Technology,  Tel: +358-9-451 5389
Laboratory of Comp. & Info. Sc.,    Fax: +358-9-451 3277
Department of Computer Sc.,
P.O Box 5400, FIN 02015 HUT,
Espoo, FINLAND
-----------------------------------------------------------------------

Re: [Numpy-discussion] fastest way to make two vectors into an array

[Joachim S. <li...@js...>]

* John Hunter [2003-01-29 22:13]:
> def somefunc(x,y):
>     X = zeros( (len(x),2), typecode=x.typecode())
>     X[:,0] = x
>     X[:,1] = y
>     do_something_with(X)
>
> Is this the fastest way?

X = transpose(array([x]+[y]))

It may not be the fastest possible way, but should be about a
factor of two faster; better than nothing.

Cheers,
Joachim

[Numpy-discussion] fastest way to make two vectors into an array

[John H. <jdh...@ac...>]

I have two equal length 1D arrays of 256-4096 complex or floating
point numbers which I need to put into a shape=(len(x),2) array.

I need to do this a lot, so I would like to use the most efficient
means.  Currently I am doing:

def somefunc(x,y):
    X = zeros( (len(x),2), typecode=x.typecode())
    X[:,0] = x
    X[:,1] = y
    do_something_with(X)

Is this the fastest way?

Thanks,
John Hunter

RE: [Numpy-discussion] Extensive use of methods instead of functions

[Perry G. <pe...@st...>]

> Yes, this is critical. Isn't there a plan to make the scalar -- rank-0
> array dicotomy a little cleaner in NumArray ?
>
Hmmm, I'd like to say yes, but I'm not sure what exactly you are
referring to. Please elaborate on how you think it should be
changed. About the only thing that comes to mind is that repr()
for rank-0 will be different for numarray than Numeric, and that
it will never be the result of any reduction or similar selection.
  
> > > I also agree that the point is not subclassing per se, it's
> > > polymorphism. It should be easy to write a class that acts 
> like an array
> > > in all the ways that you need it to. 
> > 
> > True, and that is a weak point of NumPy.
> 
> Is this getting any better with NumArray?
>  
Again, I hope so, but I find this too general to know if it satisfies
anyone's specific goals. I'd like to see specific examples. I think
it is often tricker than people initially think.

Perry

[Numpy-discussion] enum values visible in numeric types instances?

[Francesc A. <fa...@op...>]

Hi,

A couple of points related with numarray type objects:

1.- When working with numeric types instances like UInt8 or Float64, is
there a way to access to their enumeration NumarrayType C counterpart?. T=
hat
can be handy when want to map from these objects and integers.

For example, right now, I'm forced to use these mappings in Pyrex:

# Conversion tables from/to classes to the numarray enum types
toenum =3D {num.Int8:tInt8,       num.UInt8:tUInt8,
          num.Int16:tInt16,     num.UInt16:tUInt16,
          num.Int32:tInt32,     num.UInt32:tUInt32,
          num.Float32:tFloat32, num.Float64:tFloat64,
          CharType:97   # ascii(97) --> 'a' # Special case (to be correct=
ed)
          }

toclass =3D {tInt8:num.Int8,       tUInt8:num.UInt8,
           tInt16:num.Int16,     tUInt16:num.UInt16,
           tInt32:num.Int32,     tUInt32:num.UInt32,
           tFloat32:num.Float32, tFloat64:num.Float64,
           97:CharType   # ascii(97) --> 'a' # Special case (to be correc=
ted)
          }

(yes, Pyrex lets you do that kind of "miracles", like mappings between
Python objects and C integers)

but if I had this access directly from the object (for example
Int8.enumType), my code (and C-extensions in general) could look simpler.

2.- I understand now why Todd was worried about CharArray objects to be
assigned to an enumerated type. In fact, if you look at the above maps, I
have to map myself this special object as the number 97 (which is the asc=
ii
value for character "a"). 97 is ok for now because it can't collide (at
least for a while) with other enumeration types.

My suggestion is that it would be a good thing to have a reserved enum ty=
pe
for CharArray. And I think that mapping CharArrays with Bool or Int8, wou=
ld
not be a good solution because chararray objects differ in some ways from
them, that it would be a mess to distinguish both objects in C-code by ju=
st
looking at its enumeration type.=20

I don't know, but maybe recarrays also merit a place in enumeration (?).=20

By the way, after the discussion with Todd I finally decided to remove al=
l
the Numeric charcodes (and related codes) from PyTables. However, I can
still manage Numeric objects by converting them to numarray and accessing
the class type with the .type() method. An you know that? the code looks
much more logical and neat, and best of all, less error-prone (well, at
least I hope so!). I definitely encourage you to do a similar transition =
in
numarray (although I guess that would be more difficult because you still
need to Numeric compatibility).

Thanks,

--=20
Francesc Alted

Re: [Numpy-discussion] Extensive use of methods instead of functions

[Chris B. <Chr...@no...>]

Konrad Hinsen wrote:
> > M = array(l)
> > Mt = M.transpose()
> >
> > just isn't that much worse than:
> >
> > Mt = transpose(l)
> 
> No, but the automatic conversion enables me to write functions that
> accept any sequence type without even having to think about it.

I've used that to, but I also frequently use something like this:

def function(A):
	A = array(A)
	...

Which is pretty simple to. 

> Moreover, it is almost essential in many situations to accept scalars
> in place of arrays, because scalars fulfill the role of rank-0 arrays.

Yes, this is critical. Isn't there a plan to make the scalar -- rank-0
array dicotomy a little cleaner in NumArray ?
 
> > I also agree that the point is not subclassing per se, it's
> > polymorphism. It should be easy to write a class that acts like an array
> > in all the ways that you need it to. 
> 
> True, and that is a weak point of NumPy.

Is this getting any better with NumArray?
 

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: FW: [Numpy-discussion] typecodes in numarray

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>A Dilluns 27 Gener 2003 20:52, Todd Miller va escriure:
>  
>
>>>Oh!, I didn't know that. In such a case, I think it's worth to consider
>>>the possibility to define records as classes descendants from
>>>metaclasses. But, of course, you have the ultimate decision.
>>>      
>>>
>>I don't know what you mean here.   Please spell it out a little more.
>>    
>>
>
>I was trying to mean that using something like :
>
>class Small(IsRecord):
>    field1 = defineType(CharType, 2, default="", position=1)
>    field2 = defineType(Int32, 1, position=2)
>    field3 = Float64
>
>as as container for recarray metadata is definitely possible instead of the
>tuple (formats="2aid",names=("field1","field2", "field3")), if using
>Python2.2. IsRecord is a metaclass (introduced in Python 2.2) that allows
>you to effectively separate the declared attributes from the implicit ones
>in normal classes.
>
>Of course, you can taylor IsRecord so as to fulfill your needs.
>
>I hope that I have expressed myself more clearly now,
>
>  
>
I looked at your docs here: 
http://pytables.sourceforge.net/html-doc/usersguide-html4.html#section4.2
and what you said above clicked.  Thanks.

Todd

Re: FW: [Numpy-discussion] typecodes in numarray

[Francesc A. <fa...@op...>]

A Dilluns 27 Gener 2003 20:52, Todd Miller va escriure:
> >
> >Oh!, I didn't know that. In such a case, I think it's worth to conside=
r
> > the possibility to define records as classes descendants from
> > metaclasses. But, of course, you have the ultimate decision.
>
> I don't know what you mean here.   Please spell it out a little more.

I was trying to mean that using something like :

class Small(IsRecord):
    field1 =3D defineType(CharType, 2, default=3D"", position=3D1)
    field2 =3D defineType(Int32, 1, position=3D2)
    field3 =3D Float64

as as container for recarray metadata is definitely possible instead of t=
he
tuple (formats=3D"2aid",names=3D("field1","field2", "field3")), if using
Python2.2. IsRecord is a metaclass (introduced in Python 2.2) that allows
you to effectively separate the declared attributes from the implicit one=
s
in normal classes.

Of course, you can taylor IsRecord so as to fulfill your needs.

I hope that I have expressed myself more clearly now,

--=20
Francesc Alted

Re: FW: [Numpy-discussion] typecodes in numarray

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>A Dilluns 27 Gener 2003 17:28, Todd Miller va escriure:
>  
>
>>>So, as numarray has to work with previous python versions, there is no
>>>point to care about that.
>>>      
>>>
>>In truth,   numarray-0.4 and up already require Python-2.2 and up.
>>    
>>
>
>Oh!, I didn't know that. In such a case, I think it's worth to consider the
>possibility to define records as classes descendants from metaclasses. But,
>of course, you have the ultimate decision.
>  
>
I don't know what you mean here.   Please spell it out a little more.

>  
>
>>>>I'm thinking the general format for this may be converting N-tuples of
>>>>types and ints into N arrays of types and ints.  And vice versa.
>>>>It's obvious how this works with numarray types.  I think the chararray
>>>>types need work and need to be mapped into the same integer enumeration
>>>>as the numeric types in a non-overlapping way.
>>>>        
>>>>
>>>I can't catch your point here. Why there should be a problem with
>>>chararrays?.
>>>      
>>>
>>What I was trying to see is that chararray types are not as well
>>designed as the numarray types,  nor are they reflected in the C-API.
>>    
>>
>
>I see. Well, is it really desirable such a unification? CharArray entities
>come from a module and NumArray from another one, and that should be ok. Why
>bother in creating a unified API or integer enumeration?. 
>
It may not be necessary.  Int8 with repitition factors may work about 
the same.

Re: FW: [Numpy-discussion] typecodes in numarray

[Todd M. <jm...@st...>]

Paul F Dubois wrote:

>IMHO you can assume any Python you want. Look to the long term here, not the
>short.
>
You lost me.  numarray-0.4 needs at least Python-2.2 or baseclasses 
don't exist.  I had a slow Python equivalent for the baseclass as I 
refactored prior to numarray-0.4,  but it's gone now.

>
>I'm a bit uncertain on MA as to whether my old design is right. Maybe I
>should be inheriting from NDarray? So that MA is more of a sibling of
>numarray rather than a wrapper of it?
>  
>
I asked Perry about this one.  His points (salted a little by me) were:

1. If you inherit from NumArray,  you also inherit from NDArray.  If you 
only inherit from NDArray,  all you get are the structural operations.

2. If you inherit from NumArray,  you can use Liskov substitution to 
pass MA's directly into extensions expecting NumArrays.  This 
substitution may or may not be good.  Also,  isinstance(anMA, numarray) 
will return True.  

3. If you inherit from NumArray,  you get numerical method definitions 
which may or may not be applicable to MA.  With a little thrashing,  we 
might also get MAs to work for ufuncs.   In fact, ufuncs are the key to 
whether or not the NumArray numerical methods add any value.

Todd

>  
>

Re: FW: [Numpy-discussion] typecodes in numarray

[Francesc A. <fa...@op...>]

A Dilluns 27 Gener 2003 17:28, Todd Miller va escriure:
> >So, as numarray has to work with previous python versions, there is no
> > point to care about that.
>
> In truth,   numarray-0.4 and up already require Python-2.2 and up.

Oh!, I didn't know that. In such a case, I think it's worth to consider t=
he
possibility to define records as classes descendants from metaclasses. Bu=
t,
of course, you have the ultimate decision.

> >>I'm thinking the general format for this may be converting N-tuples o=
f
> >>types and ints into N arrays of types and ints.  And vice versa.
> >>It's obvious how this works with numarray types.  I think the chararr=
ay
> >>types need work and need to be mapped into the same integer enumerati=
on
> >>as the numeric types in a non-overlapping way.
> >
> >I can't catch your point here. Why there should be a problem with
> >chararrays?.
>
> What I was trying to see is that chararray types are not as well
> designed as the numarray types,  nor are they reflected in the C-API.

I see. Well, is it really desirable such a unification? CharArray entitie=
s
come from a module and NumArray from another one, and that should be ok. =
Why
bother in creating a unified API or integer enumeration?. I think this
should be not a big drawback for C-extension crafters (although, to say t=
he
truth, that would be very elegant if you manage to do that, but maybe it =
is
not worth the effort, I don't know).

--=20
Francesc Alted

Re: FW: [Numpy-discussion] typecodes in numarray

[Todd M. <jm...@st...>]

Chris Barker wrote:

>Francesc Alted wrote:
>
>  
>
>>So, as numarray has to work with previous python versions, 
>>    
>>
>
>Why? Anyone using NumArray is either starting from scratch or porting
>from Numeric, so having to port to a newer version of Python is a very
>small deal. 
>  
>
Just to make it very clear:  numarray-0.4 and up require Python-2.2 or 
higher.  

Up until numarray-0.4 (released in November),  that was not the case, 
and numarray ran (and was tested!) on Python-2.0 and higher.

The desire to increase C-level Numeric compatability and to improve 
simple indexing speed led us to a C baseclass, which is only supported 
in Python-2.2 and  up.

Todd

Re: FW: [Numpy-discussion] typecodes in numarray

[Chris B. <Chr...@no...>]

Francesc Alted wrote:

> So, as numarray has to work with previous python versions, 

Why? Anyone using NumArray is either starting from scratch or porting
from Numeric, so having to port to a newer version of Python is a very
small deal. 


-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: FW: [Numpy-discussion] typecodes in numarray

[Francesc A. <fa...@op...>]

A Dilluns 27 Gener 2003 15:42, Todd Miller va escriure:
> Yes.  So one question is:  if we were to add type-repetition tuples to
> recarray as an alternative to the current character code strings,  woul=
d
> that be any form of improvement to recarray from your perspective?

Well, at least, charcodes can be avoided. I think it's a big win... or ma=
ybe
not as big?

>
> As I see it,  recarray currently has a clean seperation between format
> and naming which permits the latter to be optional.  Before changing
> that,  I'd need a clear argument why.  (I didn't design and generally
> don't even maintain recarray).

One argument is the fact that a map is very clear to the user, although t=
hat
such a map can be built *after* the names and format are passed to the
recarray constructor and be accessible as an atribute. However, the latte=
r
solution is worse IMO, because the user has to supply two separate pieces=
 of
information when, actually, these should be regarded as a unity. Anyway,
this maybe a subjective perception.

> >This would not be difficult to support because, by accessing to the
> >Small().__dict__, you get also a dictionary. In addition, the latter w=
ill
> >ensure (by construction) that you are not using a non-valid python
> >identifier, which is mandatory in my current implementation. I find th=
ese
> >containers (dictionaries and classes) both elegant and convenient.
>
> I'm not trying to be Mr. Negative here,  but one thing to keep in mind

Oh dear, you are right!. In fact, I forgot that to make this to work, you
need to use the metaclasses introduced in Python 2.2 (see Alex Martelli's
post: http://mail.python.org/pipermail/python-list/2002-July/112007.html)=
=2E
I was following this recipe, but I forgot that I was using Python 2.2.

So, as numarray has to work with previous python versions, there is no po=
int
to care about that.

>
> I'm sure that's not the easiest way to capture struct layout,  but I
> take your point.   Since position matters to me,  I'd prefer that
> capturing them was implicit.   Since it doesn't to you, it seems OK for
> it to be explicit.   Either default mode can support the other,  but
> capturing order with tuples is free,  while capturing order with a
> __dict__ will take some kind of extra work.

That's right. We have some different needs and priorities, and we should
take the approach better suited to each other. But exchanging points of v=
iew
is always a great thing.

>
> I'm thinking the general format for this may be converting N-tuples of
> types and ints into N arrays of types and ints.  And vice versa.
> It's obvious how this works with numarray types.  I think the chararray
> types need work and need to be mapped into the same integer enumeration
> as the numeric types in a non-overlapping way.
>

I can't catch your point here. Why there should be a problem with
chararrays?.

--=20
Francesc Alted

Re: FW: [Numpy-discussion] typecodes in numarray

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>A Dilluns 27 Gener 2003 15:42, Todd Miller va escriure:
>  
>
>>Yes.  So one question is:  if we were to add type-repetition tuples to
>>recarray as an alternative to the current character code strings,  would
>>that be any form of improvement to recarray from your perspective?
>>    
>>
>
>Well, at least, charcodes can be avoided. I think it's a big win... or maybe
>not as big?
>  
>
I think that avoiding the charcodes would be an improvement. 
 Type-repetition tuples provide a clear well defined way to define data 
formats.   It's not so clear that it eliminates the requirement for 
on-going Numeric compatability,  but it might.

>  
>
>>As I see it,  recarray currently has a clean seperation between format
>>and naming which permits the latter to be optional.  Before changing
>>that,  I'd need a clear argument why.  (I didn't design and generally
>>don't even maintain recarray).
>>    
>>
>
>One argument is the fact that a map is very clear to the user, although that
>such a map can be built *after* the names and format are passed to the
>recarray constructor and be accessible as an atribute. However, the latter
>solution is worse IMO, because the user has to supply two separate pieces of
>information when, actually, these should be regarded as a unity. Anyway,
>this maybe a subjective perception.
>  
>
Well,  I think there's truth to the danger of seperating names from data 
declarations,  but it is easy to map keys(), values() to the seperate 
pieces in a different layer if necessary.  

>This would not be difficult to support because, by accessing to the
>Small().__dict__, you get also a dictionary. In addition, the latter will
>ensure (by construction) that you are not using a non-valid python
>identifier, which is mandatory in my current implementation. I find these
>containers (dictionaries and classes) both elegant and convenient.
>  
>
>>I'm not trying to be Mr. Negative here,  but one thing to keep in mind
>>    
>>
>
>Oh dear, you are right!. 
>
For a few seconds there,  I thought I was on a roll!  

>In fact, I forgot that to make this to work, you
>need to use the metaclasses introduced in Python 2.2 (see Alex Martelli's
>post: http://mail.python.org/pipermail/python-list/2002-July/112007.html).
>I was following this recipe, but I forgot that I was using Python 2.2.
>
>So, as numarray has to work with previous python versions, there is no point
>to care about that.
>  
>
In truth,   numarray-0.4 and up already require Python-2.2 and up.

>I'm sure that's not the easiest way to capture struct layout,  but I
>take your point.   Since position matters to me,  I'd prefer that
>capturing them was implicit.   Since it doesn't to you, it seems OK for
>it to be explicit.   Either default mode can support the other,  but
>capturing order with tuples is free,  while capturing order with a
>__dict__ will take some kind of extra work.
>  
>
>
>That's right. We have some different needs and priorities, and we should
>take the approach better suited to each other. But exchanging points of view
>is always a great thing.
>
>  
>
>>I'm thinking the general format for this may be converting N-tuples of
>>types and ints into N arrays of types and ints.  And vice versa.
>>It's obvious how this works with numarray types.  I think the chararray
>>types need work and need to be mapped into the same integer enumeration
>>as the numeric types in a non-overlapping way.
>>
>>    
>>
>
>I can't catch your point here. Why there should be a problem with
>chararrays?.
>
What I was trying to see is that chararray types are not as well 
designed as the numarray types,  nor are they reflected in the C-API.

>  
>

Re: FW: [Numpy-discussion] typecodes in numarray

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>Yeah. We just differ in the way to arrange this metadata to be passed to the
>recarray constructor. But I think this is secondary compared to the
>flexibility that a verbose approach offers compared with the actual string
>format. 
>
Yes.  So one question is:  if we were to add type-repetition tuples to 
recarray as an alternative to the current character code strings,  would 
that be any form of improvement to recarray from your perspective?

As I see it,  recarray currently has a clean seperation between format 
and naming which permits the latter to be optional.  Before changing 
that,  I'd need a clear argument why.  (I didn't design and generally 
don't even maintain recarray).

>In fact, more than one container might be supported to define the
>metadata; one can start with tuples as you suggest, but in the future other
>ways can be added (if considered convenient).
>  
>
>For example, I think I'll stick with the dictionary option for PyTables, but
>also a class declaration for the metadata would be supported, like in :
>
>class Small(IsRecord):
>    var1 = defineType(CharType, 2, "")
>    var2 = defineType(Int32, 1)
>    var3 = Float64
>
>This would not be difficult to support because, by accessing to the
>Small().__dict__, you get also a dictionary. In addition, the latter will
>ensure (by construction) that you are not using a non-valid python
>identifier, which is mandatory in my current implementation. I find these
>containers (dictionaries and classes) both elegant and convenient.
>  
>
I'm not trying to be Mr. Negative here,  but one thing to keep in mind 
is this:

 >>> class C:
...     pass
...
 >>> c = C()
 >>> dir(c.__dict__)
['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__', 
'__doc__', '__eq__', '__ge__', '__getattribute__', '__getitem__', 
'__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', 
'__lt__', '__ne__', '__new__', '__reduce__', '__repr__', '__setattr__', 
'__setitem__', '__str__', 'clear', 'copy', 'fromkeys', 'get', 'has_key', 
'items', 'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 
'popitem', 'setdefault', 'update', 'values']

Which is to say,  the instance dictionary is a little cluttered,  and it 
might not be that easy to determine which objects in it are there to 
define the data format.

>>Just like the type repetition tuple except also including field names
>>and default values.   I don't think you lost me.  For what we do,  the
>>exact physical layout of the "struct" is important, so order matters.  I
>>see order as part of the
>>meta-data,  but I don't usually deal with meta-entities so maybe I've
>>got that part wrong.  :)
>>
>
>Well, if you need positional fields, you may add a (optional) parameter,
>called for example, "position" so that you can fix it. 
>  
>
I'm sure that's not the easiest way to capture struct layout,  but I 
take your point.   Since position matters to me,  I'd prefer that 
capturing them was implicit.   Since it doesn't to you, it seems OK for 
it to be explicit.   Either default mode can support the other,  but 
capturing order with tuples is free,  while capturing order with a 
__dict__ will take some kind of extra work.

>>I was thinking that if the above was an issue,  we could write an API
>>function(s) to "compile" the type-repetition tuple into arrays of ints
>>which describe the type of each field and corresponding repetition factor.
>>    
>>
>
>Yeah, I agree that this would be the best solution. That way, the charcodes
>will be factored out from the code, and by just providing such and API (both
>in Python and C), would be enough to reconstruct them, if needed. That will
>allow a more consistent numarray internal code. 
>  
>
I'm thinking the general format for this may be converting N-tuples of 
types and ints into N arrays of types and ints.  And vice versa.
It's obvious how this works with numarray types.  I think the chararray 
types need work and need to be mapped into the same integer enumeration 
as the numeric types in a non-overlapping way.

>See you Monday,
>  
>
>
>Right, how did you know that? :)
>  
>
Insightful on weekends anyway, 
Todd

Re: FW: [Numpy-discussion] typecodes in numarray

[Francesc A. <fa...@op...>]

A Dissabte 25 Gener 2003 20:30, Todd Miller va escriure:
>
> Still think I'd prefer something seperable:
>
> recarrStruct =3D (   (CharType, 16),
>                             UInt8,
>                             Int16,
>                             Int32,
>                             Int32,
>                             Float32,
>                             (Float64, 32),
>                             Int64 )
>
> recarrFields =3D ["name",
>   "TDCcount",
>   "ADCcount",
>    "grid_i",
>    "grid_j",
>    "pressure",
>    "temperature",
>    "idnumber"]
>
> I guess it might not be quite as good for large structs.

Me too...

>
> It's a good sign that both you and I thought of an identical tuple
> format; it's the obvious
> minimal one.

Yeah. We just differ in the way to arrange this metadata to be passed to =
the
recarray constructor. But I think this is secondary compared to the
flexibility that a verbose approach offers compared with the actual strin=
g
format. In fact, more than one container might be supported to define the
metadata; one can start with tuples as you suggest, but in the future oth=
er
ways can be added (if considered convenient).

For example, I think I'll stick with the dictionary option for PyTables, =
but
also a class declaration for the metadata would be supported, like in :

class Small(IsRecord):
    var1 =3D defineType(CharType, 2, "")
    var2 =3D defineType(Int32, 1)
    var3 =3D Float64

This would not be difficult to support because, by accessing to the
Small().__dict__, you get also a dictionary. In addition, the latter will
ensure (by construction) that you are not using a non-valid python
identifier, which is mandatory in my current implementation. I find these
containers (dictionaries and classes) both elegant and convenient.

>
> Just like the type repetition tuple except also including field names
> and default values.   I don't think you lost me.  For what we do,  the
> exact physical layout of the "struct" is important, so order matters.  =
I
> see order as part of the
> meta-data,  but I don't usually deal with meta-entities so maybe I've
> got that part wrong.  :)
>

Well, if you need positional fields, you may add a (optional) parameter,
called for example, "position" so that you can fix it.=20

>
> I was thinking that if the above was an issue,  we could write an API
> function(s) to "compile" the type-repetition tuple into arrays of ints
> which describe the type of each field and corresponding repetition fact=
or.

Yeah, I agree that this would be the best solution. That way, the charcod=
es
will be factored out from the code, and by just providing such and API (b=
oth
in Python and C), would be enough to reconstruct them, if needed. That wi=
ll
allow a more consistent numarray internal code.=20

>
> See you Monday,

Right, how did you know that? :)

--=20
Francesc Alted

Re: [Numpy-discussion] complex diagonal matrix

[Joachim S. <li...@js...>]

* ba...@ph... [26.01.2003 11:40]:
> I just wondered if there is a "nicer" way of generating
> a complex diagonal matrix than
>   a)
>      v=arange(10,typecode=Complex)
>      mat=diag(v)
>   b)
>      v=arange(10)
>      mat=diag(v)+0j
>
> Namely, wouldn't something like
>   v=arange(10)
>   mat=diag(v,typecode=Complex)
> be nicer?

No, because diag() is supposed to create a diagonal, but *not* to
cast to another type. If you wanted to add that "functionality" to
functions like diag(), you would also have to add it to functions
like reshape() etc., i.e. practically everywhere.

The way it is handled now is reasonably simple and flexible, and
there is really no advantage of your suggestion compared to
approach a).

Cheers,
Joachim

Re: [Numpy-discussion] complex diagonal matrix

[Konrad H. <hi...@cn...>]

ba...@ph... writes:

> I just wondered if there is a "nicer" way of generating
> a complex diagonal matrix than
>   a)
>      v=arange(10,typecode=Complex)
>      mat=diag(v)
>   b)
>      v=arange(10)
>      mat=diag(v)+0j
> 
> Namely, wouldn't something like
>   v=arange(10)
>   mat=diag(v,typecode=Complex)
> be nicer?

Why would that be nicer?

Personally, I prefer to have explicit typecodes limited to a very
small number of array generators, and have all other functions apply
the standard type-preservation rules.

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

[Numpy-discussion] complex diagonal matrix

[<ba...@ph...>]

Hi,

I just wondered if there is a "nicer" way of generating
a complex diagonal matrix than
  a)
     v=arange(10,typecode=Complex)
     mat=diag(v)
  b)
     v=arange(10)
     mat=diag(v)+0j

Namely, wouldn't something like
  v=arange(10)
  mat=diag(v,typecode=Complex)
be nicer?

BTW: I somehow found that in the (excellent) documentation
of Numeric the definitions from Mlab.py are a bit hidden.
In my case I know nothing about matlab and I somehow expected
that this type of routines are to be found in the section
(together with zeros,ones etc. etc....)
Also diag is not listed in the index
 http://www.pfdubois.com/numpy/html2/numpy-22.html#A
or ?

Arnd

Re: FW: [Numpy-discussion] typecodes in numarray

[Todd M. <jm...@st...>]

Francesc Alted wrote:

>A Divendres 24 Gener 2003 21:15, Todd Miller va escriure:
>  
>
>>>My current thinking is something like:
>>>
>>>recarrDescr = {
>>>   "name"        : defineType(CharType, 16, ""),  # 16-character String
>>>   "TDCcount"    : defineType(UInt8, 1, 0),    # unsigned byte
>>>   "ADCcount"    : defineType(Int16, 1, 0),    # signed short integer
>>>   "grid_i"      : defineType(Int32, 1, 9),    # integer
>>>   "grid_j"      : defineType(Int32, 1, 9),    # integer
>>>   "pressure"    : defineType(Float32, 1, 1.),  # float 
>>>(single-precision) "temperature" : defineType(Float64, 32, arange(32)), 
>>># double[32] "idnumber"    : defineType(Int64, 1, 0),    # signed long
>>>long }
>>>      
>>>
Still think I'd prefer something seperable:

recarrStruct = (   (CharType, 16),
                            UInt8,
                            Int16,
                            Int32,
                            Int32,
                            Float32,
                            (Float64, 32),
                            Int64 )

recarrFields = ["name",
  "TDCcount",
  "ADCcount",
   "grid_i",
   "grid_j",
   "pressure",
   "temperature",
   "idnumber"]

I guess it might not be quite as good for large structs.

>>>where defineType is a class that accepts (type, shape, default)
>>>parameters. It can be extended safely in the future if more needs appear.
>>>      
>>>
>>You're way ahead of me here.  The only thing I don't like about this is
>>the additional relative complexity because of the addition of field
>>names and default values.   It would be nice to layer this more.
>>
>>    
>>
>
>Well, I think a map between field names and values is valuable from the
>user's point of view. It may help him to label the different information on
>the recarray. Moreover, if __getattr__ and __setattr__ methods (or
>__getitem__ and __setitem__) would get implemented on recarray (as they are
>in my recarray2 version, for example), the field name can become a very
>convenient manner to access a specific field by name (this introduce the
>limitation that field name must be a valid python identifier, but I think
>this is not a big restriction). By looking at the description dictionary,
>the user can have a quick idea of what he can find in every field (with no
>need of counting, which can be a big advantage specially for long records).
>
That's true and sounds nice.  I'm just thinking records with named 
fields should be derived
from records with positional fields.  If the functionality is layered, 
 you can use as much
complexity as you need.

It's a good sign that both you and I thought of an identical tuple 
format; it's the obvious
minimal one.

>
>With regard to default values, you can make this parameter (even the shape)
>a keyword parameter in order to make it optional. 
>
OK.  That's a good point.

>  
>
>>One more thing I don't understand looking at this:  a dictionary is 
>>unordered.
>>    
>>
>
>Yeah, but this can be regarded as an advantage rather than a drawback in the
>sense that you can choose the order you (the developer) prefer. For example,
>I was using first a alphanumerical order to arrange the data fields, but
>now, I'm considering that a arrangement that optimizes the alignment of the
>fields could be far better. As for one, say that you have a (Int8, Int32,
>Float64) record; in principle it could be easy to create a routine that
>arranges this record in the form (Float64,Int32, Int8) that optimizes the
>different field access (it may be even possible to introduce automatic
>padding later on if recarrays would support them in the future).
>
>Maybe you are getting confused 
>
Yes and no. :)

>in thinking that recarrDescr will create the
>recarray. Not at all, this a *metadata* definition that can be passed to the
>actual recarray funtion for recarray creation. 
>
Just like the type repetition tuple except also including field names 
and default values.   I don't think you lost me.  For what we do,  the 
exact physical layout of the "struct" is important, so order matters.  I 
see order as part of the
meta-data,  but I don't usually deal with meta-entities so maybe I've 
got that part wrong.  :)

>Its function would be
>similar to the formats parameter (with typical values like "3a,4i,3w") in
>recarray.array, but with more verbosity and all the reported advantages.
>
>  
>
>>>instead of
>>>
>>>((Int16, 3),
>>>(Int32, 4),
>>>(Float64, 20),
>>>)
>>>      
>>>
>>This is pretty much exactly what I was thinking.  It is straightforward
>>to imagine and difficult to forget.
>>
>>    
>>
>>>the former being more handy in lots of situations.
>>>      
>>>
>>Would you please name some of these so we can explore handling them both
>>ways?
>>
>>    
>>
>
>Well, I'm afraid that the best advantage would be when dealing with
>recarrays in C extension modules. In this kind of situation it would be far
>better to deal with a "3a4i3w" array than a tuple of python objects. But
>maybe I'm wrong and the latter is not so-complicated to manage; however, I
>used to work a lot with records (even before meeting recarray) and I was
>quite comfortable with formats in string mode.
>
I was thinking that if the above was an issue,  we could write an API 
function(s) to "compile" the type-repetition tuple into arrays of ints 
which describe the type of each field and corresponding repetition factor.

>
>Or perhaps it would be enough to provide a method for converting from the
>standard metadata layout (dictionary or tuple or whatever), to a string
>format. This should be not very difficult.
>  
>
Almost exactly what I suggested above.

See you Monday,
Todd