numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] numarray unfriendly to user defined types

[Todd M. <jm...@st...>]

On Thu, 2003-09-18 at 19:18, Tim Hochberg wrote:
> 
Sorry for the delay;  I was pretty much shut down by hurricane Isabelle
from the end of last week.

I hit and bypassed this issue trying to port MA.

> I've run into another issue in my attempt to transition over to 
> numarray: it's less friendly to user defined types than Numeric. I think 
> this is mainly accidental friednliness on Numeric's part, but it's handy 
> nonetheless. The attached file illustrates the issue. Given some object, 
> in this case *zero*, that a numarray array does not now how to handle, 
> it ends up raising an exception instead of giving the object a chance to 
> try. Thus ``zero + numarray.arange(5)`` works while ``numarray.arange(5) 
> + zero`` fails, since in the first case Zero.__add__ is called first, 
> while in the second NumArray.__add__ is called first.
> 
> This should probably be fixed, but it's not clear what the best way is. 
> My first thought was to always give the other object a chance to use 
> __rop__ first. This is simple and easy to explain, but fails miserably 
> in the common case of multiplying a list and an array. Just to be 
> concrete, __mul__ would be replaced by::
> 
>     def __mul__(self, operand):
>         try:
>             return operand.__rmul__(self)
>         except:
>             return ufunc.multiply(self, operand)
> 
> Next thought is to catch exceptions when they occur in numarray and then 
> give the other operand a chance::
> 
>     def __mul__(self, operand):
>         try:
>            return ufunc.multiply(self, operand)
>         except:
>             return operand.__rmul__(self)
> 
> This appears like it would fix my particular problem, but still is not 
> ideal. Since numarray is the base of libraries that it will know nothing 
> about, it should defer to classes it doesn't know about whenever 
> possible. 

That does sound like the right heuristic and is echoed (somewhat) in the
core language. 

> Otherewise it's not possible (or maybe just hard)  to create  
> new classes that try to be clever, but still interact with numarray in a 
> reasonable way. 

Looking in Python's Objects/abstract.c, the function binop1 shows one
way to squirm around this issue:  subclass from NumArray.   It would be
interesting to know how Numeric does it.  

> In my case I'm thinking of proxy objects that don't do 
> some computations till they are actually required. So my current 
> thinking is that __mul__ and friends would be best implemented as::
> 
>     def __mul__(self, operand):
>         if not isinstance(operand, knownTypes):

If the "not" belongs there, I'm lost.

>             try:
>                   return ufunc.multiply(self, operand)
>             except:
>                 pass
>         return operand.__rmul__(self)
> 
> Where knownTypes is something like (int, long, float, complex, tuple, list).
> 
> Anyway, that's my excessively long two cents on this.
> 
I agree with your heuristic,  but I am leery of putting an unqualified
try/except in front of the numarray ufunc code.  On the other hand,
qualifying it seems too complicated.  

What about adding a tuple of types to be deferred to?

	def __mul__(self, operand):
		if isinstance(operand, _numarray_deferred_types):
			operand.__rmul__(self)
		else:
			self.__mul__(operand)

Then other libraries could register classes with something like:

numarray.defer_to(my_class)

which is slightly painful, but avoids masking the ufunc exceptions and
only needs to be done once for each library base class.  How does that
sound?

Todd

> -tim
> 
> 
> 
> 
> ----
> 

> import numarray as na
> import Numeric as np
> 
> class Zero:
>     def __add__(self, other):
>         return other
>     __radd__ = __add__
> zero = Zero()
>     
> #~ print zero + np.arange(5)
> #~ print np.arange(5) + zero
> #~ print zero + na.arange(5) 
> #~ print na.arange(5) + zero
> 
> a = na.arange(5)
> 
> import copy
> copy.deepcopy(a)
-- 
Todd Miller 			jm...@st...
STSCI / ESS / SSB

[Numpy-discussion] [ANN] PyTables 0.7.2 released

[Francesc A. <fa...@op...>]

Announcing PyTables 0.7.2
-------------------------

PyTables is a hierarchical database package designed to efficently
manage very large amounts of data. PyTables is built on top of the
HDF5 library and the numarray package. It features an object-oriented
interface that, combined with natural naming and C-code generated from
Pyrex sources, makes it a fast, yet extremely easy to use tool for
interactively save and retrieve large amounts of data. Besides, it
provides flexible indexed access on disk to anywhere in the data you
want to go.
 
On this release you will not find any exciting new features. It is
mainly a maintenance release where the next issues has been addressed:
       - a memory leak was fixed
       - memory consumption is being addressed and lowered
       - much faster opening of files
       - Some important index patterns cases in table reads has been
         optimized

More in detail:

What's new
-----------

        - Fixed a nasty memory leak located on the C libraries (it was
          happening during HDF5 attribute writes). After that, the
          memory consumption when using large object trees has dropped
          quite a bit. However, there remains some small leaks that
          has been tracked down to the underlying numarray
          library. These leaks has been reported, and hopefully they
          should be fixed more sooner than later.

        - Table buffers are built dinamically now, so if Tables are
	  not accessed for reading or writing this memory will not be
	  booked. This will help to reduce the memory consumption.

	- The opening of files with lots of nodes has been accelerated
	  between a factor 2 or 3. For example, a file with 10 groups
	  and 3000 tables that takes 9.3 seconds to open in 0.7.1, now
	  takes only 2.8 seconds.

	- The Table.read() method has been refactored and optimized
	  and some parts of its code has been moved to Pyrex. In
	  particular, in the special case of step=1, up to a factor 5
	  of speedup (reaching 160 MB/s on a Pentium4 @ 2 GHz) when
	  reading table contents can be achieved now.

	- Done some cosmetic changes in the user manual, but, as no
          new features has been added, you won't need to read the
          manual again :-)

What is a table?
----------------

A table is defined as a collection of records whose values are stored
in fixed-length fields. All records have the same structure and all
values in each field have the same data type.  The terms
"fixed-length" and "strict data types" seems to be quite a strange
requirement for an language like Python, that supports dynamic data
types, but they serve a useful function if the goal is to save very
large quantities of data (such as is generated by many scientific
applications, for example) in an efficient manner that reduces demand
on CPU time and I/O resources.

What is HDF5?
-------------

For those people who know nothing about HDF5, it is is a general
purpose library and file format for storing scientific data made at
NCSA. HDF5 can store two primary objects: datasets and groups. A
dataset is essentially a multidimensional array of data elements, and
a group is a structure for organizing objects in an HDF5 file. Using
these two basic constructs, one can create and store almost any kind of
scientific data structure, such as images, arrays of vectors, and
structured and unstructured grids. You can also mix and match them in
HDF5 files according to your needs.

Platforms
---------

I'm using Linux as the main development platform, but PyTables should
be easy to compile/install on other UNIX machines. This package has
also passed all the tests on a UltraSparc platform with Solaris 7 and
Solaris 8. It also compiles and passes all the tests on a SGI
Origin2000 with MIPS R12000 processors and running IRIX 6.5.

Regarding Windows platforms, PyTables has been tested with Windows
2000 and Windows XP, but it should also work with other flavors.

An example?
-----------

For online code examples, have a look at

http://pytables.sourceforge.net/tut/tutorial1-1.html

and 

http://pytables.sourceforge.net/tut/tutorial1-2.html

Web site
--------

Go to the PyTables web site for more details:

http://pytables.sourceforge.net/

Share your experience
---------------------

Let me know of any bugs, suggestions, gripes, kudos, etc. you may
have.

Have fun!

-- Francesc Alted

[Numpy-discussion] numarray unfriendly to user defined types

[Tim H. <tim...@ie...>]

I've run into another issue in my attempt to transition over to 
numarray: it's less friendly to user defined types than Numeric. I think 
this is mainly accidental friednliness on Numeric's part, but it's handy 
nonetheless. The attached file illustrates the issue. Given some object, 
in this case *zero*, that a numarray array does not now how to handle, 
it ends up raising an exception instead of giving the object a chance to 
try. Thus ``zero + numarray.arange(5)`` works while ``numarray.arange(5) 
+ zero`` fails, since in the first case Zero.__add__ is called first, 
while in the second NumArray.__add__ is called first.

This should probably be fixed, but it's not clear what the best way is. 
My first thought was to always give the other object a chance to use 
__rop__ first. This is simple and easy to explain, but fails miserably 
in the common case of multiplying a list and an array. Just to be 
concrete, __mul__ would be replaced by::

    def __mul__(self, operand):
        try:
            return operand.__rmul__(self)
        except:
            return ufunc.multiply(self, operand)

Next thought is to catch exceptions when they occur in numarray and then 
give the other operand a chance::

    def __mul__(self, operand):
        try:
           return ufunc.multiply(self, operand)
        except:
            return operand.__rmul__(self)

This appears like it would fix my particular problem, but still is not 
ideal. Since numarray is the base of libraries that it will know nothing 
about, it should defer to classes it doesn't know about whenever 
possible. Otherewise it's not possible (or maybe just hard)  to create  
new classes that try to be clever, but still interact with numarray in a 
reasonable way. In my case I'm thinking of proxy objects that don't do 
some computations till they are actually required. So my current 
thinking is that __mul__ and friends would be best implemented as::

    def __mul__(self, operand):
        if not isinstance(operand, knownTypes):
            try:
                  return ufunc.multiply(self, operand)
            except:
                pass
        return operand.__rmul__(self)

Where knownTypes is something like (int, long, float, complex, tuple, list).

Anyway, that's my excessively long two cents on this.

-tim

Re: [Numpy-discussion] array sort questions

[CL WU <ane...@ho...>]

Thanks again, Tim. It a wonderful example to show how efficient python 
can run if it is well written.

Best,

Chunlei

Tim Hochberg wrote:

> Hi Chunlei,
>
> I just realized one other thing that you should probably be aware of. 
> You could write a much faster version of sortrank in pure python by 
> doing your sorts differently. Python's built in sort is very fast, but 
> as soon as you start passing in comparison functions it slows down 
> dramatically. The trick is to arange the data you need to sort so that 
> you don't need an auxilliary function  (know 
> asDecorate-Sort-Undecorate or the Schwartzian transform). Thus, the 
> following is almost certainly a lot faster than your original 
> sortrank, although probably still slower than the argsort solution.
>
> def sortrank(list):
>    index = range(len(list))
>    li_a = zip(list, index)
>    li_a.sort()
>    li_b = [(li_a[i][1], i) for i in index]
>    li_b.sort()
>    return [x[1] for x in li_b]
>
> Regards,
>
> -tim
>
>
>
>
>
>
> CL WU wrote:
>
>>>> I hope there is an efficient function in array level to do the same 
>>>> work.
>>>> >>> from Numeric import *
>>>> >>> a=array([5,2,3])
>>>> >>> argsort(a)
>>>> array([1, 2, 0])
>>>> >>> def sortrank(list):
>>>> ...     n=len(list)
>>>> ...     li_a=[(i,list[i]) for i in range(n)]
>>>> ...     li_a.sort(lambda a,b:cmp(a[1],b[1]))
>>>> ...     li_b=[(i,li_a[i]) for i in range(n)]
>>>> ...     li_b.sort(lambda a,b:cmp(a[1][0],b[1][0]))
>>>> ...     return [x[0] for x in li_b]
>>>> ...    >>> sortrank(a)
>>>> [2, 0, 1]
>>>> >>> def sortrank2(li):
>>>> ...     li_sorted=li[:]
>>>> ...     li_sorted.sort()
>>>> ...     return [li_sorted.index(x) for x in li]
>>>> >>> sortrank1(list(a))
>>>> [2, 0, 1]
>>>>
>>>>
>>>> Thanks again.
>>>>
>>>> Chunlei
>>>>
>>>> Tim Hochberg wrote:
>>>>
>>>>> CL WU wrote:
>>>>>
>>>>>> Hi, group,
>>>>>>          I am new to numpy. I have 2 questions for array sort.
>>>>>>
>>>>>> 1. How to sort an array by its one column or one row?
>>>>>>    I know python build-in sort() can do it for list by passing 
>>>>>> own cmp function. but array function sort() will sort each column 
>>>>>> or row seperately,as I know. I don't want to convert array to 
>>>>>> list to sort and then convert back to array.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I think you want argsort plus take. For example, the following 
>>>>> sorts on the second column of a:
>>>>>
>>>>>     a = array([[4,5,6], [1,2,3], [7,8,9]])
>>>>>     arg = argsort(a[:,1])
>>>>>     take(a, arg, 0)
>>>>>
>>>>>> 2. How to get the rank of a rank-0 array? The first "rank" means 
>>>>>> the order of each element after sorting, instead of the 
>>>>>> "dimension" meaning in numpy.     Just like "rank()" function in 
>>>>>> splus.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> If  I understand you correctly, you want argsort as mentioned above.
>>>>>
>>>>> Regards,
>>>>>
>>>>> -tim
>>>>>
>>>>>
>>>>>>
>>>>>> Thank you
>>>>>>
>>>>>> Chunlei
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -------------------------------------------------------
>>>>>> This sf.net email is sponsored by:ThinkGeek
>>>>>> Welcome to geek heaven.
>>>>>> http://thinkgeek.com/sf
>>>>>> _______________________________________________
>>>>>> Numpy-discussion mailing list
>>>>>> Num...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------
>>>>> This sf.net email is sponsored by:ThinkGeek
>>>>> Welcome to geek heaven.
>>>>> http://thinkgeek.com/sf
>>>>> _______________________________________________
>>>>> Numpy-discussion mailing list
>>>>> Num...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------
>>>> This sf.net email is sponsored by:ThinkGeek
>>>> Welcome to geek heaven.
>>>> http://thinkgeek.com/sf
>>>> _______________________________________________
>>>> Numpy-discussion mailing list
>>>> Num...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
>

Re: [Numpy-discussion] array sort questions

[Tim H. <tim...@ie...>]

Hi Chunlei,

I just realized one other thing that you should probably be aware of. 
You could write a much faster version of sortrank in pure python by 
doing your sorts differently. Python's built in sort is very fast, but 
as soon as you start passing in comparison functions it slows down 
dramatically. The trick is to arange the data you need to sort so that 
you don't need an auxilliary function  (know asDecorate-Sort-Undecorate 
or the Schwartzian transform). Thus, the following is almost certainly a 
lot faster than your original sortrank, although probably still slower 
than the argsort solution.

def sortrank(list):
    index = range(len(list))
    li_a = zip(list, index)
    li_a.sort()
    li_b = [(li_a[i][1], i) for i in index]
    li_b.sort()
    return [x[1] for x in li_b]

Regards,

-tim






CL WU wrote:

>>> I hope there is an efficient function in array level to do the same 
>>> work.
>>> >>> from Numeric import *
>>> >>> a=array([5,2,3])
>>> >>> argsort(a)
>>> array([1, 2, 0])
>>> >>> def sortrank(list):
>>> ...     n=len(list)
>>> ...     li_a=[(i,list[i]) for i in range(n)]
>>> ...     li_a.sort(lambda a,b:cmp(a[1],b[1]))
>>> ...     li_b=[(i,li_a[i]) for i in range(n)]
>>> ...     li_b.sort(lambda a,b:cmp(a[1][0],b[1][0]))
>>> ...     return [x[0] for x in li_b]
>>> ...    >>> sortrank(a)
>>> [2, 0, 1]
>>> >>> def sortrank2(li):
>>> ...     li_sorted=li[:]
>>> ...     li_sorted.sort()
>>> ...     return [li_sorted.index(x) for x in li]
>>> >>> sortrank1(list(a))
>>> [2, 0, 1]
>>>
>>>
>>> Thanks again.
>>>
>>> Chunlei
>>>
>>> Tim Hochberg wrote:
>>>
>>>> CL WU wrote:
>>>>
>>>>> Hi, group,
>>>>>          I am new to numpy. I have 2 questions for array sort.
>>>>>
>>>>> 1. How to sort an array by its one column or one row?
>>>>>    I know python build-in sort() can do it for list by passing own 
>>>>> cmp function. but array function sort() will sort each column or 
>>>>> row seperately,as I know. I don't want to convert array to list to 
>>>>> sort and then convert back to array.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I think you want argsort plus take. For example, the following 
>>>> sorts on the second column of a:
>>>>
>>>>     a = array([[4,5,6], [1,2,3], [7,8,9]])
>>>>     arg = argsort(a[:,1])
>>>>     take(a, arg, 0)
>>>>
>>>>> 2. How to get the rank of a rank-0 array? The first "rank" means 
>>>>> the order of each element after sorting, instead of the 
>>>>> "dimension" meaning in numpy.     Just like "rank()" function in 
>>>>> splus.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> If  I understand you correctly, you want argsort as mentioned above.
>>>>
>>>> Regards,
>>>>
>>>> -tim
>>>>
>>>>
>>>>>
>>>>> Thank you
>>>>>
>>>>> Chunlei
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------------------------------
>>>>> This sf.net email is sponsored by:ThinkGeek
>>>>> Welcome to geek heaven.
>>>>> http://thinkgeek.com/sf
>>>>> _______________________________________________
>>>>> Numpy-discussion mailing list
>>>>> Num...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------
>>>> This sf.net email is sponsored by:ThinkGeek
>>>> Welcome to geek heaven.
>>>> http://thinkgeek.com/sf
>>>> _______________________________________________
>>>> Numpy-discussion mailing list
>>>> Num...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>>
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> This sf.net email is sponsored by:ThinkGeek
>>> Welcome to geek heaven.
>>> http://thinkgeek.com/sf
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Num...@li...
>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>
>>
>>
>>
>>
>
>
>

Re: [Numpy-discussion] Numarray - Numeric incompatibility

[Todd M. <jm...@st...>]

On Thu, 2003-09-18 at 13:53, Tim Hochberg wrote:
> 
> I'm just starting to move some of my code over to numarray and I was 
> dismayed to find that basic operation between Numeric and numarray 
> arrays fail.
> 
>  >>> import Numeric as np
>  >>> import numarray as na
>  >>> a = na.arange(5)
>  >>> p = np.arange(5)
>  >>> a + p
> ['vector', 'vector']
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "C:\Python23\Lib\site-packages\numarray\numarraycore.py", line 
> 648, in __add__
>     def __add__(self, operand): return ufunc.add(self, operand)
>   File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 818, in 
> _cache_miss2
>     key = (_digest(n1), _digest(n2), _digest(out), safethread.get_ident())
> KeyError: '_digest force cache miss'
> 
> I suspect (hope!) that this is just a bug and not something inherent in 
> numarray. 

It's an interoperability issue.  Please let us know if you find others.

> I dug around in unfunc.py a bit and it appears that the bug is 
> shallow and can be fixed simply by replacing::
> 
>          if not (_sequence(n1) or _sequence(n2)):
>             key = (_digest(n1), _digest(n2), _digest(out), 
> safethread.get_ident())
>             self._cache[ key ] = cached
> 
> with::
> 
>         try:
>             key = (_digest(n1), _digest(n2), _digest(out), 
> safethread.get_ident())
>         except KeyError:
>             pass
>         else:
>             self._cache[ key ] = cached
> 
> in _cache_miss2 and _cache_miss1. If this were done, _sequence could 
> probably be deleted as well.
> 
> I'm not very familiar with the numarray code yet, so it's quite possible 
> I'm missing something, but I'm willing to do more digging to fix this if 
> this turns out to not be sufficient.
> 

I ran into the same problem trying to port MA to numarray, and came up
with an identical work around.  

A fix like this will be part of numarray-0.8.

Todd
-- 
Todd Miller 			jm...@st...
STSCI / ESS / SSB

[Numpy-discussion] Numarray - Numeric incompatibility

[Tim H. <tim...@ie...>]

I'm just starting to move some of my code over to numarray and I was 
dismayed to find that basic operation between Numeric and numarray 
arrays fail.

 >>> import Numeric as np
 >>> import numarray as na
 >>> a = na.arange(5)
 >>> p = np.arange(5)
 >>> a + p
['vector', 'vector']
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\Python23\Lib\site-packages\numarray\numarraycore.py", line 
648, in __add__
    def __add__(self, operand): return ufunc.add(self, operand)
  File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 818, in 
_cache_miss2
    key = (_digest(n1), _digest(n2), _digest(out), safethread.get_ident())
KeyError: '_digest force cache miss'

I suspect (hope!) that this is just a bug and not something inherent in 
numarray. I dug around in unfunc.py a bit and it appears that the bug is 
shallow and can be fixed simply by replacing::

         if not (_sequence(n1) or _sequence(n2)):
            key = (_digest(n1), _digest(n2), _digest(out), 
safethread.get_ident())
            self._cache[ key ] = cached

with::

        try:
            key = (_digest(n1), _digest(n2), _digest(out), 
safethread.get_ident())
        except KeyError:
            pass
        else:
            self._cache[ key ] = cached

in _cache_miss2 and _cache_miss1. If this were done, _sequence could 
probably be deleted as well.

I'm not very familiar with the numarray code yet, so it's quite possible 
I'm missing something, but I'm willing to do more digging to fix this if 
this turns out to not be sufficient.

Regards,

-tim

Re: [Numpy-discussion] array sort questions

[CL WU <ane...@ho...>]

Great! It works much more efficiently. Thank you so much.

Best,

Chunlei

Tim Hochberg wrote:

> CL WU wrote:
>
>> Thank you, Tim.  argsort() and take() does provide a easy way to sort 
>> an array based on any col or row. But for the second question, it 
>> doesn't return the result I want.
>> As below, softrank or softrank1  are functions I am currently using 
>> for get the rank of a vector(first is more efficient). It returns the 
>> index of each value from original array/list in sorted array/list.
>
>
> Hmmm. It seems that argsort and sortrank are inverses of a sort, so it 
> should be possible to do what you want efficiently, but I'm not sure how.
>
> <think>
>
> Ah, it appears to be quite simple. I believe:
>
> argsort(argsort(a))
>
> is equivalent to your sortrank and should be much faster.
>
> regards,
>
> -tim
>
>
>> I hope there is an efficient function in array level to do the same 
>> work.
>> >>> from Numeric import *
>> >>> a=array([5,2,3])
>> >>> argsort(a)
>> array([1, 2, 0])
>> >>> def sortrank(list):
>> ...     n=len(list)
>> ...     li_a=[(i,list[i]) for i in range(n)]
>> ...     li_a.sort(lambda a,b:cmp(a[1],b[1]))
>> ...     li_b=[(i,li_a[i]) for i in range(n)]
>> ...     li_b.sort(lambda a,b:cmp(a[1][0],b[1][0]))
>> ...     return [x[0] for x in li_b]
>> ...    >>> sortrank(a)
>> [2, 0, 1]
>> >>> def sortrank2(li):
>> ...     li_sorted=li[:]
>> ...     li_sorted.sort()
>> ...     return [li_sorted.index(x) for x in li]
>> >>> sortrank1(list(a))
>> [2, 0, 1]
>>
>>
>> Thanks again.
>>
>> Chunlei
>>
>> Tim Hochberg wrote:
>>
>>> CL WU wrote:
>>>
>>>> Hi, group,
>>>>          I am new to numpy. I have 2 questions for array sort.
>>>>
>>>> 1. How to sort an array by its one column or one row?
>>>>    I know python build-in sort() can do it for list by passing own 
>>>> cmp function. but array function sort() will sort each column or 
>>>> row seperately,as I know. I don't want to convert array to list to 
>>>> sort and then convert back to array.
>>>
>>>
>>>
>>>
>>> I think you want argsort plus take. For example, the following sorts 
>>> on the second column of a:
>>>
>>>     a = array([[4,5,6], [1,2,3], [7,8,9]])
>>>     arg = argsort(a[:,1])
>>>     take(a, arg, 0)
>>>
>>>> 2. How to get the rank of a rank-0 array? The first "rank" means 
>>>> the order of each element after sorting, instead of the "dimension" 
>>>> meaning in numpy.     Just like "rank()" function in splus.
>>>
>>>
>>>
>>>
>>> If  I understand you correctly, you want argsort as mentioned above.
>>>
>>> Regards,
>>>
>>> -tim
>>>
>>>
>>>>
>>>> Thank you
>>>>
>>>> Chunlei
>>>>
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------
>>>> This sf.net email is sponsored by:ThinkGeek
>>>> Welcome to geek heaven.
>>>> http://thinkgeek.com/sf
>>>> _______________________________________________
>>>> Numpy-discussion mailing list
>>>> Num...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>>
>>>
>>>
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> This sf.net email is sponsored by:ThinkGeek
>>> Welcome to geek heaven.
>>> http://thinkgeek.com/sf
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Num...@li...
>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>>
>>
>>
>>
>>
>> -------------------------------------------------------
>> This sf.net email is sponsored by:ThinkGeek
>> Welcome to geek heaven.
>> http://thinkgeek.com/sf
>> _______________________________________________
>> Numpy-discussion mailing list
>> Num...@li...
>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>
>
>
>
>

Re: [Numpy-discussion] array sort questions

[CL WU <ane...@ho...>]

Thank you, Tim.  argsort() and take() does provide a easy way to sort an 
array based on any col or row. But for the second question, it doesn't 
return the result I want.
As below, softrank or softrank1  are functions I am currently using for 
get the rank of a vector(first is more efficient). It returns the index 
of each value from original array/list in sorted array/list.
I hope there is an efficient function in array level to do the same work.
 >>> from Numeric import *
 >>> a=array([5,2,3])
 >>> argsort(a)
array([1, 2, 0])
 >>> def sortrank(list):
...     n=len(list)
...     li_a=[(i,list[i]) for i in range(n)]
...     li_a.sort(lambda a,b:cmp(a[1],b[1]))
...     li_b=[(i,li_a[i]) for i in range(n)]
...     li_b.sort(lambda a,b:cmp(a[1][0],b[1][0]))
...     return [x[0] for x in li_b]
...    
 >>> sortrank(a)
[2, 0, 1]
 >>> def sortrank2(li):
...     li_sorted=li[:]
...     li_sorted.sort()
...     return [li_sorted.index(x) for x in li]
 >>> sortrank1(list(a))
[2, 0, 1]


Thanks again.

Chunlei

Tim Hochberg wrote:

> CL WU wrote:
>
>> Hi, group,
>>          I am new to numpy. I have 2 questions for array sort.
>>
>> 1. How to sort an array by its one column or one row?
>>    I know python build-in sort() can do it for list by passing own 
>> cmp function. but array function sort() will sort each column or row 
>> seperately,as I know. I don't want to convert array to list to sort 
>> and then convert back to array.
>
>
> I think you want argsort plus take. For example, the following sorts 
> on the second column of a:
>
>     a = array([[4,5,6], [1,2,3], [7,8,9]])
>     arg = argsort(a[:,1])
>     take(a, arg, 0)
>
>> 2. How to get the rank of a rank-0 array? The first "rank" means the 
>> order of each element after sorting, instead of the "dimension" 
>> meaning in numpy.     Just like "rank()" function in splus.
>
>
> If  I understand you correctly, you want argsort as mentioned above.
>
> Regards,
>
> -tim
>
>
>>
>> Thank you
>>
>> Chunlei
>>
>>
>>
>>
>> -------------------------------------------------------
>> This sf.net email is sponsored by:ThinkGeek
>> Welcome to geek heaven.
>> http://thinkgeek.com/sf
>> _______________________________________________
>> Numpy-discussion mailing list
>> Num...@li...
>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>>
>
>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>

Re: [Numpy-discussion] array sort questions

[Tim H. <tim...@ie...>]

CL WU wrote:

> Hi, group,
>          I am new to numpy. I have 2 questions for array sort.
>
> 1. How to sort an array by its one column or one row?
>    I know python build-in sort() can do it for list by passing own cmp 
> function. but array function sort() will sort each column or row 
> seperately,as I know. I don't want to convert array to list to sort 
> and then convert back to array.

I think you want argsort plus take. For example, the following sorts on 
the second column of a:

     a = array([[4,5,6], [1,2,3], [7,8,9]])
     arg = argsort(a[:,1])
     take(a, arg, 0)

> 2. How to get the rank of a rank-0 array? The first "rank" means the 
> order of each element after sorting, instead of the "dimension" 
> meaning in numpy.     Just like "rank()" function in splus.

If  I understand you correctly, you want argsort as mentioned above.

Regards,

-tim


>
> Thank you
>
> Chunlei
>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>

[Numpy-discussion] array sort questions

[CL WU <ane...@ho...>]

Hi, group,
          I am new to numpy. I have 2 questions for array sort.

1. How to sort an array by its one column or one row?
    I know python build-in sort() can do it for list by passing own cmp 
function. but array function sort() will sort each column or row 
seperately,as I know. I don't want to convert array to list to sort and 
then convert back to array.

2. How to get the rank of a rank-0 array? The first "rank" means the 
order of each element after sorting, instead of the "dimension" meaning 
in numpy.     Just like "rank()" function in splus.

Thank you

Chunlei

Re: [Numpy-discussion] numarray latest version

[Todd M. <jm...@st...>]

On Wed, 2003-09-17 at 07:05, Nadav Horesh wrote:
> How comes that numarray 0.6.2 succeeds 0.7? 

numarray-0.6.2 is numarray-0.6.1 which makes the licensing clearer at
runtime by including it in __init__.py for numarray.  That's it.  No new
(real) code, just licensing,  which is basically the modified BSD
license.    It's not really intended for use outside the astronomical
community,  so I didn't announce it.

numarray-0.6.2 is being used by STSCI in a larger system, and it has
been tested in that context.  

numarray-0.7 is newer, better, and with several bug fixes,  but not the
same level of testing.

It is unfortunate that numarray-0.6.2 is listed first on Source Forge, 
but as far as I know I can't control that.

> Are there 2 active branches?

Not for long.

Todd

>   Nadav.
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 
Todd Miller 			jm...@st...
STSCI / ESS / SSB

[Numpy-discussion] numarray latest version

[Nadav H. <na...@Vi...>]

How comes that numarray 0.6.2 succeeds 0.7? Are there 2 active branches?

  Nadav.

Re: [Numpy-discussion] Re: MacPython and Numeric

[Andrea R. <ari...@pi...>]

Fink project (see <http://fink.sourceforge.net/>) offers Numeric  
already linked against Apple's vecLib. Here is the output about that  
package:

> [ibook:~/Installs/IPython] andrea% fink describe numeric-py22
> Information about 2973 packages read in 5 seconds.
>
>
> numeric-py22-23.0-1: Multidim'l numerical arrays for Python, using  
> Apple vecLib
>   Links against optimized Apple vecLib BLAS/LAPACK, instead of
>   using slower, built-in blas and lapack routines.
>   Should now be just as fast as numeric-atlas.
>  .
>  Web site: http://www.pfdubois.com/numpy
>  .
>  Maintainer: Jeffrey Whitaker <js...@cd...>

I think you can ask the maintainer for more information or look at the  
Fink's installation script if you know how debs system works. Here it  
is a link to the scripts:

<http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/fink/dists/10.2/ 
unstable/main/finkinfo/sci/numeric-py22-23.0-1.info>
<http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/fink/dists/10.2/ 
unstable/main/finkinfo/sci/numeric-py22-23.0-1.patch>

Hope this help.

Cheers,
	Andrea.

On Monday, Sep 15, 2003, at 10:36 Europe/Rome, Jack Jansen wrote:

> Folks,
> I got this report on Numeric as included with MacPython (through the  
> package manager) and it looks
> promising. But: I'm building Numeric simply by running setup.py, so in  
> order to get the added performance
> Numeric's setup.py would need to be taught about veclib.
>
> Is anyone willing to look into this?
>
---
Andrea Riciputi

"Science is like sex: sometimes something useful comes out,
   but that is not the reason we are doing it" -- (Richard Feynman)

[Numpy-discussion] Re: MacPython and Numeric

[Jack J. <Jac...@cw...>]

Folks,
I got this report on Numeric as included with MacPython (through the 
package manager) and it looks
promising. But: I'm building Numeric simply by running setup.py, so in 
order to get the added performance
Numeric's setup.py would need to be taught about veclib.

Is anyone willing to look into this?

On Sunday, September 14, 2003, at 09:35 PM, Alun Carr wrote:

> Jack,
>
> I would like to thank you for all your hard work in producing 
> MacPython 2.3 for OS X. It's superb.
>
> One point to note: I think that you may have compiled the Numeric 
> stuff from scratch. Apple supply hand-tuned BLAS, LAPACK, and FFT code 
> in their vecLib framework. Despite the name, according to Apple's 
> documentation this framework is suitable for both G3 and G4 
> processors, and it will automatically use vectorized code if it 
> detects a G4.
>
> From <http://developer.apple.com/hardware/ve/quickstart.html>:
>
>> If you are a developer who would like to make more use of AltiVec in 
>> your application, you should first explore the functionality that is 
>> available in Apple's vecLib.framework . This framework can be found 
>> in /System/Library/Frameworks on any MacOS X installation. There you 
>> will find many highly optimized routines for a diversity of tasks. 
>> There are single and double precision FFT's. Some of these are up to 
>> five times faster than FFT-W . There are also convolutions, 1D and 2D 
>> matrix operations and much, much more. MacOS X.2 (Jaguar) rounds this 
>> out with a full BLAS and LAPACK implementation in both single and 
>> double precision. These functions also work on G3 and earlier 
>> processors, meaning that you can call them safely regardless of what 
>> machine you are running on.
>
> For future releases of Numeric for MacPython, it might be worth using 
> vecLib, to (i) reduce the effort on your part, (ii) reduce the size of 
> the distribution, and (iii) to use code which has been optimized by 
> Apple for maximum performance.
>
> Anyway, thanks again for your efforts, and please keep up the good 
> work.
>
> Best regards,
>
> Alun
>
> ==
>
> Dr Alun J. Carr                Phone: +353-(0)1-716 1989 (Office)
> Mechanical Engineering Dept           +353-(0)1-716 1752 (BRC)
> University College Dublin      Fax:   +353-(0)1-283 0534
> Belfield                       Email: alu...@uc... (UCD)
> Dublin 4                              ca...@at... (Home)
> Ireland
>
--
Jack Jansen, <Jac...@cw...>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma 
Goldman

Re: [Numpy-discussion] Boolean array formatting poll

[Robert K. <ke...@ca...>]

On Tue, Sep 09, 2003 at 04:15:36PM -0400, Todd Miller wrote:
> Now that Python has a bool type, I have received a patch for numarray's
> arrayprint module which alters the formatting of Boolean arrays from:
> 
> >>>> a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a
> array([0, 1, 0, 1, 0, 1], type=Bool)
> 
> to:
> 
> >>>> a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a
> array([False, True, False, True, False, True], type=Bool)
> 
> Which format do you prefer?   Please vote for or against changing the
> format:  +1 (for)  or -1 (against).

-1

If I'm using those kinds of values in arrays, I'm more likely to be
thinking in terms of 1's and 0's not True's and False's anyways.

E.g.

frequency = float(sum(a)) / len(a)

-- 
Robert Kern
ke...@ca...

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

Re: [Numpy-discussion] Boolean array formatting poll

[Chris B. <Chr...@no...>]

Todd Miller wrote:

> >>> numarray.ones((10,), type=numarray.Bool)*5
> array([5, 5, 5, 5, 5, 5, 5, 5, 5, 5])
> 
> So, yes.

In that case, definately -1: if the data really are integers, that might
as well be clear from the display

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Why stripping spaces in strings?

[Francesc A. <fa...@op...>]

Yups, RawCharrArray do the trick:

>>> strings.array([["hello world  ","pp\0"]], kind=strings.CharArray)
CharArray([['hello world', 'pp']])
>>> strings.array([["hello world  ","pp\0"]], kind=strings.RawCharArray)
RawCharArray([['hello world  ', 'pp\x00          ']])

Thanks for the hint,

A Dimecres 10 Setembre 2003 14:40, Todd Miller va escriure:
> Take a look at RawCharArray instead of CharArray.  CharArray is a
> RawCharArray with specific stripping and padding behaviors and some
> extra methods.  RawCharArray should be the starting point for this kind
> of discussion;  it may do what you want as-is, or you can tell me where
> it's falling short of what you need.
>
> Todd

-- 
Francesc Alted

Re: [Numpy-discussion] Boolean array formatting poll

[Todd M. <jm...@st...>]

<snip>

> 
> For what I mean, note in Python 2.3:
> 
> >>> a = True
> >>> a
> True
> >>> a * 5
> 5
> 
> 
> Can you do this in NumArray?

The numarray analog is:

>>> numarray.ones((10,), type=numarray.Bool)*5
array([5, 5, 5, 5, 5, 5, 5, 5, 5, 5])

So, yes.


-- 
Todd Miller 			jm...@st...
STSCI / ESS / SSB

Re: [Numpy-discussion] Boolean array formatting poll

[Fernando P. <fp...@co...>]

Pearu Peterson wrote:
> Considering the number of `-1`s, how about
> 
> 
>>>>a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a
> 
>  array([F, T, F, T, F, T], type=Bool)

Somehow I seem to parse faster 0/1 arrays than F/T (the symbols are visually 
more different).

-0.5, but better than the change to True/False.

Cheers,

f

Re: [Numpy-discussion] Boolean array formatting poll

[Chris B. <Chr...@no...>]

Pearu Peterson wrote:

> >>> a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a
>  array([F, T, F, T, F, T], type=Bool)

That's better. However, I"m still inclined to prefer [1, 0, 0, 1, ....

Someone mentioned making clear the distiction between bool and int
arrays. HOw distict are they? In Python, the bool type is really just a
pretty wrapper around an Int anway. I like it, but if in NumArray, a
bool is really and int in disguise, I'd be just as happy to ahve it be
0, 1, 0, .... If however, a bool really is a totally different beast (or
will be in a future version), then maybe it's worth making the clear
distiction.

-Chris

For what I mean, note in Python 2.3:

>>> a = True
>>> a
True
>>> a * 5
5


Can you do this in NumArray?




-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] Boolean array formatting poll

[Michiel J. L. de H. <md...@im...>]

-1, for readability.

1's and 0's can be distinguished more easily visually than T's and F's.

Michiel, U Tokyo.

Pearu Peterson wrote:
> Considering the number of `-1`s, how about
> 
> 
>>>>a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a
> 
>  array([F, T, F, T, F, T], type=Bool)
> 
> which is short and readable.
> 
> Pearu
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
> 
> 

-- 
Michiel de Hoon
Assistant Professor
University of Tokyo, Institute of Medical Science
Human Genome Center
4-6-1 Shirokane-dai, Minato-ku
Tokyo 108-8639
Japan
http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

Re: [Numpy-discussion] Why stripping spaces in strings?

[Todd M. <jm...@st...>]

Take a look at RawCharArray instead of CharArray.  CharArray is a
RawCharArray with specific stripping and padding behaviors and some
extra methods.  RawCharArray should be the starting point for this kind
of discussion;  it may do what you want as-is, or you can tell me where
it's falling short of what you need.

Todd

On Wed, 2003-09-10 at 05:17, Francesc Alted wrote:
> Hi,
> 
> A user of pytables recently asked me why it truncates the trailing spaces on
> strings. I have to explain him that the information is there, but the
> strings module of the numarray package deliberately strips off the trailing
> spaces when printing:
> 
> >>> from numarray import strings
> >>> a=strings.array("hello world  ")
> >>> print a
> ['hello world']
> >>> print "<%s>" % a
> <['hello world']>
> >>> len(a[0])
> 11
> 
> however:
> 
> >>> print "<%s>" % a._data
> <hello world  >
> >>> a._byteView()
> array([[104, 101, 108, 108, 111,  32, 119, 111, 114, 108, 100,  32,  32]], 
> type=UInt8)
> >>> a._itemsize
> 13
> 
> In the beginning I thought that this was a nice feature, but when people is
> using numarray.strings objects as containers (for example for Pickled
> objects), that can be rather confusing, and I would prefer to see all the
> trailing blanks.
> 
> But as I am not completely sure, I just would like to discuss the pro's and
> con's of the current printing apprach.
> 
> Cheers,
> 
> -- 
> Francesc Alted
> 
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 
Todd Miller 			jm...@st...
STSCI / ESS / SSB

Re: [Numpy-discussion] Boolean array formatting poll

[Paul B. <ba...@st...>]

+1 for consistency and clarity.

Todd Miller wrote:
> Now that Python has a bool type, I have received a patch for numarray's
> arrayprint module which alters the formatting of Boolean arrays from:
> 
> 
>>>>>a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a
> 
> array([0, 1, 0, 1, 0, 1], type=Bool)
> 
> to:
> 
> 
>>>>>a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a
> 
> array([False, True, False, True, False, True], type=Bool)
> 
> Which format do you prefer?   Please vote for or against changing the
> format:  +1 (for)  or -1 (against).
> 


-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Group
FAX:   410-338-4767    Baltimore, MD 21218

Re: [Numpy-discussion] Boolean array formatting poll

[Colin J. W. <cj...@sy...>]

Chris Barker wrote:

>Todd Miller wrote:
>
>  
>
>>>>>>a=numarray.arange(6, type=numarray.Bool); a[::2] = 0; a
>>>>>>            
>>>>>>
>>array([False, True, False, True, False, True], type=Bool)
>>
>>Which format do you prefer?   Please vote for or against changing the
>>format:  +1 (for)  or -1 (against).
>>    
>>
>
>-1
>
>It just makes it harder to read, particularly for a large array
>
>-Chris
>
Perhaps the important thing is to facilitate visual comparison between 
one row and another or between one array and another.  Maybe 'False' and 
'True ' (with a trailing space) could be used as representations of the 
Boolean values.

If a compressed representation is desired then 'F' and 'T' might be used.

This could be specified in a site constant which, for the distribution, 
would default to one of ('0, '1'), ('False', 'True'), ('False', 'True ') 
or ('F', 'T').

My feeling is that, since it can easily be done, it is desirable to 
distinguish between Integer and Boolean values.

Colin W.

>
>
>  
>