numpy-discussion

[Numpy-discussion] Freezing Numeric Python code problem under Linux

[Nor P. <npi...@es...>]

Hi,

	i am having troubles freezing some python code with call the Numeric
17.1.1 module.
	Here is an example of what happens under Linux Red Hat 6.2 and Python
2.0:

$ cat t.py 
import Numeric
a = Numeric.ones([10,10])
b = a * 10
print b

$ python ~/Python-2.0/Tools/freeze/freeze.py -o tt t.py
...
$ cd tt
$ ./t
Traceback (most recent call last):
  File "t.py", line 1, in ?
  File "/scisoft/python/lib/python2.0/site-packages/Numeric/Numeric.py",
line 79, in ?
    import multiarray
ImportError:
/scisoft/python/lib/python2.0/site-packages/Numeric/multiarray.so:
undefined symbol: _Py_NoneStruct


	The very same code, under Solaris 2.6 and Python 2.0 works just fine.
	Code which does not use the Numeric package freeze just fine under
Linux, so I think that this point to some problem/incompatibility of
Numeric with freeze.py.
	Does anybody have a suggestion, or a work-around??


		Nor

[Numpy-discussion] Initialization of array?

[Daehyok S. <sd...@em...>]

When I initialize an array, I use
a = ones(shape)*initial_val

But, I am wondering if Numpy has more efficient way.
For example,
a = array(initial_value, shape)

Peter

Re: [Numpy-discussion] Initialization of array?

[Chris B. <cb...@jp...>]

Daehyok Shin wrote:
> When I initialize an array, I use
> a = ones(shape)*initial_val
> 
> But, I am wondering if Numpy has more efficient way.
> For example,
> a = array(initial_value, shape)

I don't know if it's any more efficient (what you have is pretty fast
already), but another option is to use resize:

>>> shape = (3,4)

>>> initial_val = 5.0

>>> resize(initial_val,shape)

array([[ 5.,  5.,  5.,  5.],
       [ 5.,  5.,  5.,  5.],
       [ 5.,  5.,  5.,  5.]])
>>>           

-Chris


-- 
Christopher Barker,
Ph.D.                                                           
cb...@jp...                      ---           ---           ---
http://www.jps.net/cbarker          -----@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Water Resources Engineering       ------   @    ------   @   ------   @
Coastal and Fluvial Hydrodynamics -------      ---------     --------    
------------------------------------------------------------------------
------------------------------------------------------------------------

Re: [Numpy-discussion] Initialization of array?

[Daehyok S. <sd...@em...>]

Comparing the performance,
resize() seems to be more effient than ones().

Daehyok Shin

----- Original Message ----- 
From: "Chris Barker" <cb...@jp...>
Cc: <num...@li...>
Sent: Wednesday, November 29, 2000 11:13 AM
Subject: Re: [Numpy-discussion] Initialization of array?


> Daehyok Shin wrote:
> > When I initialize an array, I use
> > a = ones(shape)*initial_val
> > 
> > But, I am wondering if Numpy has more efficient way.
> > For example,
> > a = array(initial_value, shape)
> 
> I don't know if it's any more efficient (what you have is pretty fast
> already), but another option is to use resize:
> 
> >>> shape = (3,4)
> 
> >>> initial_val = 5.0
> 
> >>> resize(initial_val,shape)
> 
> array([[ 5.,  5.,  5.,  5.],
>        [ 5.,  5.,  5.,  5.],
>        [ 5.,  5.,  5.,  5.]])
> >>>           
> 
> -Chris
> 
> 
> -- 
> Christopher Barker,
> Ph.D.                                                           
> cb...@jp...                      ---           ---           ---
> http://www.jps.net/cbarker          -----@@       -----@@       -----@@
>                                    ------@@@     ------@@@     ------@@@
> Water Resources Engineering       ------   @    ------   @   ------   @
> Coastal and Fluvial Hydrodynamics -------      ---------     --------    
> ------------------------------------------------------------------------
> ------------------------------------------------------------------------
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> http://lists.sourceforge.net/mailman/listinfo/numpy-discussion
> 

Re: [Numpy-discussion] Initialization of array?

[<ro...@ho...>]

>>>>> "DS" == Daehyok Shin <sd...@em...> writes:

 DS> When I initialize an array, I use a = ones(shape)*initial_val

 DS> But, I am wondering if Numpy has more efficient way. For example,
 DS> a = array(initial_value, shape)

Looking at the definition of "ones":

def ones(shape, typecode='l', savespace=0):
    """ones(shape, typecode=Int, savespace=0) returns an array of the given
    dimensions which is initialized to all ones. 
    """
    return zeros(shape, typecode, savespace)+array(1, typecode)

It looks like you could try a=zeros(shape)+initial_val instead.

Hm.. I might do some experimenting.

Rob

-- 
=====   ro...@ho...          http://www.hooft.net/people/rob/  =====
=====   R&D, Nonius BV, Delft  http://www.nonius.nl/             =====
===== PGPid 0xFA19277D ========================== Use Linux! =========

Re: [Numpy-discussion] Initialization of array?

[Daehyok S. <sd...@em...>]

Initialization on huge arrays is frequent operations in scientific
programming.
It must be efficient as much as possible.
So, I was surprisized to see the inner codes of ones() in Numpy.
It maybe use calloc() rather than malloc() in C level,
then for(..) for addition.
Why not use malloc() and for(...) simultaneously in C level
with a command such as:

a = arrray(1,shape=(10000,10000))

Daehyok Shin

----- Original Message -----
From: "Rob W. W. Hooft" <ro...@ho...>
To: "Daehyok Shin" <sd...@em...>
Cc: <num...@li...>
Sent: Wednesday, November 29, 2000 1:00 PM
Subject: Re: [Numpy-discussion] Initialization of array?


> >>>>> "DS" == Daehyok Shin <sd...@em...> writes:
>
>  DS> When I initialize an array, I use a = ones(shape)*initial_val
>
>  DS> But, I am wondering if Numpy has more efficient way. For example,
>  DS> a = array(initial_value, shape)
>
> Looking at the definition of "ones":
>
> def ones(shape, typecode='l', savespace=0):
>     """ones(shape, typecode=Int, savespace=0) returns an array of the
given
>     dimensions which is initialized to all ones.
>     """
>     return zeros(shape, typecode, savespace)+array(1, typecode)
>
> It looks like you could try a=zeros(shape)+initial_val instead.
>
> Hm.. I might do some experimenting.
>
> Rob
>
> --
> =====   ro...@ho...          http://www.hooft.net/people/rob/  =====
> =====   R&D, Nonius BV, Delft  http://www.nonius.nl/             =====
> ===== PGPid 0xFA19277D ========================== Use Linux! =========
>

Re: [Numpy-discussion] Initialization of array?

[<ro...@ho...>]

I had another look at the definition of "ones" and of another routine
I frequently use: arange. It appears that even without rewriting them
in C, some speedup can be achieved:

 - in ones(), the + 1 should be done "in place", saving about 15%, more
   if you run out of processor cache:

    amigo[167]~%3% /usr/local/bin/python test_ones.py
    Numeric.ones             10 ->    0.098ms
    Numeric.ones            100 ->    0.103ms
    Numeric.ones           1000 ->    0.147ms
    Numeric.ones          10000 ->    0.830ms
    Numeric.ones         100000 ->   11.900ms
    Numeric.zeros            10 ->    0.021ms
    Numeric.zeros           100 ->    0.022ms
    Numeric.zeros          1000 ->    0.026ms
    Numeric.zeros         10000 ->    0.290ms
    Numeric.zeros        100000 ->    4.000ms
    Add inplace              10 ->    0.091ms
    Add inplace             100 ->    0.094ms
    Add inplace            1000 ->    0.127ms
    Add inplace           10000 ->    0.690ms
    Add inplace          100000 ->    8.100ms
    Reshape 1                10 ->    0.320ms
    Reshape 1               100 ->    0.436ms
    Reshape 1              1000 ->    1.553ms
    Reshape 1             10000 ->   12.910ms
    Reshape 1            100000 ->  141.200ms
    
   Also notice that zeros() is 4-5 times faster than ones(), so
   it may pay to reimplement ones in C as well (it is used in
   indices() and arange()). 

   The "resize 1" alternative is much slower.

 - in arange, additional 10% can be saved by adding brackets
   around (start+(stop-stop)) (in addition to the gain by the
   faster "ones"):

    amigo[168]~%3% /usr/local/bin/python test_arange.py
    Numeric.arange           10 ->    0.390ms
    Numeric.arange          100 ->    0.410ms
    Numeric.arange         1000 ->    0.670ms
    Numeric.arange        10000 ->    4.100ms
    Numeric.arange       100000 ->   59.000ms
    Optimized                10 ->    0.340ms
    Optimized               100 ->    0.360ms
    Optimized              1000 ->    0.580ms
    Optimized             10000 ->    3.500ms
    Optimized            100000 ->   48.000ms


Regards,

Rob Hooft

Re: [Numpy-discussion] Initialization of array?

[Konrad H. <hi...@cn...>]

> I had another look at the definition of "ones" and of another routine
> I frequently use: arange. It appears that even without rewriting them
> in C, some speedup can be achieved:
> 
>  - in ones(), the + 1 should be done "in place", saving about 15%, more
>    if you run out of processor cache:

I'd also try assignment in place:

def ones(shape, typecode='l', savespace=0):
   a = zeros(shape, typecode, savespace)
   a[len(shape)*[slice(0, None)]] = 1
   return a

Konrad.
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: hi...@cn...
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

Re: [Numpy-discussion] Initialization of array?

[<ro...@ho...>]

 KH> I had another look at the definition of "ones" and of another routine
 KH> I frequently use: arange. It appears that even without rewriting them
 KH> in C, some speedup can be achieved:
 KH> 
 KH> - in ones(), the + 1 should be done "in place", saving about 15%, more
 KH> if you run out of processor cache:

 KH> I'd also try assignment in place:

 KH> def ones(shape, typecode='l', savespace=0):
 KH>    a = zeros(shape, typecode, savespace)
 KH>    a[len(shape)*[slice(0, None)]] = 1
 KH>    return a

This is even faster, but it is better to write "a[...] = 1", because
your manual calculation of "..." gives a large overhead for small arrays.

On another machine this time:
    Numeric.ones             10 ->    0.254ms
    Numeric.ones            100 ->    0.268ms
    Numeric.ones           1000 ->    0.340ms
    Numeric.ones          10000 ->    1.960ms
    Numeric.ones         100000 ->   29.300ms
    Numeric.zeros            10 ->    0.055ms
    Numeric.zeros           100 ->    0.059ms
    Numeric.zeros          1000 ->    0.068ms
    Numeric.zeros         10000 ->    0.430ms
    Numeric.zeros        100000 ->    9.800ms
    Add inplace              10 ->    0.246ms
    Add inplace             100 ->    0.255ms
    Add inplace            1000 ->    0.312ms
    Add inplace           10000 ->    1.270ms
    Add inplace          100000 ->   18.100ms
    Assign inplace           10 ->    0.192ms
    Assign inplace          100 ->    0.201ms
    Assign inplace         1000 ->    0.242ms
    Assign inplace        10000 ->    1.010ms
    Assign inplace       100000 ->   16.300ms
    Reshape 1                10 ->    0.842ms
    Reshape 1               100 ->    1.175ms
    Reshape 1              1000 ->    4.100ms
    Reshape 1             10000 ->   35.100ms
    Reshape 1            100000 ->  368.600ms

Rob

-- 
=====   ro...@ho...          http://www.hooft.net/people/rob/  =====
=====   R&D, Nonius BV, Delft  http://www.nonius.nl/             =====
===== PGPid 0xFA19277D ========================== Use Linux! =========