numpy-discussion

[Numpy-discussion] Numpy Benchmarking

[Dennis V. P. <dv...@MI...>]

Hi, all.

I've run some benchmarks comparing the performance of scipy, numpy, 
Numeric and numarray vs. MATLAB. There's also the beginnings of a 
benchmark framework included. The results are online at:

     http://web.mit.edu/jonas/www/bench/

They were produced on a Thinkpad T42 with an Intel Pentium M 1.7GHz 
processor running Ubuntu Dapper Drake (6.06). All the languages/packages 
were built from source, and, in the case of numpy and scipy, linked to 
ATLAS. Each datapoint represents the arithmetic mean of ten trials.

The results have some interesting implications. For example, numpy and 
scipy perform approximately the same except when it comes to matrix 
inversion, MATLAB beats out all the Python packages when it comes to 
matrix addition, and numpy seems to be beaten by its predecessors in some 
cases. Why is this the case? What are some other, additional benchmarks I 
could try?

Dennis V. Perepelitsa
MIT Class of 2008, Course VIII and XVIII-C
Picower Institute for Learning and Memory

Re: [Numpy-discussion] Numpy Benchmarking

[Robert K. <rob...@gm...>]

Dennis V. Perepelitsa wrote:
> Hi, all.
> 
> I've run some benchmarks comparing the performance of scipy, numpy, 
> Numeric and numarray vs. MATLAB. There's also the beginnings of a 
> benchmark framework included. The results are online at:
> 
>      http://web.mit.edu/jonas/www/bench/
> 
> They were produced on a Thinkpad T42 with an Intel Pentium M 1.7GHz 
> processor running Ubuntu Dapper Drake (6.06). All the languages/packages 
> were built from source, and, in the case of numpy and scipy, linked to 
> ATLAS. Each datapoint represents the arithmetic mean of ten trials.

I have two suggestions based on a two-second glance at this:

1) Use time.time() on UNIX and time.clock() on Windows. The usual snippet of 
code I use for this:

import sys
import time
if sys.platform == 'win32':
     now = time.clock
else:
     now = time.time

t1 = now()
...
t2 = now()

2) Never take the mean of repeated time trials. Take the minimum if you need to 
summarize a set of trials.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Re: [Numpy-discussion] Numpy Benchmarking

[Keith G. <kwg...@gm...>]

On 6/27/06, Dennis V. Perepelitsa <dv...@mi...> wrote:

> I've run some benchmarks comparing the performance of scipy, numpy,
> Numeric and numarray vs. MATLAB.

I enjoyed looking at the results.

The most interesting result, for me, was that inverting a matrix is
much faster in scipy than numpy. How can that be? I would have guessed
that numpy handled the inversion for scipy since numpy is the core.

The two calls were scipy.linalg.inv(m) and numpy.linalg.inv(m).

Re: [Numpy-discussion] Numpy Benchmarking

[Travis O. <oli...@ie...>]

Keith Goodman wrote:
> On 6/27/06, Dennis V. Perepelitsa <dv...@mi...> wrote:
>
>   
>> I've run some benchmarks comparing the performance of scipy, numpy,
>> Numeric and numarray vs. MATLAB.
>>     
>
> I enjoyed looking at the results.
>
> The most interesting result, for me, was that inverting a matrix is
> much faster in scipy than numpy. How can that be? I would have guessed
> that numpy handled the inversion for scipy since numpy is the core.
>
> The two calls were scipy.linalg.inv(m) and numpy.linalg.inv(m).
>   
NumPy uses Numeric's old wrapper to lapack algorithms. 

SciPy uses it's own f2py-generated wrapper (it doesn't rely on the NumPy 
wrapper).

The numpy.dual library exists so you can use the SciPy calls if the 
person has SciPy installed or the NumPy ones otherwise.  It exists 
precisely for the purpose of seamlessly taking advantage of 
algorithms/interfaces that exist in NumPy but  are improved in SciPy.


-Travis

Re: [Numpy-discussion] Numpy Benchmarking

[Keith G. <kwg...@gm...>]

On 6/27/06, Travis Oliphant <oli...@ie...> wrote:

> The numpy.dual library exists so you can use the SciPy calls if the
> person has SciPy installed or the NumPy ones otherwise.  It exists
> precisely for the purpose of seamlessly taking advantage of
> algorithms/interfaces that exist in NumPy but  are improved in SciPy.

That sounds very interesting. It would make a great addition to the
scipy performance page:

http://scipy.org/PerformanceTips

So if I need any of the following functions I should import them from
scipy or from numpy.dual? And all of them are faster?

fft
ifft
fftn
ifftn
fft2
ifft2
norm
inv
svd
solve
det
eig
eigvals
eigh
eigvalsh
lstsq
pinv
cholesky

http://svn.scipy.org/svn/numpy/trunk/numpy/dual.py

Re: [Numpy-discussion] Numpy Benchmarking

[Keith G. <kwg...@gm...>]

On 6/27/06, Keith Goodman <kwg...@gm...> wrote:
> On 6/27/06, Travis Oliphant <oli...@ie...> wrote:
>
> > The numpy.dual library exists so you can use the SciPy calls if the
> > person has SciPy installed or the NumPy ones otherwise.  It exists
> > precisely for the purpose of seamlessly taking advantage of
> > algorithms/interfaces that exist in NumPy but  are improved in SciPy.
>
> That sounds very interesting. It would make a great addition to the
> scipy performance page:
>
> http://scipy.org/PerformanceTips
>
> So if I need any of the following functions I should import them from
> scipy or from numpy.dual? And all of them are faster?
>
> fft
> ifft
> fftn
> ifftn
> fft2
> ifft2
> norm
> inv
> svd
> solve
> det
> eig
> eigvals
> eigh
> eigvalsh
> lstsq
> pinv
> cholesky
>
> http://svn.scipy.org/svn/numpy/trunk/numpy/dual.py
>

Scipy computes the inverse of a matrix faster than numpy (except if
the dimensions of x are small). But scipy is slower than numpy for
eigh (I only checked for symmetric positive definite matrices):

from numpy import asmatrix, randn
from numpy.linalg import eigh as Neigh
from scipy.linalg import eigh as Seigh
import time

def test(N):

  x = asmatrix(randn(N,2*N))
  x = x * x.T

  t0 = time.time()
  eigval, eigvec = Neigh(x)
  t1 = time.time()

  t2 = time.time()
  eigval, eigvec = Seigh(x)
  t3 = time.time()

  print 'NumPy:', t1-t0, 'seconds'
  print 'SciPy:', t3-t2, 'seconds'

>> dual.test(10)
NumPy: 0.000217914581299 seconds
SciPy: 0.000226020812988 seconds

>> dual.test(100)
NumPy: 0.0123109817505 seconds
SciPy: 0.0321230888367 seconds

>> dual.test(200)
NumPy: 0.0793058872223 seconds
SciPy: 0.082535982132 seconds

>> dual.test(500)
NumPy: 0.59161400795 seconds
SciPy: 1.41600894928 seconds

Re: [Numpy-discussion] Numpy Benchmarking

[Robert K. <rob...@gm...>]

Keith Goodman wrote:
> Scipy computes the inverse of a matrix faster than numpy (except if
> the dimensions of x are small). But scipy is slower than numpy for
> eigh (I only checked for symmetric positive definite matrices):

Looks like scipy uses *SYEV and numpy uses the better *SYEVD (the D stands for 
divide-and-conquer) routine. Both should probably be using the RRR versions 
(*SYEVR) if I'm reading the advice in the LUG correctly.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Re: [Numpy-discussion] Numpy Benchmarking

[Travis O. <oli...@ie...>]

Dennis V. Perepelitsa wrote:
> Hi, all.
>
> I've run some benchmarks comparing the performance of scipy, numpy, 
> Numeric and numarray vs. MATLAB. There's also the beginnings of a 
> benchmark framework included. The results are online at:
>
>      http://web.mit.edu/jonas/www/bench/
>
> They were produced on a Thinkpad T42 with an Intel Pentium M 1.7GHz 
> processor running Ubuntu Dapper Drake (6.06). All the languages/packages 
> were built from source, and, in the case of numpy and scipy, linked to 
> ATLAS. Each datapoint represents the arithmetic mean of ten trials.
>   
I agree with Robert that a minimum would be a better way to aggregate 
results.

> The results have some interesting implications. For example, numpy and 
> scipy perform approximately the same except when it comes to matrix 
> inversion, MATLAB beats out all the Python packages when it comes to 
> matrix addition, and numpy seems to be beaten by its predecessors in some 
> cases. Why is this the case? 
In terms of creating zeros matrices, you are creating double-precision 
matrices for NumPy but only single-precision for Numeric and numarray.

Try using numpy.float32 or 'f' when creating numpy arrays.  The float is 
the Python type-object and represents a double-precision number.   Or, 
if you are trying to use double precision for all cases (say for 
comparison to MATLAB) then use 'd' in numarray and Numeric.

For comparing numpy with numarray and Numeric there are some benchmarks 
in the SVN tree of NumPy under benchmarks. 

These benchmarks have been helpful in the past in pointing out areas 
where we could improve the code of NumPy, so I'm grateful for your efforts.

-Travis

Re: [Numpy-discussion] Numpy Benchmarking

[<je...@fy...>]

Dennis V. Perepelitsa wrote:

>Hi, all.
>
>I've run some benchmarks comparing the performance of scipy, numpy,=20
>Numeric and numarray vs. MATLAB. There's also the beginnings of a=20
>benchmark framework included. The results are online at:
>
>     http://web.mit.edu/jonas/www/bench/
> =20
>

It's a little hard to see the curves for small matrix size, N.  How=20
about plotting the time divided by the theoretical number of operations=20
- which would be N^2 or N^3.

Jens J=F8rgen

Re: [Numpy-discussion] Numpy Benchmarking

[Filip W. <fi...@ft...>]

Jens wrote:

> Dennis V. Perepelitsa wrote:

>>Hi, all.
>>
>>I've run some benchmarks comparing the performance of scipy, numpy, 
>>Numeric and numarray vs. MATLAB. There's also the beginnings of a 
>>benchmark framework included. The results are online at:
>>
>>     http://web.mit.edu/jonas/www/bench/
>>  
>>

> It's a little hard to see the curves for small matrix size, N.  How 
> about plotting the time divided by the theoretical number of operations
> - which would be N^2 or N^3.

Or use some logarithmic scale (one or both axis) where applicable.

fw