numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] r_, c_, hstack, and vstack with 1-d arrays

[Bill B. <wb...@gm...>]

I looked into the various concatenation methods a bit more to better
understand what's going on under the hood.

Here's essentially what these different methods do:

vstack(tup):
    concatenate( map(atleast_2d,tup), axis=0 )

hstack(tup):
    concatenate( map(atleast_1d,tup),axis=1 )

column_stack(tup):
    arrays = map( transpose,map(atleast_2d,tup) )
    concatenate(arrays,1)

(note that column_stack transposes *everything* not just 1-d inputs,
so it doesn't do quite what I thought it did, i.e. only transposing
1-d inputs)

The above 3 are pretty much exactly the code used by numpy.  That's
all there is to those 3 functions.
For r_ and c_ I'm summarizing, but effectively they seem to be doing
something like:

r_[args]:
    concatenate( map(atleast_1d,args),axis=0 )

c_[args]:
    concatenate( map(atleast_1d,args),axis=1 )

c_ behaves almost exactly like hstack -- with the addition of range
literals being allowed.

r_ is most like vstack, but a little different since it effectively
uses atleast_1d, instead of atleast_2d.  So you have
>>> numpy.vstack((1,2,3,4))
array([[1],
       [2],
       [3],
       [4]])
but
>>> numpy.r_[1,2,3,4]
array([1, 2, 3, 4])

However for cases like that with just 0-d or 1-d inputs, c_ behaves
identically to r_, so if you wanted to get a 1-d output you could have
just used c_.

So I take back what I said about wishing c_ were like column_stack.
Column stack is weird.
Instead, I think the right thing to do would be to make r_ behave more
like vstack.  I think that would make things more consistent, and make
for less for the user to remember.

After making that change, to make things even more consistent, it
might make sense to rename r_ and c_  to v_ and h_ instead.  Then it's
easy to remember  'v_' is like 'vstack',  'h_' is like hstack.

Furthermore, I propose that column_stack should only transpose its 1d
inputs.  "Stack colums" defnitely doesn't imply to me that something
that already has columns will be transposed.  Currently it is
documented to only work on 1d inputs, so hopefully that's a change
that wouldn't affect too many people.  The function in
numpy/lib/shape_base.py could be replaced with this:

def column_stack(tup):
    def transpose_1d(array):
         if array.ndim<2: return _nx.transpose(atleast_2d(array))
         else: return array
    arrays = map(transpose_1d,map(atleast_1d,tup))
    return _nx.concatenate(arrays,1)

If r_, and c_ get renamed to v_, h_, then c_ could be re-introduced
with behavior similar to column_stack.

Finally, I noticed that the atleast_nd methods return arrays
regardless of input type.  At a minimum, atleast_1d and atleast_2d on
matrices should return matrices.  I'm not sure about atleast_3d, since
matrices can't be 3d.  (But my opinon is that the matrix type should
be allowed to be 3d).  Anyway, since these methods are used by the
*stack methods, those also do not currently preserve the matrix type
(in SVN numpy).

SUMMARY:
* make r_ behave like "vstack plus range literals"
* make column_stack only transpose its 1d inputs.
* rename r_,c_ to v_,h_ (or something else) to make their connection
with vstack and hstack clearer.  Maybe vs_ and hs_ would be better?
* make a new vertsion of 'c_' that acts like column_stack so that
theres a nice parallel v_<=>vstack,  h_<=>hstack, c_<=>column_stack
* make atleast_*d methods preserve the input type whenever possible

Thoughts?
--bb

Re: [Numpy-discussion] [SciPy-user] Would like to simplify my 3 where statements

[David G. <dav...@gm...>]

On 7/20/06, Michael Sorich <mic...@gm...> wrote:
> Can you give an specific example of how this would work? The codes
> really is ugly and it is not clear to me what exactly it does.

ok here's a quick example:

import numpy
n=5
i=3
j=4
A=numpy.random.randint(0,2,(n,n)) #make random graph
A=A-diag(diag(A)) #remove diagonal
A=triu(A)+transpose(triu(A)) #make symmetric

Now say A is the adjacency matrix for a graph and I want to know which
nodes are neighbours of A, but I want to exclude node i from
consideration. So if A is:

array([[0, 1, 0, 1, 0],
       [1, 0, 1, 0, 1],
       [1, 0, 0, 0, 0],
       [0, 0, 1, 0, 1],
       [1, 1, 0, 1, 0]])

the neighbours are array([1]) for i=3, j=4.

One way to do it is to do:

ans=where(A[:,j]==1)[0]
if A[i,j] == 1:
    ans -= 1

which is probably faster than my method. I don't really care so much
about speed though. I'm just trying to figure out different ways of
doing this using numpy.

Dave

Re: [Numpy-discussion] fast way of doing "cross-multiplications" ?

[Tim H. <tim...@co...>]

Fernando Perez wrote:
> On 7/18/06, Tim Hochberg <tim...@co...> wrote:
>> Eric Emsellem wrote:
>> > thanks for the tips. (indeed your "add.reduce" is correct: I just 
>> wrote
>> > this down too quickly, in the script I have a "sum" included).
>> >
>> > And yes you are right for the memory issue, so I may just keep the 
>> loop
>> > in and try to make it work on a fast PC...(or use parallel processes)
>> >
>> > (is "sum" different than "add.reduce"?)
>> >
>> > thanks again to both Bill Baxter and Perry Greenfield for their fast
>> > (and helpful!) answers.
>> >
>> I just wanted to add that there are faster, but considerably complicated
>> ways to attack this class of problems. The one I've looked at in the
>> past was the fast multipole method and I believe there are others. I'm
>> not sure whether these can be implemented efficiently in numpy, but you
>> may want to take a look into this kind of more sophisticated/complicated
>> approach if brute forcing the calculation doesn't work.
>
> Indeed, FMM is the best known method that can turn this O(n^2) problem
> into O(n*log(n)).  As Tim indicates, there is no easy way out of this
> one.  Incidentally, there is a talk about FMM on the scipy'06
> schedule, in case you are going to attend.
>
> An alternative approach to FMM (conceptually similar in some ways) is
> described here:
>
> http://amath.colorado.edu/faculty/fperez/talks/0604_sanum_mwadap.pdf
>
> Unfortunately this isn't exactly a half-hour optimization effort, as
> the required machinery is pretty heavy duty.  And yes, this has all
> been done in python and runs on current numpy/scipy (though it has
> Fortran, C and C++ sprinkled as needed).
I'm interested in hearing in what situations people end up dropping into 
C/C++ or Fortran. It would be interesting to see if general solutions 
could be found for making some of them fast enough at the Python level. 
Here are some situations that I've run into where numpy has trouble 
computing results efficiently.

   1. The value at a given point is the computed one of two (or more)
      ways depending on some condition. The subcomputations are
      expensive, so one does not want to compute them both and then
      chose a result using 'where'.  I think I have a decent way to
      attack this at the Python level using argsort: basically, sort all
      of the operands, find the boundaries between the regions where
      different computations are done, compute the expensive
      subcomputations on the appropriate blocks and then unsort the
      data. It would be interesting to see if this could be automated in
      a clean way, perhaps within numexpr.
   2. Generalized reduce. We want to reduce an array using an arbitrary
      function f(x, y) that is not a standard ufunc. In the particular
      case I've run into, x and y would be vectors or perhaps record
      arrays, and the functions would couple the various records when
      producing the output.  I don't see how this case can be handled
      efficiently at the Python level, short of using something like
      Weave to produce a custom ufunc and calling reduce on it. I don't
      see how something like numexpr could handle it, unfortunately,
      since it gains it's speed by operating on blocks of data.

I'm sure there are other cases that I've run into, but that's all I'm 
coming up with right now.

-tim

Re: [Numpy-discussion] NumPy 1.0b1 at the door...

[Travis O. <oli...@ie...>]

Stefan van der Walt wrote:
> Hi Travis
>
> Albert and I are busy doing some final beta testing.  Quick question:
> can Fortran-order arrays be contiguous, ever?
>
> In [62]: N.empty((3,3),order='F').flags['CONTIGUOUS']
> Out[62]: False
>   

Thank you very much for the testing.  You two have been amazingly helpful.

No.  CONTIGUOUS means "C-CONTIGUOUS"

Now, 0-d and 1-d arrays can be both Fortran-contiguous *and* 
C-contiguous which is why "FNC" exists to test against that case.


-Travis

Re: [Numpy-discussion] Beta release on Thursday

[Nick F. <nv...@MI...>]

On Jul 20, 2006, at 6:00 PM, Travis Oliphant wrote:
>> Is it possible to incorporate v7 mat-file support before the new- 
>> feature freeze?
> That is in SciPy.  I'm talking about the NumPy 1.0beta release.    
> But, I would like to get the v7 mat-file support into SciPy soon.
> Perhaps you are arguing about putting in mat-file support into  
> NumPy.   While I appreciate the idea, I like it sitting on the  
> SciPy side of the fence
>
>>   I have the fixes posted to ticket #14.  Several people have  
>> contacted me about the ticket.
> Thanks for the ticket.  I will get around to it.  I've been swamped  
> with NumPy.  I'm really relying on the other contributors for SciPy  
> fixes.

My apologies.  I often forget that the release cycles are not phase- 
locked.  The same characters play on both stages, so if I'm not  
paying sufficient attention, I mix things up.

I'll go ask for a commit in SciPy land.

Take care,
Nick

[Numpy-discussion] Stacking matrices

[Tim H. <tim...@co...>]

The recent message by Ferenc.Pintye (how does one pronounce that BTW) 
reminded me of something I've been meaning to discuss: I think we can do 
a better job dealing with stacked matrices. By stacked matrices I mean 3 
(or more) dimensional arrays where the last two dimensions are 
considered to be matrices. Let us suppose that one wants to find the 
eigenvalues of 200,000 3x3 matrices as Ferenc does. At present you have 
to call linalg.eigvals 200,000 time; the overhead is going to kill you. 
I've run into this problem before myself in the past (and kludged my way 
around it), so the problem is not limited to Ferenc.

I see no reason that this could not be fixed for the linalg functions. 
The details would vary from function to function, but for eigvals, for 
example, when passed an NxMxM array, it would return an NxM array of  
eigenvalues. One can actually get pretty far just modifying the python 
functions in linalg.py, or at least one could in numarray and I'm 
guessing that numpy is similar. That is because there is a fair amount 
of overhead in setting up the function calls and this can all be done 
just once. I have code lying around that does this for solve, inverse 
and determinant that works with numarray that would probably not be hard 
to adapt to numpy if we were interested in making this change of 
interface to linalg.

Another place that this same effect has bitten me in the past is with 
dot. Dot does work on higher dimensional arrays, but it attempts to do a 
tensor product of sorts. While this sounds nifty, in practice I've found 
it less than useful. However, because dot computes both scalar and 
matrix products it's not clear that a set of rules that was more or less 
backwards compatible and that extended to higher dimensions in a more 
useful way than at present could be crafted. Probably something with 
different behavior, perhaps that always performed matrix products on the 
last two axes, could be added with a name other than dot.  This avoids 
backwards compatibility issues as well.

None of this stuff is worth holding up the the release 1.0 over, but I 
though I would mention it while it was on my mind.

-tim

Re: [Numpy-discussion] Beta release on Thursday

[Travis O. <oli...@ie...>]

Nick Fotopoulos wrote:
> On July 17, 2006, at 9:01 PM, Travis Oliphant wrote:
> > I'd like to make release 1.0beta on Thursday.  Please submit 
> bug-reports
> > and fixes before then.
> >
> > -Travis
>
> Is it possible to incorporate v7 mat-file support before the 
> new-feature freeze?
That is in SciPy.  I'm talking about the NumPy 1.0beta release.   But, I 
would like to get the v7 mat-file support into SciPy soon.  

Perhaps you are arguing about putting in mat-file support into NumPy.   
While I appreciate the idea, I like it sitting on the SciPy side of the 
fence

>   I have the fixes posted to ticket #14.  Several people have 
> contacted me about the ticket.
Thanks for the ticket.  I will get around to it.  I've been swamped with 
NumPy.  I'm really relying on the other contributors for SciPy fixes.

-Travis

Re: [Numpy-discussion] NumPy 1.0b1 at the door...

[Robert K. <rob...@gm...>]

Travis Oliphant wrote:
> I'm getting ready to tag the trunk as the 1.0b1 release (will be done in 
> 2 hours unless I hear from somebody).    I've taken care of most of the 
> relevant open tickets.  Some of the build issues are still there but 
> they can be fixed during the beta release period which I expect to be 
> several months.

I'm testing the change from using pkgload in __init__ to explicit imports now. 
If you can hold off until I finish with that, I'd appreciate it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[Numpy-discussion] NumPy 1.0b1 at the door...

[Travis O. <oli...@ie...>]

I'm getting ready to tag the trunk as the 1.0b1 release (will be done in 
2 hours unless I hear from somebody).    I've taken care of most of the 
relevant open tickets.  Some of the build issues are still there but 
they can be fixed during the beta release period which I expect to be 
several months.


-Travis

Re: [Numpy-discussion] Beta release on Thursday

[Nick F. <nv...@MI...>]

On July 17, 2006, at 9:01 PM, Travis Oliphant wrote:
 > I'd like to make release 1.0beta on Thursday.  Please submit bug- 
reports
 > and fixes before then.
 >
 > -Travis

Is it possible to incorporate v7 mat-file support before the new- 
feature freeze?  I have the fixes posted to ticket #14.  Several  
people have contacted me about the ticket.

Thanks,
Nick

Re: [Numpy-discussion] vector to tensor matrix speed up

[Tim H. <tim...@co...>]

Fer...@eu... wrote:
>
>
> Hi users,
>
>       i have some problem in Numpy with indexing speed for array to tensor
> matrix transport.
> With 200000 cycles it's 9sec ! (without sort(eigvals() - functions)
>
> Many thanks
> f.
>   

The big problem you have here is that you are operating on your matrices 
one element at a time. Overhead is going to kill you doing this. The key 
thing is to operate on your data in chunks -- I'll show one way below 
that speeds thing up by a factor of 40 or so *neglecting 
sort(eigvals())*. Unfortunately, sort(eigvals()) is still likely to kill 
you since there's not really a good way to chunk that up. (I'll have 
more to say about that in a more general
sense in another email). Here's the two versions I compared, perhaps 
this will help you out:


    from numpy import *
    import timeit
    out = zeros((200000,11),float)
   
    def f0(out):
        for j in arange(0,200000):
            pass
   
    def f1(out):
        # stress values/matrix in array form for 200000 points
        #
        #.....out = filling matrix ...etc...
        #
        # stress tensor matrix
        eig = zeros((3,3),float)
        #
        #output for eigvalues
        eigwert = array([0,0,0])
        #
        for j in arange(0,200000):
              eig[0,0] = out[j,1]
              eig[1,1] = out[j,2]
              eig[2,2] = out[j,3]
              #
              eig[0,1] = out[j,4]
              eig[0,2] = out[j,6]
              eig[1,0] = out[j,4]
              eig[1,2] = out[j,5]
              eig[2,0] = out[j,6]
              eig[2,1] = out[j,5]
              #
              #~ eigwert = sort(eigvals(eig))
              out[j,7] = eigwert[2]
              out[j,8] = eigwert[1]
              out[j,9] = eigwert[0]
              out[j,10] = abs(eigwert[0]-eigwert[2])
        #
    def f2(out):
        # stress values/matrix in array form for 200000 points
       
        #
        #.....out = filling matrix ...etc...
        #
        # stress tensor matrix
   
        eig = zeros((100,3,3),float)
        #
        #output for eigvalues
        eigwert = zeros([100,3], dtype=float)
        #
        local_abs = abs
        rangen = range(0,200000,100)
        for j in rangen:
              eig[:,0,0] = out[j:j+100,1]
              eig[:,1,1] = out[j:j+100,2]
              eig[:,2,2] = out[j:j+100,3]
              #
              eig[:,1,0] = eig[:,0,1] = out[j:j+100,4]
              eig[:,2,0] = eig[:,0,2] = out[j:j+100,6]
              eig[:,2,1] = eig[:,1,2] = out[j:j+100,5]
              #
              #~ eigwert = sort(eigvals(eig))
        for j in rangen:
              out[j:j+100,7:10] = eigwert # Changed order of out here
              out[j:j+100,10] = abs(eigwert[:,0]-eigwert[:,2])
       
    if __name__ == '__main__':
        print timeit.Timer("f0(out)", "from scratch import f0, 
out").timeit(1)
        print timeit.Timer("f1(out)", "from scratch import f1, 
out").timeit(1)
        print timeit.Timer("f2(out)", "from scratch import f2, 
out").timeit(1)
>
> # stress values/matrix in array form for 200000 points
> out = zeros((200000,11),Float32)
> #
> #.....out = filling matrix ...etc...
> #
> # stress tensor matrix
> eig = zeros((3,3),Float32)
> #
> #output for eigvalues
> eigwert = array([0,0,0])
> #
> for j in arange(0,200000):
>       eig[0,0] = out[j,1]
>       eig[1,1] = out[j,2]
>       eig[2,2] = out[j,3]
>       #
>       eig[0,1] = out[j,4]
>       eig[0,2] = out[j,6]
>       eig[1,0] = out[j,4]
>       eig[1,2] = out[j,5]
>       eig[2,0] = out[j,6]
>       eig[2,1] = out[j,5]
>       #
>       eigwert = sort(eigvals(eig))
>       out[j,7] = eigwert[2]
>       out[j,8] = eigwert[1]
>       out[j,9] = eigwert[0]
>       out[j,10] = abs(eigwert[0]-eigwert[2])
> #
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>   

Re: [Numpy-discussion] numpy.var and numpy.std

[Travis O. <oli...@ie...>]

Tom Denniston wrote:
> Are these functions supposed to compute variance and standard
> deviation?  They don't have docstrings and I can't find anything about
> them in Travis' manual.
>
>   
I just wanted to advertise the fact that there are two utility functions 
in NumPy that can come in handy for people wanting to add docstrings to 
built-in types.

These utility functions by-pass the need to fiddle with doc-strings in 
C-code (which can be a pain) and re-compile every time a doc-string is 
changed.

These are meant to be run once....

They are:

add_docstring(obj, string)

Add a string as the docstring of obj.  A wide-variety of builtin types 
are understood.  If it doesn't know how to add it, you get a TypeError.  
If the object already has a doc-string you get a RuntimeError. 


add_newdoc(place, obj, doc)

This is a wrapper around docstring to make it easier to add docs to many 
attributes of an object which is stored in place.  It never fails.

place and obj are both strings
doc is a string, list, or tuple

place is the module name
obj is the object in that module

If doc is a string  add it to place.obj

If doc is a tuple it is (attr, doc)  ---> the first element is the 
attribute of obj and the second is the docstring.

If doc is a list it is [(attr1, doc1), (attr2, doc2), ...]  --> the 
elements of the list are all tuples to add docstrings to many attributes 
of an object.



numpy/add_newdocs.py

is a place where docstrings can be added.


-Travis

Re: [Numpy-discussion] numpy.var and numpy.std

[Travis O. <oli...@ie...>]

Tom Denniston wrote:
> Are these functions supposed to compute variance and standard
> deviation?  They don't have docstrings and I can't find anything about
> them in Travis' manual.
>   
These are in the manual but only as a general reference to the fact that 
array methods also have functional counter-parts.  In other words,

a.meth() --> meth(a) 

The latter is more general because a does not have to be an array already.


-Travis

Re: [Numpy-discussion] numpy.var and numpy.std

[Tom D. <tom...@al...>]

Sounds great.  Thanks David.

--Tom

On 7/20/06, David Huard <dav...@gm...> wrote:
> Yes,
>
> In SVN, std has a docstring and I submitted one for var in ticket #174.
>
> David
>
> 2006/7/20, Tom Denniston <tom...@al...
> >:
> >
> Are these functions supposed to compute variance and standard
> deviation?  They don't have docstrings and I can't find anything about
> them in Travis' manual.
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>

Re: [Numpy-discussion] numpy.var and numpy.std

[David H. <dav...@gm...>]

Yes,

In SVN, std has a docstring and I submitted one for var in ticket #174.

David

2006/7/20, Tom Denniston <tom...@al...>:
>
> Are these functions supposed to compute variance and standard
> deviation?  They don't have docstrings and I can't find anything about
> them in Travis' manual.
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>

[Numpy-discussion] numpy.var and numpy.std

[Tom D. <tom...@al...>]

Are these functions supposed to compute variance and standard
deviation?  They don't have docstrings and I can't find anything about
them in Travis' manual.

[Numpy-discussion] vector to tensor matrix speed up

[<Fer...@eu...>]

Hi users,

      i have some problem in Numpy with indexing speed for array to tensor
matrix transport.
With 200000 cycles it's 9sec ! (without sort(eigvals() - functions)

Many thanks
f.


# stress values/matrix in array form for 200000 points
out = zeros((200000,11),Float32)
#
#.....out = filling matrix ...etc...
#
# stress tensor matrix
eig = zeros((3,3),Float32)
#
#output for eigvalues
eigwert = array([0,0,0])
#
for j in arange(0,200000):
      eig[0,0] = out[j,1]
      eig[1,1] = out[j,2]
      eig[2,2] = out[j,3]
      #
      eig[0,1] = out[j,4]
      eig[0,2] = out[j,6]
      eig[1,0] = out[j,4]
      eig[1,2] = out[j,5]
      eig[2,0] = out[j,6]
      eig[2,1] = out[j,5]
      #
      eigwert = sort(eigvals(eig))
      out[j,7] = eigwert[2]
      out[j,8] = eigwert[1]
      out[j,9] = eigwert[0]
      out[j,10] = abs(eigwert[0]-eigwert[2])
#

Re: [Numpy-discussion] Pretty printing an array: no ellipses?

[Stefan v. d. W. <st...@su...>]

On Wed, Jul 19, 2006 at 03:06:00PM -0700, Webb Sprague wrote:
> I am not sure where to look for this, sorry if it is RTFM or JPS
> ("just plain stupid"):
>=20
> Is there a way to set a default to print the entire array, rather than
> an ellipses version of it?  If not, why doesn't
> pprint.pformat(numpy.random.normal(0,1,(100, 100)), width=3D1000) at
> least give me something off the screen?

Try

N.set_printoptions(threshold=3DN.nan)

St=E9fan

Re: [Numpy-discussion] Profiling line-by-line

[Arnd B. <arn...@we...>]

On Wed, 19 Jul 2006, David Grant wrote:

> Is there any way to do line-by-line profiling in Python? The profiling
> results can tell me how much time is spent in all functions, but within a
> given function I can't get any idea of how much time was spent on each line.
> For example, in the example below, I can see that graphWidth.py is taking
> all the time, but there are many lines of code in graphWidth.py that aren't
> function calls, and I have no way of knowing which lines are the
> bottlenecks. I'm using hotspot currently, by the way.
>
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>         1    0.215    0.215    0.221    0.221 graphWidth.py:6(graphWidth)
>        27    0.001    0.000    0.003    0.000 oldnumeric.py:472(all)
>        26    0.002    0.000    0.002    0.000 oldnumeric.py:410(sum)
>        26    0.001    0.000    0.002    0.000 oldnumeric.py:163(_wrapit)
>        26    0.001    0.000    0.001    0.000 oldnumeric.py:283(argmin)
>        26    0.000    0.000    0.000    0.000 numeric.py:111(asarray)
>         0    0.000             0.000          profile:0(profiler)
>
> Thanks,

You might give hotshot2kcachegrind a try.
See
http://mail.enthought.com/pipermail/enthought-dev/2006-January/001075.html
for more details.
The screenshots
http://www.physik.tu-dresden.de/~baecker/tmp/bench_traits/kcachegrind_screenshot.png
http://kcachegrind.sourceforge.net/cgi-bin/show.cgi/KcacheGrindShot4
might give you an idea how things will look.

More importantly note that profiling in connection
with ufuncs  seems problematic:
See this thread (unfortunately split into several pieces,
not sure if I got all of them):
http://thread.gmane.org/gmane.comp.python.numeric.general/5309/focus=5309
http://thread.gmane.org/gmane.comp.python.numeric.general/5311/focus=5316
http://thread.gmane.org/gmane.comp.python.numeric.general/5337/focus=5337

(I always wanted to write this up for the wiki, but "real work"
is interfering too strongly at the moment ;-).

Good luck with profiling,

Arnd

[Numpy-discussion] Profiling line-by-line

[David G. <dav...@gm...>]

Is there any way to do line-by-line profiling in Python? The profiling
results can tell me how much time is spent in all functions, but within a
given function I can't get any idea of how much time was spent on each line.
For example, in the example below, I can see that graphWidth.py is taking
all the time, but there are many lines of code in graphWidth.py that aren't
function calls, and I have no way of knowing which lines are the
bottlenecks. I'm using hotspot currently, by the way.

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.215    0.215    0.221    0.221 graphWidth.py:6(graphWidth)
       27    0.001    0.000    0.003    0.000 oldnumeric.py:472(all)
       26    0.002    0.000    0.002    0.000 oldnumeric.py:410(sum)
       26    0.001    0.000    0.002    0.000 oldnumeric.py:163(_wrapit)
       26    0.001    0.000    0.001    0.000 oldnumeric.py:283(argmin)
       26    0.000    0.000    0.000    0.000 numeric.py:111(asarray)
        0    0.000             0.000          profile:0(profiler)

Thanks,
-- 
David Grant

Re: [Numpy-discussion] Atlas and dual core

[Stephen W. <ste...@cs...>]

Keith Goodman wrote:

>Is there much speed to be gained by compiling atlas for a dual core system?
>  
>
I imagine the answer is yes.  However, Clint Whaley literally just today 
announced the first release of a beta version of ATLAS whose 
configuration utility actually supports Core Duo, although probably only 
under OSX.  Check Clint's message at

http://sourceforge.net/mailarchive/forum.php?thread_id=25989825&forum_id=426

By the way, everyone should also read the immediately previous message, 
posted a few weeks earlier by Clint.  gcc 4.x gives significantly worse 
floating point performance than gcc 3.x on most x86 platforms.  
Something to be aware of if you're benchmarking.  He's communicated 
about the problem to the gcc developers, but is, ah, less than confident 
they'll fix the problem.

Steve

[Numpy-discussion] =?GB2312?B?cmU61MvTw0lFvLzK9czhuN/Qp8LKvqvS5sn6svo=?=

[<xa...@16...>]

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<title>无标题文档</title>
<style type="text/css">
<!--
.td {
	font-size: 12px;
	color: #313131;
	line-height: 20px;
	font-family: "Arial", "Helvetica", "sans-serif";
}
-->
</style>
</head>

<body leftmargin="0" background="http://bo.sohu.com//images/img20040502/dj_bg.gif">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
  <tr>
    <td height="31" background="http://igame.sina.com.cn/club/images/topmenu/topMenu_8.gif" class="td"><div align="center"><font color="#FFFFFF">主办单位：易腾企业管理咨询有限公司</font></div></td>
  </tr>
</table>
<br>
<table width="673" border="0" align="center" cellpadding="0" cellspacing="0">
  <tr>
    <td height="62" bgcolor="#8C8C8C"> 
      <div align="center"> 
        <table width="100%" border="0" cellspacing="1" cellpadding="0" height="69">
          <tr>
            <td height="67" bgcolor="#F3F3F3"><div align="center">
                <span lang="zh-cn"><font size="6" color="#FF0000"><b>运用IE技术提高效率精益生产</b></font></span></div></td>
          </tr>
        </table>
      </div></td>
  </tr>
</table>

<table width="673" border="0" align="center" cellpadding="0" cellspacing="0" class="td" height="1411">
  <tr>
    <td height="1415" bgcolor="#FFFFFF"> <div align="center"> 
        <table width="99%" border="0" cellspacing="0" cellpadding="0">
          <tr> 
            <td width="17%" height="20" bgcolor="#BF0000" class="td"> 
              <div align="center"><font color="#FFFFFF">[课 程 背 景]</font></div></td>         
            <td width="83%" class="td">　</td>
          </tr>
          <tr> 
            <td height="74" colspan="2" class="td">
              <p ALIGN="JUSTIFY"><font LANG="ZH-CN">
              &nbsp;&nbsp;&nbsp;</font><font lang="ZH-CN" size="2"> </font><span lang="zh-cn"><font size="2">
				&nbsp;  
				企业均希望将资源充分利用，以达到效率高、交期准、浪费少、成本低的目的，这正是百年来IE技术迅猛发展的原因。IE的知识和方法在美日等发达国家已经成为普遍化的常识和理念，通过取消、合并、重排、简化等优化手法不断改进操作流程，在高工资高福利的背景下仍然凭借高效率取得了竞争性成本。<br>
              &nbsp;&nbsp; 培训对象：董事长总经理、营运副总、生技部门经理、IE部门经理、生产控制及物流经理、采购部经理、产品设计及工艺师、车间主任、现场主管、优秀班组长...</font></span></td> 
          </tr>
        </table>
      </div>
      <div align="center" style="width: 671; height: 1"> </div>
      <div align="center"> 
        <table width="99%" height="84" border="0" cellpadding="0" cellspacing="0">
          <tr> 
            <td width="17%" height="20" bgcolor="#0080C0" class="td"> <div align="center"><font color="#FFFFFF">[课         
                程 大 纲]</font></div></td>        
            <td width="83%" class="td">　</td>
          </tr>
          <tr> 
            <td height="64" colspan="2" class="td"> <p><font size="2"><b>一、精益生产推进体系与IE管理与改善的关系</b><br>
              精益推进体系以5S为基础，在准时化、柔性化生产的原则下，追求零浪费的生产<br>
              精益生产“零浪费”与IE改善的关系<br>
              价值流改善压缩制造周期改善案例介绍<br>
              工业工程(IE)－提高效率降低成本的利器&nbsp;<br>
              IE部管理职责－IE改善在工厂中的定位<br>
              <b>            二、程序分析-流程再造消除工艺浪费的基本手段</b><br>
              产品工艺分析方法与流水化改善技巧　　<br>
              改善的着眼点－选择改善方法的技巧<br>
              程序分析的改善－精益“一个流”改善案例<br>
              <b>            三、动作经济原则(principle of motion economy)</b><br> 
              流程经济原则（principle of process economy）<br>  
              生产流水化同步化改善的经济原则<br>
              动作经济原则下的十二种动作浪费<br>
              动素分析－发现并改善无效动作或浪费的方法<br>
              解放双手的治具－双手作业改善录像分享<br>
              IE全程改善案例讲解－发现工作的乐趣与奥妙<br>
              <b>            四、时间分析与标准时间－平衡生产消除浪费</b><br>
              何谓时间分析－量化分析八大浪费的基础&nbsp;<br>
              标准时间 Standard Time客观公平的工时定额<br>  
              时间宽放（Allowance）－作业难度评估的技巧<br>
              标准时间的设定－生产绩效管理的基础<br>
              工艺规格书－设定产能控制成本的标准<br>
              <b>            五、PTS预置时间标准法―欧美、日本通用的制定工时标准的技术手法</b><br>
              预置时间标准法的概要　&nbsp;<br>
              模特法（MOD）概要与基本原理<br>
              MOD法的应用案例分析与现场演练<br>
              著名外企现场PTS改善完整录像案例介绍<br>
              <b>            六、Line balancing生产线平衡（均衡化与同步化改善的基础手法）</b><br> 
              Line balancing 定义－生产线效率的评估<br>  
              Line balancing 改善案例<br>  
              CELL生产的8项条件―单元生产的实施步骤<br>
              “一个流”与单元生产(Cell production)比较<br>  
              生产节拍的确定与控制<br>
              CELL流水化改革的五准则<br>
              <b>            七、布局LAYOUT改善压缩生产周期实现CELL生产</b><br>
              布局LAYOUT改善的理念方法<br>
              LAYOUT改善的基本原则、程序步骤及重要手法<br>
              CELL U拉布局设计图（设计说明）<br>  
              中外企业在布局水平上的差异比较（精益布局改善案例）<br>
              同步化（INLINE）布局（著名外企布局图）<br>
              <b>            八、标准作业Standard Operation效率标准化改善</b><br> 
              何谓作业标准OS （Operation Standard）<br>  
              作业标准的编制目的与方法<br>
              何谓标准作业（SO）－此方法来源于精益生产模式当1名作业员操作3台机械作业时，记录作业步骤和标准时间及产能标准的方法。<br>
              <b>            九、快速转产与快速换模（SMED）－生产制造过程中的快速切换技术</b><br>
              快速转产的概念和方案<br>
              转产的过程普遍存在问题与重点留意事项　&nbsp;<br>
              单分换模(SMED)的目的与生产交货期的关系&nbsp;<br>
              SMED快速换模八步法&nbsp;&nbsp;<br>
            SMED快速转产换模改善，应对快速交货及多品种、小批量案例：换模时间占总停机时间比率降低：21.0%→16.8%，占总稼动时间：3.9%→2.1%、单机换模时间减少30％～70％<br>
              <b>            十、生产改革－效率改善的方法</b><br>
              生产改革的方向--“彻底消除一切浪费”<br>
              认识浪费―精益生产的八大浪费<br>
              浪费产生与固化的过程<br>
              消除浪费提高效率的四个阶段（案例分享）<br>
              <b>            十一、著名外企经典改善实例分享－配有现场录像</b><br>
              冲压一个流改善：生产周期三天变一天<br>
              生产周期Lead Time改善案例：独资日企案例<br>  
            案例效果：生产周期由10天到4天半的飞跃，同时库存下降45％,资金周转率提高一倍以上，仓库面积减少53％…。<br>
              注塑改善：生产同步化改善削除浪费<br>
            案例效果：（WIP）下降50％、注塑产品的生产周期由4天减为2天惊人方法，报告以三集过程录像再现完整现场改善过程。&nbsp;<br>
              注塑改善：品质改善异物不良低减案例。<br>
              组装改善实例：效率提升36％的装配改善案例<b>　</b></font></p></td>
          </tr>
        </table>
        <table width="99%" height="84" border="0" cellpadding="0" cellspacing="0">
          <tr> 
            <td width="17%" height="20" bgcolor="#0080C0" class="td"> 
              <div align="center"><font color="#FFFFFF">[导 师 简 介]</font></div></td>       
            <td width="83%" class="td">　</td>
          </tr>
          <tr> 
            <td height="64" colspan="2" class="td"> <p><font size="2">&nbsp;&nbsp;&nbsp;&nbsp;   
			Mr   
			Wang，管理工程硕士、高级经济师，国际职业培训师协会认证职业培训师，历任跨国公司生产负责人、工业工程经理、管理会计分析师、营运总监等高级管理职务多年，同时还担任&lt;价值工程&gt;杂志审稿人、辽宁省营口市商业银行独立董事等职务，对企业管理有较深入的研究。王老师主要从事IE技术应用、成本控制、管理会计决策等课程的讲授，先后为IBM、TDK、松下、可口可乐、康师傅、汇源果汁、雪津啤酒、吉百利食品、冠捷电子、INTEX明达塑胶、正新橡胶、美国ITT集团、广上科技、美的空调,中兴通讯、京信通信、联想电脑，应用材料(中国)公司、艾克森-金山石化、中国化工进出口公司、正大集团大福饲料、厦华集团、灿坤股份、NEC东金电子、太原钢铁集<span style="letter-spacing: -1pt">、PHILIPS</span>、深圳开发科技、大冷王运输制冷、三洋华强、TCL等知名企业提供项目辅导或专题培训。王老师授课经验丰富，风格幽默诙谐、逻辑清晰、过程互动，案例生动、深受学员喜爱。</font></p></td>
          </tr>
        </table>
      </div>
      <div align="center"> 
        <table width="667" border="0" cellpadding="0" cellspacing="0" height="46">
          <tr> 
            <td width="111" height="20" bgcolor="#0080C0" class="td"> <div align="center"><font color="#FFFFFF">[时间/地点/报名]</font></div></td>
            <td width="552" class="td" height="20">　</td>
          </tr>
          <tr> 
            <td height="26" colspan="2" class="td" width="665"> 
              <p><font size="2">时间:&nbsp;7月29-30日&nbsp;    
              (周六/日)&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;    
              &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 地点:   
				上海</font></p>
              </td>
          </tr>
        </table>
      </div>
      <table width="99%" height="27" border="0" align="center" cellpadding="0" cellspacing="0">
        <tr> 
          <td height="27" class="td"> <p><font size="2">费用:&nbsp;1800元/人(<font face="宋体">含课程费、教材、午餐、茶水等</font>)&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;优惠:四人以上参加,赠予一名名额   
			</font>
            </p>
            </td>
        </tr>
      </table>
      <table width="99%" height="32" border="0" align="center" cellpadding="0" cellspacing="0">
        <tr> 
          <td height="12" class="td"> <font size="2">报名/咨询电话:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
            &nbsp;谢小姐&nbsp; 1 3 6 8 1 7 9 5 7 4 0&nbsp;&nbsp;&nbsp;(上海以外客户请加拨0 
            ) <br>  
            注: 如您不需要此邮件,请将邮箱发送至:&nbsp;ts...@to...（并在邮件标题注明订退）</font></td>  
        </tr>
      </table> </td>
  </tr>
</table>
</body>
</html>

[Numpy-discussion] Pretty printing an array: no ellipses?

[Webb S. <web...@gm...>]

I am not sure where to look for this, sorry if it is RTFM or JPS
("just plain stupid"):

Is there a way to set a default to print the entire array, rather than
an ellipses version of it?  If not, why doesn't
pprint.pformat(numpy.random.normal(0,1,(100, 100)), width=1000) at
least give me something off the screen?

Thx!

Re: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__)

[Kevin J. <ja...@bi...>]

On 7/19/06, Travis Oliphant <oli...@ee...> wrote:
>
> Are you now using SVN NumPy, or are we still looking at NumPy 0.9.8numbers?



Sorry -- SVN updated as of about two hours ago.

Thanks,
-Kevin

Re: [Numpy-discussion] Performance of defmatrix.py:103(__array_finalize__)

[Travis O. <oli...@ee...>]

Kevin Jacobs <ja...@bi...> wrote:

> On 7/19/06, *Travis Oliphant* <oli...@ie... 
> <mailto:oli...@ie...>> wrote:
>
>     Kevin Jacobs <ja...@bi...
>     <mailto:ja...@bi...>> wrote:
>     > On 7/19/06, *Kevin Jacobs <ja...@bi...
>     <mailto:ja...@bi...>
>     > <mailto: ja...@bi...
>     <mailto:ja...@bi...>>>* <bio...@gm...
>     <mailto:bio...@gm...>
>     > <mailto:bio...@gm... <mailto:bio...@gm...>>>
>     wrote:
>     >
>     >     Is it expected that the following script would spend over
>     half of
>     >     the total execution time in
>     defmatrix.py:103(__array_finalize__)?
>     >
>     >
>     > Confirmed that a similar overhead exists for arrays, though not as
>     > extreme.  My application requires the ability to assign random
>     > elements, so any speed ups here will result be quite significant.
>     >
>     There is an optimization for this kind of assignment in current NumPy
>     SVN which "should" avoid the array creation over-head.
>
>
>
> There still seems to be a huge performance penalty with using a matrix 
> to set elements.  Consider this code where X is initially a matrix:


Are you now using SVN NumPy, or are we still looking at NumPy 0.9.8 numbers?

-Travis