numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] super-packages [was "Hello and my first patch"]

[A. M. A. <per...@gm...>]

On 06/10/06, Christopher Barker <Chr...@no...> wrote:
> A. M. Archibald wrote:
> > Is setting out to create a "MATLAB replacement" a good goal?
>
> No.

I see what you mean, but I still think that using MATLAB as a
yardstick ("Can users do everything with the python superpackage that
they could with MATLAB?" "Is it of comparable or lesser difficulty?")
is a good idea, at least until we've clearly outstripped it in all
categories.

> MATLAB two advantages (and they are big ones!): integrated plotting and
> a large and broad library of ready-to-use functions for a wide variety
> of computing.

This is what I mean about using it as a yardstick.

As a MATLAB survivor, would you be interested in going to
http://www.scipy.org/NumPy_for_Matlab_Users and adding numerical tools
that MATLAB has, so that we can use it as a checklist for what tools
we'd like to have (first)? I put a short list there, but it's years
since I last used MATLAB.

> MPL has almost solved one, and SciPy is getting better an better -- so
> we're really on the right track!

I agree.

A. M. Archibald

Re: [Numpy-discussion] super-packages [was "Hello and my first patch"]

[Christopher B. <Chr...@no...>]

A. M. Archibald wrote:
> Is setting out to create a "MATLAB replacement" a good goal?

No.

> the goal should be to create a useful tool for scientists.

Yes.

> but I think MATLAB is clearly already that, so (at first) it can serve
> as a definite objective. Of course, anyone who wants to make scipy
> *better* than MATLAB should be warmly encouraged...

It already *is* better!

I say this as a former, committed MATLAB user. I personally think Python 
is a much better foundation language, and numpy a better array 
implementation.

MATLAB two advantages (and they are big ones!): integrated plotting and 
a large and broad library of ready-to-use functions for a wide variety 
of computing.

MPL has almost solved one, and SciPy is getting better an better -- so 
we're really on the right track!

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer
                                     		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chr...@no...

Re: [Numpy-discussion] "Manually" broadcasting arrays in Python

[Stefan v. d. W. <st...@su...>]

On Wed, Oct 04, 2006 at 01:37:55AM -0400, A. M. Archibald wrote:
> Would it be useful for me to contribute the tiny script I wrote to
> trigger it as a regression test?
>=20
> A. M. Archibald
>=20
> from numpy import vectorize, zeros
>=20
> vt =3D vectorize(lambda *args: args)
> # Removing either of the following lines cures the segfault
> vt(zeros((1,2,1)), zeros((2,1,1)), zeros((1,1,2)))
> vt(zeros((1,2,1)), zeros((2,1,1)), zeros((1,1,2)), zeros((2,2)))

I also get a segfault when running this, but the strange thing is that
I can't catch it with valgrind (which also segfaults).

I've filed a ticket at

http://projects.scipy.org/scipy/numpy/ticket/325

Regards
St=E9fan

[Numpy-discussion] PyTables passes all tests with NumPy at its core

[Francesc A. <fa...@ca...>]

Hi,

I'm very happy to announce that, after several weeks of effort, PyTables is=
=20
able to run its complete test suite (made of more than 3000 test units) usi=
ng=20
*NumPy* at its core (now numarray is just an optional package :-).

However, and before the new PyTables will hit the public (most probably=20
released as PyTables 2.0), we should still find more time to add more tests=
=20
(specially for numarray, which we plan to continue supporting), update=20
manual, and making sure that the new package deploys well on every supporte=
d=20
plaform. So there is a lot of work to be done yet, but things are in good=20
shape.

I have to say that although NumPy was an already well tested package for=20
homogeneous data containers, PyTables has strong requirements for=20
heterogeneous arrays as well, and most of my efforts lately have been to=20
locate the flaws in that area and reporting them properly to the NumPy issu=
e=20
tracker. Fortunately, at the NumPy end there was a formidable team (and in=
=20
particular, a formidable Travis Oliphant) ready for eat all bugs that I was=
=20
able to provide.  As a result, I'd say that NumPy is quite mature for deali=
ng=20
with arrays of records as well, which is great news :-)

Cheers!

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Numpy-discussion] super-packages [was "Hello and my first patch"]

[A. M. A. <per...@gm...>]

On 06/10/06, Jon Peirce <Jon...@no...> wrote:
>  > Perhaps that is the best way to move forward along with the work on a
>  > "pylab" super-package.
>
> [...]
>
> But why not scipy? It seems the right name for a super-package.

It does, particularly if (as is being discussed) the current content
of scipy might get subdivided into separately buildable modules
(weave, fortran, non-fortran, say). scipy would then refer to the lot
- numpy, matplotlib, weave, scipy-fortran, scipy-nonfortran, and any
others that seem essential.

Is setting out to create a "MATLAB replacement" a good goal? I think
so. Really the goal should be to create a useful tool for scientists,
but I think MATLAB is clearly already that, so (at first) it can serve
as a definite objective. Of course, anyone who wants to make scipy
*better* than MATLAB should be warmly encouraged...

A. M. Archibald

Re: [Numpy-discussion] [OT] Interesting interview with Pearu (f2py and others...)

[George N. <gn...@go...>]

On 06/10/06, Fernando Perez <fpe...@gm...> wrote:
> Hi all,
>
> since this is not about me, I think it's fair to plug an interview :)
>
> http://www.pythonthreads.com/articles/interviews/once-i-learned-about-python,-i-stopped-trying-out-different-languages.html
>
> contains a very nice interview with Pearu, where in fact I learned a
> number of interesting things not only about him but also about what's
> happening with f2py.
>
> Good job, Pearu :)

Great to hear that f2py is being developed so actively.

Feature suggestion:
Fortran 90 has interface blocks that allow the calling routine to pick
up whichever of a set of routines has the appropriate datatype. So one
has foo_real32,foo_real64 etc held within the interface block, but
calling foo will automatically choose the correct routine.

I don't know whether it's possible to make this work with f2py, but it
would be very useful if it could be.

--George Nurser.

Re: [Numpy-discussion] ide for python/numpy/scipy/mpl development ?

[Robert K. <rob...@gm...>]

Eric Emsellem wrote:
> Hi,
> 
> I am looking for an IDE to develop python programs  and I am not sure
> what to take.
> The two critical items for me are 1/ a good debugger (simple and
> efficient) 2/ something simple to manage the files.
> 
> I would also very much like to keep some basic things such as (if possible):
> - editing with gvim

This probably is the most limiting factor. I use pida because it embeds gvim 
into a PyGTK frame with all of the IDE goodies around it.

   http://pida.berlios.de/

I believe it can use one of the PyGTK debugger GUIs, but I've never used it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

[Numpy-discussion] ide for python/numpy/scipy/mpl development ?

[Eric E. <ems...@ob...>]

Hi,

I am looking for an IDE to develop python programs  and I am not sure
what to take.
The two critical items for me are 1/ a good debugger (simple and
efficient) 2/ something simple to manage the files.

I would also very much like to keep some basic things such as (if possible):
- editing with gvim
- running things with Ipython -pylab

So the question is: does such an IDE exists? I have looked through a
number of them, but they look either very complex or too simple.

Any advice here is very welcome!

thanks, and cheers

Eric

Re: [Numpy-discussion] Memory errors

[Francesc A. <fa...@ca...>]

A Divendres 06 Octubre 2006 15:30, Vikalpa Jetly va escriure:
> avoid the memory error. Related question is that I need to test for
> multiple conditions on the same array and set values to 1 or 0. I see that
> statements like b[a>200 or a<50] =3D 1 do not work. So is the way to do t=
his
> simply to run a separate statement in the form b[condition]=3D 1 for each
> test?

=46or this, you have to use the binary operators instead of logical ones. F=
or=20
example:

In [43]: a=3Dnumpy.arange(10)

In [44]: b=3Dnumpy.arange(10)

In [45]: b[(a < 2) | (a > 8)] =3D 1

In [46]: b
Out[46]: array([1, 1, 2, 3, 4, 5, 6, 7, 8, 1])

[be sure to use parentesizes appropriately]

Cheers,

=2D-=20
>0,0<   Francesc Altet =A0 =A0 http://www.carabos.com/
V   V   C=E1rabos Coop. V. =A0=A0Enjoy Data
 "-"

Re: [Numpy-discussion] Memory errors

[Vikalpa J. <vj...@i3...>]

Thanks Travis and Robert. I am just getting my feet wet in numpy. Both
approaches i.e:

b = zeros_like(a)
b[a>200] = 1

or

b = (a > 200).astype(numpy.uint8)

avoid the memory error. Related question is that I need to test for multiple
conditions on the same array and set values to 1 or 0. I see that statements
like b[a>200 or a<50] = 1 do not work. So is the way to do this simply to
run a separate statement in the form b[condition]= 1 for each test?

Also since my output has to be a binary array, can the new array be defined
as binary type or nibble, potentially reducing memory overhead?

Thanks.
Vikalpa


-----Original Message-----
From: num...@li...
[mailto:num...@li...] On Behalf Of Travis
Oliphant
Sent: Thursday, October 05, 2006 5:44 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Memory errors

>
The MemoryError is a direct result when system malloc fails.    Rather 
than use where with two scalars (you're resulting array will be int32 
and therefore 4-times larger).

Use

b = zeros_like(a)
b[a>200] = 1

which will consume less memory.

-Travis
 

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Numpy-discussion mailing list
Num...@li...
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

[Numpy-discussion] super-packages [was "Hello and my first patch"]

[Jon P. <Jon...@no...>]

 > Perhaps that is the best way to move forward along with the work on a
 > "pylab" super-package.

A few years ago it seemed that scipy was developing to BE the 
super-package. It was to have plotting and some numpy flavour and all 
sorts of goodies for scientists. The name still implies that. People are 
now talking about a super-package that includes scipy as one of three 
packages.

Definitely it seems that pylab is a bad name since it's already used by 
matplotlib and the last thing we want is another confusing name 
convention that becomes hard to google.

But why not scipy? It seems the right name for a super-package.

Jon


This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Re: [Numpy-discussion] repmat

[Stefan v. d. W. <st...@su...>]

On Fri, Oct 06, 2006 at 10:30:43AM +0900, Bill Baxter wrote:
> If this is somehow controversial for some reason then let's discuss
> it.  But so far the only response has been "copying data is a bad
> idea", which is really a separate issue.

An interesting issue, though.  I've often wondered about an array
view where data is visible in more than one place.  For example, you
have an array

x =3D N.arange((100))

on which you'd like to apply a filter, which, say, finds the moving
variance of the 5 closest elements.  Here, a new representation of x
would be useful:

In [23]: indices =3D N.arange(5) + N.arange(96).reshape((96,1))

In [24]: indices
Out[24]:=20
array([[ 0,  1,  2,  3,  4],
       [ 1,  2,  3,  4,  5],
       ...
       [95, 96, 97, 98, 99]])


In [25]: y =3D x.index_view(indices) # returns array with same values as
   	                           # y =3D x[indices]
                                   # except that no values are copied

In [29]: y
Out[29]:=20
array([[ 0,  1,  2,  3,  4],
       [ 1,  2,  3,  4,  5],
       [ 2,  3,  4,  5,  6],
       ...

after which you would be able to do

In [30]: y.var(axis=3D1)
Out[30]:=20
array([ 2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,
        2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,  2.,

While there would be a lot of jumping around in indices, at the loops
can be done in C, which should speed up the process.  Of course, with
small arrays I can simply copy the data, but it becomes troublesome
with images and other such large signals.

I am not sure if providing the indices is the best way to initialise
such an array (otherwise maybe a rule for index calculation?) or how
feasible it would be to implement, so I'd be glad to hear anyone's
thoughts on the topic.

Regards
St=E9fan

Re: [Numpy-discussion] On loop and broadcasting (again)

[David C. <da...@ar...>]

Travis Oliphant wrote:
> David Cournapeau wrote:
>
>> Hi,
>>
>>   The email from Albert made me look again on some surprising results I 
>> got a few months ago when starting my first "serious" numpy project. I 
>> noticed that when computing multivariate gaussian densities, centering 
>> the data was more expensive than everything else, including 
>> exponentiation. Now that I have some experience with numpy, and 
>> following the previous discussion, I tried the following script:
>>  
>>
>
> Try it again with the new code in SVN.
>
It looks like your modification solved this particular issue !:

     10    0.645    0.065    0.645    0.065 
storage_email.py:8(center_broadcast)
     10    0.625    0.062    0.625    0.062 
storage_email.py:26(center_manual)
      1    0.333    0.333    0.333    0.333 
/usr/lib/python2.4/site-packages/numpy/lib/shape_base.py:530(repmat)

Now, using broadcast is as fast as doing the substraction itself (which 
does not include the necessary repmat). I tried it on my laptop, where I 
can safely use beta code (python 2.4 + SVN numpy), which  explains the 
timing differences compared to my previous email; as the memory is 
limited on the laptop, I only benchmarked the brodcasting. For smaller 
arrays, I couldn't see major differences in relative timing for the 
other implementations.

Thank you very much !

David

Re: [Numpy-discussion] repmat

[David C. <da...@ar...>]

Bill Baxter wrote:
>
>> I'm not completely sold on the whole reparray thing: does repmat have
>> any use outside of the linear algebra convention of squashing a 3D array
>> into a 2D matrix (or however that goes)? If not, perhaps it should just
>> get left alone and exciled to matrix specific functions namespace. So:
>>     
>
> Well, I'm not sure about N-d, but I've definitely had a need for 1-d
> version of it.  That's what prompted me to look into this.  Also I
> think I read a comment in the other thread about pyEM that repmat()
> turned out to be one of the faster ways to implement
> something-or-other (?). 
It was in a very specific case, and it could have been replaced by an 
other function in this case without any loss of performance. The 
following is my experience with matlab/repmat: generally, repmat was 
necessary in many cases in matlab either for efficiency (big scalar 
matrix were slow to generate using ones and zeros, for example) and 
because matlab had almost no broadcasting rules, repmat was almost 
mandatory for matlab. In numpy, the syntax at least does not require 
repmat (at least, I never *had* to use it in numpy, and I used it a lot 
with matlab).

Moreoever, the slowness of some broadcasting seem solved in recent 
subversion (see my answer on the specific thread),

David
>  At the very least you could think of tiling
> 3D image data as one application, in fact 'tile()' is probably a
> better name for the function than 'reparray'.
>
> I was initially suprised that 'repeat()' featured so prominently in
> numpy.  I initially assumed it meant "repmat", and was surprised when
> it didn't do that.  I thought "why on earth would you want to do
> that?".  But it does turn out to have lots of uses (like implementing
> "repmat" ;-))
>
>   
>> +1 on deprecating, moving or otherwise getting rid of repmat.
>> Neutral on whether to replace it with something else, but if it is
>> replaced #3 looks like the correct route, but not with a different
>> argument name than 'shape'. Perhaps 'reps'.
>>     
>
> Hey!  Looks like we agree on the name of the argument, apparently.  :-)
> I do think there is a strong connection with "shape" though, that
> 'reps' hides. Maybe 'repshape' or 'tileshape'?  I'd be happy with
> reps, though.  Agreed that 'shape' is bad.  Makes it sound like you
> should specify the actual .shape of the output.
>
> --bb
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>   

Re: [Numpy-discussion] contribution about history of Numeric

[Fernando P. <fpe...@gm...>]

On 10/5/06, Paul Dubois <pfd...@gm...> wrote:
> I was reading the 'History of SciPy' page and noticed the discussion about
> Numeric. Here's the true story about why the various names for the original:
> numpy, Numeric, Numerical.

I added this to the wiki page (http://www.scipy.org/History_of_SciPy),
though I don't know how to make a block quote in wiki markup, so I
left it without any visual clue.

Thanks for the info!

Best,

f

[Numpy-discussion] contribution about history of Numeric

[Paul D. <pfd...@gm...>]

I was reading the 'History of SciPy' page and noticed the discussion about
Numeric. Here's the true story about why the various names for the original:
numpy, Numeric, Numerical.

At the time Source Forge was pretty young, and I decided to put the project
there. We all said 'numpy' informally not Numerical Python but the module
name was Numeric. I created the project as numpy. I have no memory of why I
didn't call it Numeric, but if it wasn't a conflict, probably I was focused
on making it clear it was for Python and/or easy to type. (the FTP's etc.
all had to go through a long directory path that involved the name). The
documentation for the CVS stuff was confusing, and I made a mistake with my
first submit of 'Numeric' (I think it was ending up with everything in
Numeric/Numeric) and then discovered I could not delete it; you had to ask
the Source Forge staff. Impatient, I did a second submit as Numerical.

In short, all my fault, but then again, SF was so security-minded that it
was hard to do anything. That's why I soon gave up on their website and
hosted it at my own site for so long.

[Numpy-discussion] [OT] Interesting interview with Pearu (f2py and others...)

[Fernando P. <fpe...@gm...>]

Hi all,

since this is not about me, I think it's fair to plug an interview :)

http://www.pythonthreads.com/articles/interviews/once-i-learned-about-python,-i-stopped-trying-out-different-languages.html

contains a very nice interview with Pearu, where in fact I learned a
number of interesting things not only about him but also about what's
happening with f2py.

Good job, Pearu :)

Best,

f

Re: [Numpy-discussion] repmat

[Bill B. <wb...@gm...>]

On 10/6/06, Tim Hochberg <tim...@ie...> wrote:
> Bill Baxter wrote:
> > In short, repmat(A, M,N) is an oddball.  It only deals with 2D arrays.
> >
> > We should do some combination of:
> > 1) change repmat(A, M,N) to repmat(A, *dims) to add multidim ability
> > to it, while maintaining backwards compatibility.
> >
> Doesn't repmat imply matrix and thus 2d arrays? It is pretty horrible
> though.

It does kind of imply that (which is why I think something like
'reparray' would be a better name) but even matlab's repmat() function
doesn't restrict you to 2D arrays, and I'm pretty sure numpy.repmat
was intended to be a clone of that.

> > 2) change repmat(A, M,N) to repmat(A, shape) to add multidim ability
> > to it, and bring it into line with the "numpy standard" of shapes
> > being passed as one tuple arg.
> >
> I presume shape actually means number of repetitions, and not shape. If
> so, shape is the wrong name for the second argument. Anyway as long as
> your breaking backward compatibility you might as well improve the name.

Yeh, that did occur to me.  Maybe 'reps' would be a better name for
the argument.

> > 3) add a new method like reparray(A, shape) that has the multidim
> > ability, and matches numpy conventions.
> >
> Same comment on the name shape.
>
>
> > 4)   if 3), move repmat out of the top-level namespace to indicate deprecation
> >
> I'm not completely sold on the whole reparray thing: does repmat have
> any use outside of the linear algebra convention of squashing a 3D array
> into a 2D matrix (or however that goes)? If not, perhaps it should just
> get left alone and exciled to matrix specific functions namespace. So:

Well, I'm not sure about N-d, but I've definitely had a need for 1-d
version of it.  That's what prompted me to look into this.  Also I
think I read a comment in the other thread about pyEM that repmat()
turned out to be one of the faster ways to implement
something-or-other (?).  At the very least you could think of tiling
3D image data as one application, in fact 'tile()' is probably a
better name for the function than 'reparray'.

I was initially suprised that 'repeat()' featured so prominently in
numpy.  I initially assumed it meant "repmat", and was surprised when
it didn't do that.  I thought "why on earth would you want to do
that?".  But it does turn out to have lots of uses (like implementing
"repmat" ;-))

> +1 on deprecating, moving or otherwise getting rid of repmat.
> Neutral on whether to replace it with something else, but if it is
> replaced #3 looks like the correct route, but not with a different
> argument name than 'shape'. Perhaps 'reps'.

Hey!  Looks like we agree on the name of the argument, apparently.  :-)
I do think there is a strong connection with "shape" though, that
'reps' hides. Maybe 'repshape' or 'tileshape'?  I'd be happy with
reps, though.  Agreed that 'shape' is bad.  Makes it sound like you
should specify the actual .shape of the output.

--bb

Re: [Numpy-discussion] repmat

[Tim H. <tim...@ie...>]

Bill Baxter wrote:
> [There seem to have been some gmail delivery problems that prevented
> my previous mail on this subject from being delivered]
>
> I've proposed that we fix repmat handle arbitrary dimensions before 1.0.
>
>    http://projects.scipy.org/scipy/numpy/ticket/292
>
> I don't think this is particularly controversial, just I'm guessing
> no-one's found the time to look at my proposed fixes.  And
> gmail/sourceforge issues haven't helped either.
>
> In short, repmat(A, M,N) is an oddball.  It only deals with 2D arrays.
>
> We should do some combination of:
> 1) change repmat(A, M,N) to repmat(A, *dims) to add multidim ability
> to it, while maintaining backwards compatibility.
>   
Doesn't repmat imply matrix and thus 2d arrays? It is pretty horrible 
though.

> 2) change repmat(A, M,N) to repmat(A, shape) to add multidim ability
> to it, and bring it into line with the "numpy standard" of shapes
> being passed as one tuple arg.
>   
I presume shape actually means number of repetitions, and not shape. If 
so, shape is the wrong name for the second argument. Anyway as long as 
your breaking backward compatibility you might as well improve the name.

> 3) add a new method like reparray(A, shape) that has the multidim
> ability, and matches numpy conventions.
>   
Same comment on the name shape.


> 4)   if 3), move repmat out of the top-level namespace to indicate deprecation
>   
I'm not completely sold on the whole reparray thing: does repmat have 
any use outside of the linear algebra convention of squashing a 3D array 
into a 2D matrix (or however that goes)? If not, perhaps it should just 
get left alone and exciled to matrix specific functions namespace. So:

+1 on deprecating, moving or otherwise getting rid of repmat.
Neutral on whether to replace it with something else, but if it is 
replaced #3 looks like the correct route, but not with a different 
argument name than 'shape'. Perhaps 'reps'.

-tim

> The tracker item has all the code necessary to implement these fixes,
> it's just a matter of deciding which one to go with.
>
> My personal preference would be a combination of 1 and 3.  Add
> reparray(A,shape) , and change repmat to repmat(A, *shape).  Then just
> implement repmat as a call to reparray.
>
> If someone will decide which route to go, I'd be happy to provide an
> actual patch against the latest SVN to implement that decision.
>
> If this is somehow controversial for some reason then let's discuss
> it.  But so far the only response has been "copying data is a bad
> idea", which is really a separate issue.
>
> --bb
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>   

[Numpy-discussion] repmat

[Bill B. <wb...@gm...>]

[There seem to have been some gmail delivery problems that prevented
my previous mail on this subject from being delivered]

I've proposed that we fix repmat handle arbitrary dimensions before 1.0.

   http://projects.scipy.org/scipy/numpy/ticket/292

I don't think this is particularly controversial, just I'm guessing
no-one's found the time to look at my proposed fixes.  And
gmail/sourceforge issues haven't helped either.

In short, repmat(A, M,N) is an oddball.  It only deals with 2D arrays.

We should do some combination of:
1) change repmat(A, M,N) to repmat(A, *dims) to add multidim ability
to it, while maintaining backwards compatibility.
2) change repmat(A, M,N) to repmat(A, shape) to add multidim ability
to it, and bring it into line with the "numpy standard" of shapes
being passed as one tuple arg.
3) add a new method like reparray(A, shape) that has the multidim
ability, and matches numpy conventions.
4)   if 3), move repmat out of the top-level namespace to indicate deprecation

The tracker item has all the code necessary to implement these fixes,
it's just a matter of deciding which one to go with.

My personal preference would be a combination of 1 and 3.  Add
reparray(A,shape) , and change repmat to repmat(A, *shape).  Then just
implement repmat as a call to reparray.

If someone will decide which route to go, I'd be happy to provide an
actual patch against the latest SVN to implement that decision.

If this is somehow controversial for some reason then let's discuss
it.  But so far the only response has been "copying data is a bad
idea", which is really a separate issue.

--bb

Re: [Numpy-discussion] Memory errors

[Robert K. <rob...@gm...>]

If you are going to start a new topic, please start a new thread as well, 
instead of replying to a message. Thanks.

Vikalpa Jetly wrote:
> I am reading a very large array (~9000,11000) of 1 byte image values. I need
> to change values in the array that meet a certain condition so I am running
> something like:
> 
> b = numpy.where(a>200,0,1)
> 
> to create a new array with the changed values. However, I get a
> "MemoryError" everytime I try this. I have over 3gb of RAM on my machine
> (most of which is available). The process runs fine on smaller datasets. Is
> there a maximum array size that numpy handles? Any alternatives/workarounds?

There is no predefined limit on the array size, just your memory.

However, note that what you are doing here is creating an int32 (or int64 if you 
are on a 64-bit machine) array since you are using Python integers in your 
where() function. You can save quite a bit of memory by using the array from 
(a>200) and simply casting it to the uint8 type. That way, you only have two 
arrays in memory at any time, a and b.

   b = (a > 200).astype(numpy.uint8)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

Re: [Numpy-discussion] Memory errors

[Travis O. <oli...@ee...>]

Vikalpa Jetly wrote:

>I am reading a very large array (~9000,11000) of 1 byte image values. I need
>to change values in the array that meet a certain condition so I am running
>something like:
>
>b = numpy.where(a>200,0,1)
>
>to create a new array with the changed values. However, I get a
>"MemoryError" everytime I try this. I have over 3gb of RAM on my machine
>(most of which is available). The process runs fine on smaller datasets. Is
>there a maximum array size that numpy handles? Any alternatives/workarounds?
>
>  
>
The MemoryError is a direct result when system malloc fails.    Rather 
than use where with two scalars (you're resulting array will be int32 
and therefore 4-times larger).

Use

b = zeros_like(a)
b[a>200] = 1

which will consume less memory.

-Travis
 

[Numpy-discussion] Memory errors

[Vikalpa J. <vj...@i3...>]

I am reading a very large array (~9000,11000) of 1 byte image values. I need
to change values in the array that meet a certain condition so I am running
something like:

b = numpy.where(a>200,0,1)

to create a new array with the changed values. However, I get a
"MemoryError" everytime I try this. I have over 3gb of RAM on my machine
(most of which is available). The process runs fine on smaller datasets. Is
there a maximum array size that numpy handles? Any alternatives/workarounds?

Thanks.
Vikalpa

-----Original Message-----
From: num...@li...
[mailto:num...@li...] On Behalf Of A. M.
Archibald
Sent: Thursday, October 05, 2006 5:02 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Hello and my first patch

On 05/10/06, Travis Oliphant <oli...@ee...> wrote:

> I think a hybrid for weave / f2py / ctypes that allows "inlining in
> multiple languages" as well as automatic extension module generation for
> "already-written" code is in order.

It might make sense to also include SWIG (since that seems to be a
popular choice for wrapping "already-written" C and C++ code).

A. M. Archibald

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Numpy-discussion mailing list
Num...@li...
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Hello and my first patch

[A. M. A. <per...@gm...>]

On 05/10/06, Travis Oliphant <oli...@ee...> wrote:

> I think a hybrid for weave / f2py / ctypes that allows "inlining in
> multiple languages" as well as automatic extension module generation for
> "already-written" code is in order.

It might make sense to also include SWIG (since that seems to be a
popular choice for wrapping "already-written" C and C++ code).

A. M. Archibald

Re: [Numpy-discussion] flat indexing of object arrays

[Travis O. <oli...@ee...>]

Matthew Brett wrote:

>Hi,
>
>On 10/5/06, Martin Wiechert <mar...@gm...> wrote:
>  
>
>>Hi list,
>>
>>when I try to assign a sequence as an element of an object array via flat
>>indexing only the first element of the sequence is assigned:
>>    
>>
>
>I've also been having trouble with flat on object arrays.
>
>Is this intended?
>
>In [1]: from numpy import *
>
>In [2]: a = arange(2)
>
>In [3]: a[1]
>Out[3]: 1
>
>In [4]: a.flat[1]
>Out[4]: 1
>
>In [5]: b = array([a], dtype=object)
>
>In [6]: b[1]
>---------------------------------------------------------------------------
>exceptions.IndexError                                Traceback (most
>recent call last)
>
>/home/mb312/devel_trees/scipy/Lib/io/<ipython console>
>
>IndexError: index out of bounds
>
>In [7]: b.flat[1]
>Out[7]: 1
>
>  
>

This is correct behavior.  Look at the shape of b.  It is being indexed 
correctly.

The problem is that it is ambiguous as to what is wanted when you write

b = array([a], dtype=object).

We have gone through the rounds on this one and the current behavior is 
our best compromise.

-Travis