Menu
▾
▴
numpy-discussion
|
From: Greg W. <gre...@gm...> - 2006-10-05 03:05:32
|
Hello All, I introduced myself on the Scipy list and I have a feeling that most of the subscribers here are also on Scipy-devel. Anyway I just submitted my first patch to numpy (ticket #316). I added the blackman-harris, Nuttall and Flat Top windowing functions and added "See also" sections to the docstrings for all the window functions. I probably didn't do the patch right though because I just did an "svn diff >patch". If there is a way you'd rather see it please let me know and I'll fix it. I look forward to contributing a lot more in the future. Have a nice day. Greg -- Linux. Because rebooting is for adding hardware. |
|
From: Travis O. <oli...@ee...> - 2006-10-05 04:16:07
|
Greg Willden wrote: > Hello All, > I introduced myself on the Scipy list and I have a feeling that most > of the subscribers here are also on Scipy-devel. Anyway I just > submitted my first patch to numpy (ticket #316). I added the > blackman-harris, Nuttall and Flat Top windowing functions and added > "See also" sections to the docstrings for all the window functions. Great contribution. Thanks a bunch. I think this will probably go into the scipy package, though. There is already a lot of windows available in the scipy.signal.window function. The window functions that are in NumPy are there only for historical purposes only (i.e. compatibility with old MLab). On the other hand, the other thought to consider is that since we have window functions in NumPy already. Let's just move them all from scipy.signal into NumPy. -Travis |
|
From: Sebastian H. <ha...@ms...> - 2006-10-05 04:35:54
|
Travis Oliphant wrote: > Greg Willden wrote: > >> Hello All, >> I introduced myself on the Scipy list and I have a feeling that most >> of the subscribers here are also on Scipy-devel. Anyway I just >> submitted my first patch to numpy (ticket #316). I added the >> blackman-harris, Nuttall and Flat Top windowing functions and added >> "See also" sections to the docstrings for all the window functions. > > Great contribution. Thanks a bunch. I think this will probably go into > the scipy package, though. There is already a lot of windows available > in the scipy.signal.window function. > > The window functions that are in NumPy are there only for historical > purposes only (i.e. compatibility with old MLab). > > On the other hand, the other thought to consider is that since we have > window functions in NumPy already. Let's just move them all from > scipy.signal into NumPy. If scipy is going to be installable as separate sub-packages maybe all window functions can be moved to scipy ;-) In other words, if the ones in numpy are there only for "historical reasons" maybe they should be cleaned out before the 1.0 release. All arguments seem similar to ndimage (which was is numarray and is now in scipy) -Sebastian > > -Travis > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys -- and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Numpy-discussion mailing list > Num...@li... > https://lists.sourceforge.net/lists/listinfo/numpy-discussion |
|
From: Greg W. <gre...@gm...> - 2006-10-05 14:02:28
|
On 10/4/06, Travis Oliphant <oli...@ee...> wrote: > > Great contribution. Thanks a bunch. I think this will probably go into > the scipy package, though. There is already a lot of windows available > in the scipy.signal.window function. > BTW. Is there some sort of clear statement about what goes in Scipy versus what goes in Numpy? It's a bit confusing for this newbie. Thanks Greg -- Linux. Because rebooting is for adding hardware. |
|
From: Robert K. <rob...@gm...> - 2006-10-05 15:47:55
|
Greg Willden wrote: > On 10/4/06, *Travis Oliphant* <oli...@ee... > <mailto:oli...@ee...>> wrote: > > Great contribution. Thanks a bunch. I think this will probably go into > the scipy package, though. There is already a lot of windows available > in the scipy.signal.window function. > > BTW. Is there some sort of clear statement about what goes in Scipy > versus what goes in Numpy? It's a bit confusing for this newbie. IMO, I'd rather see this and similar functions go into scipy. New functions that apply semantics to arrays (in this case, treating them as time series), I think should go into scipy. New functions that treat arrays simply as arrays and are generally useful can probably go into numpy. There's some grey area in this scheme, of course. New non-uniform random number generators might need to go into numpy, for now, simply due to technical reasons. Fleshing out the masked array and matrix classes would be similar, I imagine. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco |
|
From: Greg W. <gre...@gm...> - 2006-10-05 15:58:19
|
On 10/5/06, Robert Kern <rob...@gm...> wrote: > > IMO, I'd rather see this and similar functions go into scipy. New > functions that > apply semantics to arrays (in this case, treating them as time series), I > think > should go into scipy. New functions that treat arrays simply as arrays and > are > generally useful can probably go into numpy. > Okay I'll take a look at the Scipy parts tonight. So do you cancel that ticket I created or do I attach a new patch against scipy or what? Thanks Greg -- Linux. Because rebooting is for adding hardware. |
|
From: John H. <jdh...@ac...> - 2006-10-05 16:33:33
|
>>>>> "Robert" == Robert Kern <rob...@gm...> writes:
Robert> IMO, I'd rather see this and similar functions go into
Robert> scipy. New functions that apply semantics to arrays (in
Robert> this case, treating them as time series), I think should
Robert> go into scipy. New functions that treat arrays simply as
Robert> arrays and are generally useful can probably go into
Robert> numpy.
I prefer Perry's longstanding suggestion: things that do not add to
distribution complexity should go into numpy. If it compiles as
easily as numpy itself, it should go into numpy where sensible. It
remains a fact of life that numpy gets a wider distribution than
scipy, and some packages are hesitant to require scipy as a prereq
because of the additional complexity or building fortran, etc. I
would be nice to have as much as possible in the most widely
distributed package IMO.
JDH
|
|
From: Greg W. <gre...@gm...> - 2006-10-05 19:25:19
|
On 10/5/06, John Hunter <jdh...@ac...> wrote: > > I prefer Perry's longstanding suggestion: things that do not add to > distribution complexity should go into numpy. If it compiles as > easily as numpy itself, it should go into numpy where sensible. It > remains a fact of life that numpy gets a wider distribution than > scipy, and some packages are hesitant to require scipy as a prereq > because of the additional complexity or building fortran, etc. I > would be nice to have as much as possible in the most widely > distributed package IMO. That is a much better policy in my view. I (gently) encourage this group (Travis?) to make this the policy for Numpy/Scipy. >From my view as a newbie to numpy/scipy/matplotlib it isn't clear where I should look for what functionality. Matplotlib plots the spectrogram but it only supports two or three window functions. Numpy supports 4 or 5 window functions and Scipy apparently supports more but Matplotlib doesn't support Scipy. Of course this is a minor example and I could just write the window function myself and then use it in Matplotlib but I want to give back so that the project can grow. I'd really like to be able to leave Matlab behind and encourage everyone else to do the same but there are still these annoyances that need to be worked out. Thanks Greg -- Linux. Because rebooting is for adding hardware. |
|
From: Robert K. <rob...@gm...> - 2006-10-05 19:40:25
|
Greg Willden wrote: > From my view as a newbie to numpy/scipy/matplotlib it isn't clear where > I should look for what functionality. Matplotlib plots the spectrogram > but it only supports two or three window functions. Numpy supports 4 or > 5 window functions and Scipy apparently supports more but Matplotlib > doesn't support Scipy. That's not true. specgram() takes a windowing function. It doesn't matter where that function comes from. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco |
|
From: Greg W. <gre...@gm...> - 2006-10-05 19:56:41
|
On 10/5/06, Robert Kern <rob...@gm...> wrote: > > Greg Willden wrote: > > > From my view as a newbie to numpy/scipy/matplotlib it isn't clear where > > I should look for what functionality. Matplotlib plots the spectrogram > > but it only supports two or three window functions. Numpy supports 4 or > > 5 window functions and Scipy apparently supports more but Matplotlib > > doesn't support Scipy. > > That's not true. specgram() takes a windowing function. It doesn't matter > where > that function comes from. The next sentence (that you snipped) afirms what you said. <quote> Of course this is a minor example and I could just write the window function myself and then use it in Matplotlib </quote> The details of my off-the-cuff example would probably be better addressed by the Matplotlib folks. (i.e. why they don't have builtin functions for more windows). You can see how it's all very confusing to someone new. "Matplotlib can plot a spectrogram but I need to use a window function from SciPy because Matplotlib only supports NumPy and NumPy doesn't have the one I want?" You get the idea. Greg -- Linux. Because rebooting is for adding hardware. |
|
From: Robert K. <rob...@gm...> - 2006-10-05 20:06:14
|
Greg Willden wrote: > On 10/5/06, *Robert Kern* <rob...@gm... > <mailto:rob...@gm...>> wrote: > > Greg Willden wrote: > > > From my view as a newbie to numpy/scipy/matplotlib it isn't > clear where > > I should look for what functionality. Matplotlib plots the > spectrogram > > but it only supports two or three window functions. Numpy > supports 4 or > > 5 window functions and Scipy apparently supports more but Matplotlib > > doesn't support Scipy. > > That's not true. specgram() takes a windowing function. It doesn't > matter where > that function comes from. > > The next sentence (that you snipped) afirms what you said. > <quote> > Of course this is a minor example and I could just write the window > function myself and then use it in Matplotlib > </quote> > > The details of my off-the-cuff example would probably be better > addressed by the Matplotlib folks. (i.e. why they don't have builtin > functions for more windows). > > You can see how it's all very confusing to someone new. > "Matplotlib can plot a spectrogram but I need to use a window function > from SciPy because Matplotlib only supports NumPy and NumPy doesn't have > the one I want?" > > You get the idea. No, I'm afraid I don't. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco |
|
From: A. M. A. <per...@gm...> - 2006-10-05 19:46:51
|
On 05/10/06, Greg Willden <gre...@gm...> wrote: > That is a much better policy in my view. > > I (gently) encourage this group (Travis?) to make this the policy for > Numpy/Scipy. > > From my view as a newbie to numpy/scipy/matplotlib it isn't clear where I > should look for what functionality. Matplotlib plots the spectrogram but it > only supports two or three window functions. Numpy supports 4 or 5 window > functions and Scipy apparently supports more but Matplotlib doesn't support > Scipy. Of course this is a minor example and I could just write the window > function myself and then use it in Matplotlib but I want to give back so > that the project can grow. I'd really like to be able to leave Matlab > behind and encourage everyone else to do the same but there are still these > annoyances that need to be worked out. Unfortunately, that policy (put it in numpy if it doesn't make the build dependencies any worse) makes it even harder for the user to figure out what is where. Say I want a fast matrix product. Do I look in numpy or scipy? It'll run faster if it uses a tuned BLAS, so it ought to have external requirements, so I'd look in scipy, but maybe there's a simple non-tuned implementation in numpy instead... Frankly, I tend to prefer the other approach to solving all these issues: distribute numpy, scipy, and matplotlib as one bundle. The requirements for scipy are not particularly onerous, particularly if it comes as part of your distribution. There are currently some problems successfully finding optimized versions of LAPACK and BLAS, but to me the benefits of bundling the packages together outweigh the difficulties: * routines and objects can be in the package in which they make the most semantic sense, rather than the one with the correct external dependencies (how is a user supposed to know whether convolution uses an external library or not?) * documentation can be cross-referenced between packages (so the Matrix class can tell people to look in scipy.linalg for inverses, for example) * users can more easily find what's available rather than rewriting from scratch * derived packages (ipython, IDEs, MATLAB-alikes) have many fewer possibilities to support I'm not arguing that the development processes need to be connected, just that making the primary distributed object a package containing all the components will make life easier for everyone involved. A. M. Archibald |
|
From: Alan G I. <ai...@am...> - 2006-10-05 20:11:45
|
> On 10/5/06, John Hunter <jdh...@ac...> wrote:=20 >> I would be nice to have as much as possible in the most=20 >> widely distributed package IMO.=20 On Thu, 5 Oct 2006, Greg Willden apparently wrote:=20 > That is a much better policy in my view.=20 A user's perspective: Well yes, all else equal, I'd like to have as much as=20 possible in the easiest to install package, BUT my top priority is that numpy be completely bulletproof. Do these goals conflict? Cheers, Alan Isaac |
|
From: Travis O. <oli...@ee...> - 2006-10-05 19:54:08
|
John Hunter wrote: >>>>>>"Robert" == Robert Kern <rob...@gm...> writes: >>>>>> >>>>>> > > Robert> IMO, I'd rather see this and similar functions go into > Robert> scipy. New functions that apply semantics to arrays (in > Robert> this case, treating them as time series), I think should > Robert> go into scipy. New functions that treat arrays simply as > Robert> arrays and are generally useful can probably go into > Robert> numpy. > >I prefer Perry's longstanding suggestion: things that do not add to >distribution complexity should go into numpy. If it compiles as >easily as numpy itself, it should go into numpy where sensible. > I don't think this is as feasible as it sounds at first. Some people complain that NumPy is too big already. SciPy is very easy to install on Windows (there is a binary available). The only major platform that still gives some trouble is Intel Mac (due to the fortran compiler situation). But, all you need is one person who can build it and distribute a binary. I think a better long-term solution is to understand how to package things better by working with people at Enthought so that when you advertise to the ex-Matlab user you point him to a "super-package" that installs a bunch of other small packages. This is a more maintainable solution as long as we set standards for 1) documentation 2) tests 3) some kind of problem-domain hierarchy The idea of just lumping more an more things into NumPy itself is not a good idea. What most users want is something that installs easily (like Enthon). How it is packaged is not as important. What developers need is a sane multi-namespace system that can be maintained separately if needed. I think we decided a while ago, that the package approach should contain indicators as to whether or not a fortran compiler was needed to build the system so that dependency on those things could be eliminated if needed. Do we want to pull scipy apart into two components: one that needs Fortran to build and another that doesn't? Perhaps that is the best way to move forward along with the work on a "pylab" super-package. -Travis |
|
From: Greg W. <gre...@gm...> - 2006-10-05 20:04:58
|
On 10/5/06, Travis Oliphant <oli...@ee...> wrote: > > Perhaps that is the best way to move forward along with the work on a > "pylab" super-package. That is exactly what I want. In the end I want a nice collection of functions, logically organized, that let me analyze/filter/plot etc. etc. etc. The key for me is "logically organized". And right now that means "So a newbie can find the function I need and the function I need is there" I'm not criticising. I'd like to help get there. Greg -- Linux. Because rebooting is for adding hardware. |
|
From: A. M. A. <per...@gm...> - 2006-10-05 20:13:14
|
On 05/10/06, Greg Willden <gre...@gm...> wrote: > On 10/5/06, Travis Oliphant <oli...@ee...> wrote: > > Perhaps that is the best way to move forward along with the work on a > > "pylab" super-package. > > That is exactly what I want. What is unsatisfactory about installing numpy+scipy+matplotlib? I've found they're generally pretty complete (except where no decent python alternative exists). > In the end I want a nice collection of functions, logically organized, that > let me analyze/filter/plot etc. etc. etc. > > The key for me is "logically organized". For the most part, the organization is pretty logical: * Basic array and matrix operations in numpy * linear algebra, differential equation, interpolation, etc. tools are in scipy, each in their own subpackage * weave is mysteriously in scipy * plotting tools are in matplotlib There are a few historical quirks, like window functions in numpy (they really belong in scipy), and some of the less-used scipy subpackages are a bit of a mess internally (scipy.interpolate for example), but otherwise I'm not sure what you want to be different. > And right now that means "So a newbie can find the function I need and the > function I need is there" > > I'm not criticising. I'd like to help get there. Install all three major packages. Use the window functions from scipy in matplotlib. Task-oriented documentation is so far a bit scant, although the scipy cookbook (http://www.scipy.org/Cookbook ) and the numpy examples list (http://www.scipy.org/Numpy_Example_List ) are a good start. A. M. Archibald |
|
From: Albert S. <fu...@gm...> - 2006-10-05 20:19:44
|
Hello all Some comments from a Windows user's perspective. On Thu, 05 Oct 2006, Travis Oliphant wrote: > John Hunter wrote: > > >>>>>>"Robert" == Robert Kern <rob...@gm...> writes: > > > > Robert> IMO, I'd rather see this and similar functions go into > > Robert> scipy. New functions that apply semantics to arrays (in > > Robert> this case, treating them as time series), I think should > > Robert> go into scipy. New functions that treat arrays simply as > > Robert> arrays and are generally useful can probably go into > > Robert> numpy. > > > >I prefer Perry's longstanding suggestion: things that do not add to > >distribution complexity should go into numpy. If it compiles as > >easily as numpy itself, it should go into numpy where sensible. > > I don't think this is as feasible as it sounds at first. Some people > complain that NumPy is too big already. I agree here. Focus NumPy on doing array library things. > SciPy is very easy to install on Windows (there is a binary available). > The only major platform that still gives some trouble is Intel Mac (due > to the fortran compiler situation). But, all you need is one person > who can build it and distribute a binary. So far, I've been shying away from SciPy, because, if I encounter a problem, I have no easy way of building from SVN on Windows. I don't think I'm the only one: few Windows users have a proper Fortran compiler. Sure, there's MinGW, but that breaks all my other tools, most notably the Visual Studio debugger and other useful things like profilers (e.g. IBM Rational Quantify). That being said, Enthought's nightly builds obviate the need of most Windows users to build from source. (Enthought rocks.) Two feature requests at this point, which would make NumPy/SciPy/Matplotlib dead easy to use on Windows, even if you're trying to stay close to trunk: 1. Please consider setting up a buildbot(*) that builds and runs the tests on every checkin. I've set up a buildbot for NumPy on my own machine; it takes a matter of minutes. Probably they already have something like this in place. (*) http://buildbot.sourceforge.net/ 2. Please consider doing separate builds per CPU with ATLAS 3.7.11, Intel MKL and ACML. By all means, make a generic build available that runs everywhere. This will require some reading of the MKL license agreement, but I think Enthought should be able to distribute Windows builds based on MKL without problems. Why go to this trouble? MATLAB R2006b uses Intel MKL on my CPU, and it is much faster than NumPy with ATLAS 3.6.0. Core Duo users also have the option of enabling OpenMP, to spread calculations to multiple cores. > I think a better long-term solution is to understand how to package > things better by working with people at Enthought so that when you > advertise to the ex-Matlab user you point him to a "super-package" that > installs a bunch of other small packages. This is a more maintainable > solution as long as we set standards for > > 1) documentation > 2) tests > 3) some kind of problem-domain hierarchy Agreed. If Enthought is willing to provide the resources, Enthon could be the perfect solution to many of the issues that we currently encounter to get decent builds on Windows. > The idea of just lumping more an more things into NumPy itself is not a > good idea. What most users want is something that installs easily (like > Enthon). How it is packaged is not as important. What developers need > is a sane multi-namespace system that can be maintained separately if > needed. > > I think we decided a while ago, that the package approach should contain > indicators as to whether or not a fortran compiler was needed to build > the system so that dependency on those things could be eliminated if > needed. > > Do we want to pull scipy apart into two components: one that needs > Fortran to build and another that doesn't? Maybe. Maybe not. On Linux this doesn't make much difference to me if I check out 3 projects or 10 -- builds are easy. On Windows, getting the support libraries, build tools and configuration right is much harder. Hard enough, that I don't think anybody wants to do it regularly. > Perhaps that is the best way to move forward along with the work on a > "pylab" super-package. Yes, please. Regards, Albert |
|
From: Travis O. <oli...@ee...> - 2006-10-05 04:54:52
|
Sebastian Haase wrote: >If scipy is going to be installable as separate sub-packages maybe >all window functions can be moved to scipy ;-) > >In other words, if the ones in numpy are there only for "historical >reasons" maybe they should be cleaned out before the 1.0 release. >All arguments seem similar to ndimage (which was is numarray and is now >in scipy) > > Not really, because these functions were in *both* Numeric and numarray. That's the trouble. And the multiple scipy packages situation needs more discussion.... We are all ears... -Travis |
|
From: Greg W. <gre...@gm...> - 2006-10-05 13:24:36
|
On 10/4/06, Travis Oliphant <oli...@ee...> wrote: > > Not really, because these functions were in *both* Numeric and > numarray. That's the trouble. > > And the multiple scipy packages situation needs more discussion.... We > are all ears... Well I started here because I have been using the Matplotlib package and it uses these compatibility functions. So I wanted more window functions available for calls to specgram. This does bring up a frustration for users new to scipy/numpy/matplotlib. In short, that there are three packages. I know the historical reasons why but I hope that the ultimate destination is a tight integration among these packages. I hope to help out toward that goal. Thanks Greg -- Linux. Because rebooting is for adding hardware. |
|
From: Christopher B. <Chr...@no...> - 2006-10-05 20:23:00
|
The situation is confusing, we all know that, and we all want to move
toward a better way.
Key to that is that SciPy needs to be easier to build and install --
that's happening, but I don't know that it's there yet. Maybe it can be
built on Fedora Core 4 now, but last I tried it couldn't be.
Anyway, few thoughts on comments made here:
> Matplotlib plots the spectrogram
Here's a key problem -- matplotlib includes WAY too much. There are
reasons, and there is history, but as a goal, I think matplotlib should
be just what it says it is -- a plotting library.
I don't know that MPL has been declared the "official" plotting package
for SciPy, but it would be nice if it were. SciPy has suffered for a
very long time without a full-featured, cross-platform, "official"
plotting package. AS far as I know, MPL comes the closest (except it
doesn't do 3-d -- darn!)
A. M. Archibald wrote:
> Frankly, I tend to prefer the other approach to solving all these
> issues: distribute numpy, scipy, and matplotlib as one bundle.
This is really the only way to set things up for someone that want what
could be used as a "matlab replacement". If we ever get settools working
just right, we should be able to do:
easy_install scipy
and have it all! woo hoo!
However, as we work to that goal, i do think it makes sense that numpy,
and matplotlib be packages unto themselves -- developed separately, etc.
In fact, SciPy should be a collection of distinct packages as well. I
think there is a real benefit to be able to install just what you need.
Not very user of numpy and/or MPL is looking for a full scientific
software package.
I'm a big advocate of people using numpy arrays for all sorts of thinks
that fit well into an n-d array, that have nothing to do with Scientific
programming. Matplotlib is also very useful for people who need a quick
plot for a web site or something. These people don't want to install the
entirety of SciPy.
> * routines and objects can be in the package in which they make the
> most semantic sense,
exactly. If it's plotting (but not computing stuff to plot) put it in
MPL, if it's generic computation (if you can't understand it with high
school math, it's not generic), put it in numpy. Of course, these aren't
clear definitions, but can still be useful as guidelines.
> * documentation can be cross-referenced between packages (so the
> Matrix class can tell people to look in scipy.linalg for inverses, for
> example)
If it were me, I'd probably have the Matrix package depend on a linear
algebra package, and have only one of those.
Travis Oliphant wrote:
> 3) some kind of problem-domain hierarchy
+1
> Do we want to pull scipy apart into two components: one that needs
> Fortran to build and another that doesn't?
yup -- I don't like it, but the state of Fortran compilers really gives
little choice.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
NOAA/OR&R/HAZMAT (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chr...@no...
|
|
From: Travis O. <oli...@ee...> - 2006-10-05 22:19:29
|
A. M. Archibald wrote: >On 05/10/06, Greg Willden <gre...@gm...> wrote: > > >>On 10/5/06, Travis Oliphant <oli...@ee...> wrote: >> >> >>>Perhaps that is the best way to move forward along with the work on a >>>"pylab" super-package. >>> >>> >>That is exactly what I want. >> >> > >What is unsatisfactory about installing numpy+scipy+matplotlib? I've >found they're generally pretty complete (except where no decent python >alternative exists). > > > >>In the end I want a nice collection of functions, logically organized, that >>let me analyze/filter/plot etc. etc. etc. >> >>The key for me is "logically organized". >> >> > > > There is a structure to it, but it's more organic because of the multiple contributors. weave should be in NumPy but nobody was willing to step up to maintain it a year ago. I may be willing to step up at this point. I would like to see weave in NumPy (maybe not the blitz libraries though...) I think a hybrid for weave / f2py / ctypes that allows "inlining in multiple languages" as well as automatic extension module generation for "already-written" code is in order. -Travis |
|
From: A. M. A. <per...@gm...> - 2006-10-05 23:02:07
|
On 05/10/06, Travis Oliphant <oli...@ee...> wrote: > I think a hybrid for weave / f2py / ctypes that allows "inlining in > multiple languages" as well as automatic extension module generation for > "already-written" code is in order. It might make sense to also include SWIG (since that seems to be a popular choice for wrapping "already-written" C and C++ code). A. M. Archibald |
|
From: Vikalpa J. <vj...@i3...> - 2006-10-05 23:29:24
|
I am reading a very large array (~9000,11000) of 1 byte image values. I need to change values in the array that meet a certain condition so I am running something like: b = numpy.where(a>200,0,1) to create a new array with the changed values. However, I get a "MemoryError" everytime I try this. I have over 3gb of RAM on my machine (most of which is available). The process runs fine on smaller datasets. Is there a maximum array size that numpy handles? Any alternatives/workarounds? Thanks. Vikalpa -----Original Message----- From: num...@li... [mailto:num...@li...] On Behalf Of A. M. Archibald Sent: Thursday, October 05, 2006 5:02 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Hello and my first patch On 05/10/06, Travis Oliphant <oli...@ee...> wrote: > I think a hybrid for weave / f2py / ctypes that allows "inlining in > multiple languages" as well as automatic extension module generation for > "already-written" code is in order. It might make sense to also include SWIG (since that seems to be a popular choice for wrapping "already-written" C and C++ code). A. M. Archibald ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Numpy-discussion mailing list Num...@li... https://lists.sourceforge.net/lists/listinfo/numpy-discussion |
|
From: Travis O. <oli...@ee...> - 2006-10-05 23:44:06
|
Vikalpa Jetly wrote: >I am reading a very large array (~9000,11000) of 1 byte image values. I need >to change values in the array that meet a certain condition so I am running >something like: > >b = numpy.where(a>200,0,1) > >to create a new array with the changed values. However, I get a >"MemoryError" everytime I try this. I have over 3gb of RAM on my machine >(most of which is available). The process runs fine on smaller datasets. Is >there a maximum array size that numpy handles? Any alternatives/workarounds? > > > The MemoryError is a direct result when system malloc fails. Rather than use where with two scalars (you're resulting array will be int32 and therefore 4-times larger). Use b = zeros_like(a) b[a>200] = 1 which will consume less memory. -Travis |
|
From: Vikalpa J. <vj...@i3...> - 2006-10-06 13:32:29
|
Thanks Travis and Robert. I am just getting my feet wet in numpy. Both approaches i.e: b = zeros_like(a) b[a>200] = 1 or b = (a > 200).astype(numpy.uint8) avoid the memory error. Related question is that I need to test for multiple conditions on the same array and set values to 1 or 0. I see that statements like b[a>200 or a<50] = 1 do not work. So is the way to do this simply to run a separate statement in the form b[condition]= 1 for each test? Also since my output has to be a binary array, can the new array be defined as binary type or nibble, potentially reducing memory overhead? Thanks. Vikalpa -----Original Message----- From: num...@li... [mailto:num...@li...] On Behalf Of Travis Oliphant Sent: Thursday, October 05, 2006 5:44 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Memory errors > The MemoryError is a direct result when system malloc fails. Rather than use where with two scalars (you're resulting array will be int32 and therefore 4-times larger). Use b = zeros_like(a) b[a>200] = 1 which will consume less memory. -Travis ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Numpy-discussion mailing list Num...@li... https://lists.sourceforge.net/lists/listinfo/numpy-discussion |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X