Menu
▾
▴
numpy-discussion
|
From: Travis O. <oli...@ie...> - 2006-01-18 17:50:32
|
Andrew Jaffe wrote: >I have some related numpy/scipy questions: > > - I recently upgraded to the most recent SVN of numpy, without doing >likewise for scipy. I found that the numpy.test() calls failed in a >couple of places -- because *scipy* hadn't been updated with the latest >dtype updates! (I can't reproduce the errors since I've since updated >scipy.) I thought the whole point of the numpy/scipy split was to avoid >'implicit' calls of scipy from numpy, wasn't it? > > Not entirely. The issue was namespace clashes between packages. This issue has come up before and is localized to the numpy.dual module. If somebody would like to re-write the module to contain a register function that scipy hooks into to register new functions for the ones listed there, then that would be a better solution. But, I haven't had time to do it. -Travis |
|
From: Robert K. <rob...@gm...> - 2006-01-18 18:07:22
|
Fernando Perez wrote: > Anyway, I won't belabor this point any longer. I'd just like to hear from > others their opinion on this matter, and if a decision is made to go ahead > with the overwriting, at least I think the rationale for it should be well > justified (and be more than "it's convenient"). The fact that over the last > few weeks we've had several surprised questions on this is, to me, an > indicator that I'm not the one uncomfortable with this decision. I haven't followed this discussion in great detail, but I believe the current situation is this: 1) If you use numpy.dft and numpy.linalg directly, you will always get the numpy versions no matter what else is installed. 2) If you want to optionally use optimized scipy versions if they are available and regular numpy versions otherwise, then you use the functions exposed in numpy.dual. You do so at your own risk. 3) pkgload() exists to support the loading of subpackages. It does not reach into numpy.dft or numpy.linalg at all. It is not relevant to this issue. 4) There are some places in numpy that use numpy.dual. I think we can address all of your concerns by changing #4. -- Robert Kern rob...@gm... "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter |
|
From: Travis O. <oli...@ie...> - 2006-01-18 18:19:58
|
Robert Kern wrote: >Fernando Perez wrote: > > > >>Anyway, I won't belabor this point any longer. I'd just like to hear from >>others their opinion on this matter, and if a decision is made to go ahead >>with the overwriting, at least I think the rationale for it should be well >>justified (and be more than "it's convenient"). The fact that over the last >>few weeks we've had several surprised questions on this is, to me, an >>indicator that I'm not the one uncomfortable with this decision. >> >> > >I haven't followed this discussion in great detail, but I believe the current >situation is this: > >1) If you use numpy.dft and numpy.linalg directly, you will always get the numpy >versions no matter what else is installed. > >2) If you want to optionally use optimized scipy versions if they are available >and regular numpy versions otherwise, then you use the functions exposed in >numpy.dual. You do so at your own risk. > >3) pkgload() exists to support the loading of subpackages. It does not reach >into numpy.dft or numpy.linalg at all. It is not relevant to this issue. > >4) There are some places in numpy that use numpy.dual. > >I think we can address all of your concerns by changing #4. > > > This is an accurate assessment. However, I do not want to eliminate number 4 as I've mentioned before. I think there is a place for having functions that can be over-written with better versions. I agree that it could be implemented better, however, with some kind of register function instead of automatically looking in scipy... -Travis |
|
From: Robert K. <rob...@gm...> - 2006-01-18 18:31:36
|
Travis Oliphant wrote: > Robert Kern wrote: >>4) There are some places in numpy that use numpy.dual. >> >>I think we can address all of your concerns by changing #4. And actually, I think we can eat our cake and have it, too, by providing a way to restrict numpy.dual to only use numpy versions. We won't provide a way to force numpy.dual to only use some_other_version. I think Fernando's examples won't be problematic, then. > This is an accurate assessment. However, I do not want to eliminate > number 4 as I've mentioned before. I think there is a place for having > functions that can be over-written with better versions. I agree that > it could be implemented better, however, with some kind of register > function instead of automatically looking in scipy... Like egg entry_points. Please, let's not reinvent this wheel again. http://peak.telecommunity.com/DevCenter/PkgResources#entry-points -- Robert Kern rob...@gm... "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter |
|
From: Fernando P. <Fer...@co...> - 2006-01-18 19:02:19
|
Travis Oliphant wrote: >>3) pkgload() exists to support the loading of subpackages. It does not reach >>into numpy.dft or numpy.linalg at all. It is not relevant to this issue. >> >>4) There are some places in numpy that use numpy.dual. >> >>I think we can address all of your concerns by changing #4. >> >> >> > > > This is an accurate assessment. However, I do not want to eliminate > number 4 as I've mentioned before. I think there is a place for having > functions that can be over-written with better versions. I agree that > it could be implemented better, however, with some kind of register > function instead of automatically looking in scipy... Mmh, I think I'm confused then: it seemed to me that pkgload() WAS overwriting numpy names, from the messages which the environment variable controls. Is that not true? Here's a recent thread: http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2974044 where this was shown: In [3]: import scipy Overwriting fft=<function fft at 0x2000000001474668> from scipy.fftpack.basic (was <function fft at 0x2000000001394a28> from numpy.dft.fftpack) Overwriting ifft=<function ifft at 0x20000000014746e0> from scipy.fftpack.basic (was <function inverse_fft at 0x2000000001394aa0> from numpy.dft.fftpack) I understood this as 'scipy.fftpack.basic.fft overwrote numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at all? I also would like to propose that, rather than using an environment variable, pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much cleaner to say pkgload(quiet=1) or pkgload(verbose=0) than relying on users configuring env. variables for something like this. Cheers, f |
|
From: Robert K. <rob...@gm...> - 2006-01-18 20:06:08
|
Fernando Perez wrote: > Travis Oliphant wrote: > >>>3) pkgload() exists to support the loading of subpackages. It does not reach >>>into numpy.dft or numpy.linalg at all. It is not relevant to this issue. >>> >>>4) There are some places in numpy that use numpy.dual. >>> >>>I think we can address all of your concerns by changing #4. >> >>This is an accurate assessment. However, I do not want to eliminate >>number 4 as I've mentioned before. I think there is a place for having >>functions that can be over-written with better versions. I agree that >>it could be implemented better, however, with some kind of register >>function instead of automatically looking in scipy... > > Mmh, I think I'm confused then: So am I, now! > it seemed to me that pkgload() WAS overwriting > numpy names, from the messages which the environment variable controls. Is > that not true? Here's a recent thread: > > http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2974044 > > where this was shown: > > In [3]: import scipy > Overwriting fft=<function fft at 0x2000000001474668> from > scipy.fftpack.basic (was <function fft at 0x2000000001394a28> from > numpy.dft.fftpack) > Overwriting ifft=<function ifft at 0x20000000014746e0> from > scipy.fftpack.basic (was <function inverse_fft at 0x2000000001394aa0> from > numpy.dft.fftpack) [~]$ python Python 2.4.1 (#2, Mar 31 2005, 00:05:10) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import scipy scipy.>>> scipy.pkgload(verbose=2) Imports to 'scipy' namespace ---------------------------- __all__.append('io') import lib -> success Overwriting lib=<module 'scipy.lib' from '/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy-0.4.4.1307-py2.4-macosx-10.4-ppc.egg/scipy/lib/__init__.pyc'> from /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy-0.4.4.1307-py2.4-macosx-10.4-ppc.egg/scipy/lib/__init__.pyc (was <module 'numpy.lib' from '/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy-0.9.4.1849-py2.4-macosx-10.4-ppc.egg/numpy/lib/__init__.pyc'> from /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy-0.9.4.1849-py2.4-macosx-10.4-ppc.egg/numpy/lib/__init__.pyc) __all__.append('signal') __all__.append('interpolate') __all__.append('lib.lapack') import cluster -> success __all__.append('montecarlo') __all__.append('fftpack') __all__.append('sparse') __all__.append('integrate') __all__.append('optimize') __all__.append('special') import lib.blas -> success __all__.append('linalg') __all__.append('stats') >>> import numpy >>> numpy.lib <module 'numpy.lib' from '/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy-0.9.4.1849-py2.4-macosx-10.4-ppc.egg/numpy/lib/__init__.pyc'> >>> scipy.lib <module 'scipy.lib' from '/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy-0.4.4.1307-py2.4-macosx-10.4-ppc.egg/scipy/lib/__init__.pyc'> [Ignore the SVN version numbers. They are faked. I was using checkouts from last night.] > I understood this as 'scipy.fftpack.basic.fft overwrote > numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at all? If it does, then I agree with you that this should change. > I also would like to propose that, rather than using an environment variable, > pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much cleaner > to say > > pkgload(quiet=1) or pkgload(verbose=0) > > than relying on users configuring env. variables for something like this. It does take a verbose keyword argument. -- Robert Kern rob...@gm... "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter |
|
From: Pearu P. <pe...@sc...> - 2006-01-18 22:14:18
|
On Wed, 18 Jan 2006, Fernando Perez wrote: > Mmh, I think I'm confused then: it seemed to me that pkgload() WAS > overwriting numpy names, from the messages which the environment variable > controls. Is that not true? Here's a recent thread: > > http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2974044 pkgload() is overwriting numpy names in *scipy namespace*. To be explicit, the following is going on in scipy/__init__.py when pkgload is called, pseudocode follows: from numpy import * # imports fft old_fft = fft from scipy.fftpack import fft print 'Overwriting',old_fft,'with',fft del old_fft And nothing else! So, scipy.fft, that was numpy.fft, is set to scipy.fftpack.fft. numpy.fft remains the same. ... > I understood this as 'scipy.fftpack.basic.fft overwrote > numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at > all? No! See above. > I also would like to propose that, rather than using an environment variable, > pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much > cleaner to say > > pkgload(quiet=1) or pkgload(verbose=0) > > than relying on users configuring env. variables for something like this. pkgload has already verbose kwd argumend. See ?pkgload for more information. Pearu |
|
From: Fernando P. <Fer...@co...> - 2006-01-18 22:28:59
|
Pearu Peterson wrote: >>I understood this as 'scipy.fftpack.basic.fft overwrote >>numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at >>all? > > > No! See above. Ah, OK. Thanks for the clarification, I misunderstood the message. >>I also would like to propose that, rather than using an environment variable, >>pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much >>cleaner to say >> >>pkgload(quiet=1) or pkgload(verbose=0) >> >>than relying on users configuring env. variables for something like this. > > > > pkgload has already verbose kwd argumend. See ?pkgload for more > information. What is this '?foo' syntax you speak of? Certainly not python... ;-) Sorry for not checking, I just don't have new-scipy on my work machine, only at home. Cheers, f |
|
From: Ed S. <sch...@ft...> - 2006-01-18 19:00:47
|
Robert Kern wrote:
>Fernando Perez wrote:
>
>
>
>>Anyway, I won't belabor this point any longer. I'd just like to hear from
>>others their opinion on this matter, and if a decision is made to go ahead
>>with the overwriting, at least I think the rationale for it should be well
>>justified (and be more than "it's convenient"). The fact that over the last
>>few weeks we've had several surprised questions on this is, to me, an
>>indicator that I'm not the one uncomfortable with this decision.
>>
>>
>
>I haven't followed this discussion in great detail, but I believe the current
>situation is this:
>
>1) If you use numpy.dft and numpy.linalg directly, you will always get the numpy
>versions no matter what else is installed.
>
>2) If you want to optionally use optimized scipy versions if they are available
>and regular numpy versions otherwise, then you use the functions exposed in
>numpy.dual. You do so at your own risk.
>
>3) pkgload() exists to support the loading of subpackages. It does not reach
>into numpy.dft or numpy.linalg at all. It is not relevant to this issue.
>
>4) There are some places in numpy that use numpy.dual.
>
>I think we can address all of your concerns by changing #4.
>
>
I've been battling for half an hour trying to import scipy.linalg. Is
this one of Fernando's scary predictions coming true? I get:
>>> from scipy import linalg
>>> dir(linalg)
['Heigenvalues', 'Heigenvectors', 'LinAlgError', 'ScipyTest',
'__builtins__', '__doc__', '__file__', '__name__', '__path__',
'cholesky', 'cholesky_decomposition', 'det', 'determinant', 'eig',
'eigenvalues', 'eigenvectors', 'eigh', 'eigvals', 'eigvalsh',
'generalized_inverse', 'inv', 'inverse', 'lapack_lite', 'linalg',
'linear_least_squares', 'lstsq', 'pinv', 'singular_value_decomposition',
'solve', 'solve_linear_equations', 'svd', 'test']
>>> linalg.__file__
'/home/schofield/Tools/lib/python2.4/site-packages/numpy/linalg/__init__.pyc'
This is the linalg package from numpy, not scipy. It's missing my
favourite pinv2 function. What is going on?! I've just cleaned
everything out and built from the latest SVN revisions. Using
pkgload('linalg') doesn't seem to help:
>>> import scipy
>>> scipy.pkgload('linalg')
>>> linalg.__file__
Traceback (most recent call last):
File "<stdin>", line 1, in ?
NameError: name 'linalg' is not defined
>>> scipy.linalg.__file__
'/home/schofield/Tools/lib/python2.4/site-packages/numpy/linalg/__init__.pyc'
The only thing that helps is calling pkgload() with no arguments:
>>> import scipy
>>> scipy.pkgload()
Overwriting lib=<module 'scipy.lib' from
'/home/schofield/Tools/lib/python2.4/site-packages/scipy/lib/__init__.pyc'>
from
/home/schofield/Tools/lib/python2.4/site-packages/scipy/lib/__init__.pyc
(was <module 'numpy.lib' from
'/home/schofield/Tools/lib/python2.4/site-packages/numpy/lib/__init__.pyc'>
from
/home/schofield/Tools/lib/python2.4/site-packages/numpy/lib/__init__.pyc)
>>> scipy.linalg.__file__
'/home/schofield/Tools/lib/python2.4/site-packages/scipy/linalg/__init__.pyc'
Unless I'm doing something very stupid, there seem to be multiple
sources of evil here. First, numpy's linalg package is available from
the scipy namespace, which seems like a recipe for Ed's madness, since
he can't find his pinv2() function. Second, scipy.pkgload('linalg')
silently fails to make any visible difference. This is probably a
simple bug. Third, ..., there is no third. But bad things usually
come in threes.
-- Ed
|
|
From: Robert K. <rob...@gm...> - 2006-01-18 20:08:48
|
Ed Schofield wrote:
> Unless I'm doing something very stupid, there seem to be multiple
> sources of evil here. First, numpy's linalg package is available from
> the scipy namespace, which seems like a recipe for Ed's madness, since
> he can't find his pinv2() function. Second, scipy.pkgload('linalg')
> silently fails to make any visible difference. This is probably a
> simple bug.
It's certainly not intended behavior. The process by which pkgload() determines
where it is being called from is messing up. pkgload lives in
numpy._import_tools, but it is also exposed in the scipy namespace, too.
Please file a bug report and assign it to Pearu.
--
Robert Kern
rob...@gm...
"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter
|