Menu
▾
▴
numpy-discussion
|
From: Fernando P. <fpe...@gm...> - 2006-06-08 22:55:24
|
Hi all,
I'm starting the transition of a large code from Numeric to numpy, so
I am now doing a fresh build with a lot more care than before,
actually reading all the intermediate messages. I am a bit puzzled
and could use some help.
This is all on an ubuntu dapper box with the atlas-sse2 packages (and
everything else recommended installed). By running as suggested in
the scipy readme:
python ~/tmp/local/lib/python2.4/site-packages/numpy/distutils/system_info.py
I get the following message at some point:
====================================
atlas_info:
( library_dirs = /usr/local/lib:/usr/lib )
( paths: /usr/lib/atlas,/usr/lib/sse2 )
looking libraries f77blas,cblas,atlas in /usr/local/lib but found None
looking libraries f77blas,cblas,atlas in /usr/local/lib but found None
looking libraries lapack_atlas in /usr/local/lib but found None
looking libraries lapack_atlas in /usr/local/lib but found None
looking libraries f77blas,cblas,atlas in /usr/lib/atlas but found None
looking libraries f77blas,cblas,atlas in /usr/lib/atlas but found None
looking libraries lapack_atlas in /usr/lib/atlas but found None
looking libraries lapack_atlas in /usr/lib/atlas but found None
( paths: /usr/lib/sse2/libf77blas.so )
( paths: /usr/lib/sse2/libcblas.so )
( paths: /usr/lib/sse2/libatlas.so )
( paths: /usr/lib/sse2/liblapack_atlas.so )
looking libraries lapack in /usr/lib/sse2 but found None
looking libraries lapack in /usr/lib/sse2 but found None
looking libraries f77blas,cblas,atlas in /usr/lib but found None
looking libraries f77blas,cblas,atlas in /usr/lib but found None
looking libraries lapack_atlas in /usr/lib but found None
looking libraries lapack_atlas in /usr/lib but found None
system_info.atlas_info
( include_dirs = /usr/local/include:/usr/include )
( paths: /usr/include/atlas_misc.h,/usr/include/atlas_enum.h,/usr/include/atlas_aux.h,/usr/include/atlas_type.h
)
/usr/local/installers/src/scipy/numpy/numpy/distutils/system_info.py:870:
UserWarning:
*********************************************************************
Could not find lapack library within the ATLAS installation.
*********************************************************************
warnings.warn(message)
( library_dirs = /usr/local/lib:/usr/lib )
( paths: /usr/lib/atlas,/usr/lib/sse2 )
FOUND:
libraries = ['f77blas', 'cblas', 'atlas']
library_dirs = ['/usr/lib/sse2']
language = c
define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
====================================
What I find very puzzling here is that later on, the following goes by:
lapack_atlas_info:
( library_dirs = /usr/local/lib:/usr/lib )
( paths: /usr/lib/atlas,/usr/lib/sse2 )
looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/local/lib
but found None
looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/local/lib
but found None
looking libraries lapack_atlas in /usr/local/lib but found None
looking libraries lapack_atlas in /usr/local/lib but found None
looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib/atlas
but found None
looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib/atlas
but found None
looking libraries lapack_atlas in /usr/lib/atlas but found None
looking libraries lapack_atlas in /usr/lib/atlas but found None
( paths: /usr/lib/sse2/liblapack_atlas.so )
( paths: /usr/lib/sse2/libf77blas.so )
( paths: /usr/lib/sse2/libcblas.so )
( paths: /usr/lib/sse2/libatlas.so )
( paths: /usr/lib/sse2/liblapack_atlas.so )
looking libraries lapack in /usr/lib/sse2 but found None
looking libraries lapack in /usr/lib/sse2 but found None
looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib but found None
looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib but found None
looking libraries lapack_atlas in /usr/lib but found None
looking libraries lapack_atlas in /usr/lib but found None
system_info.lapack_atlas_info
( include_dirs = /usr/local/include:/usr/include )
( paths: /usr/include/atlas_misc.h,/usr/include/atlas_enum.h,/usr/include/atlas_aux.h,/usr/include/atlas_type.h
)
( library_dirs = /usr/local/lib:/usr/lib )
( paths: /usr/lib/atlas,/usr/lib/sse2 )
FOUND:
libraries = ['lapack_atlas', 'f77blas', 'cblas', 'atlas']
library_dirs = ['/usr/lib/sse2']
language = c
define_macros = [('ATLAS_WITH_LAPACK_ATLAS', None)]
==============================================
Does the second mean that it /is/ finding the right libraries? Since
the first search in atlas_info is also printing
( paths: /usr/lib/sse2/liblapack_atlas.so )
I don't quite understand why it then reports the warning.
For reference, here's the content of the relevant directories on my system:
==============================================
longs[sse2]> ls /usr/lib/sse2
libatlas.a libcblas.a libf77blas.a liblapack_atlas.a
libatlas.so@ libcblas.so@ libf77blas.so@ liblapack_atlas.so@
libatlas.so.3@ libcblas.so.3@ libf77blas.so.3@ liblapack_atlas.so.3@
libatlas.so.3.0 libcblas.so.3.0 libf77blas.so.3.0 liblapack_atlas.so.3.0
longs[sse2]> ls /usr/lib/atlas/sse2/
libblas.a libblas.so.3@ liblapack.a liblapack.so.3@
libblas.so@ libblas.so.3.0 liblapack.so@ liblapack.so.3.0
==============================================
In summary, I don't really know if this is actually finding what it
wants or not, given the two messages.
Cheers,
f
ps - it's worth mentioning that the sequence:
python ~/tmp/local/lib/python2.4/site-packages/numpy/distutils/system_info.py
gets itself into a nasty recursion where it fires the interactive
session 3 times in a row. And in doing so, it splits its own output
in a funny way:
[...]
blas_opt_info:
========================================================================
Starting interactive session
------------------------------------------------------------------------
Tasks:
i - Show python/platform/machine information
ie - Show environment information
c - Show C compilers information
c<name> - Set C compiler (current:None)
f - Show Fortran compilers information
f<name> - Set Fortran compiler (current:None)
e - Edit proposed sys.argv[1:].
Task aliases:
0 - Configure
1 - Build
2 - Install
2<prefix> - Install with prefix.
3 - Inplace build
4 - Source distribution
5 - Binary distribution
Proposed sys.argv =
['/home/fperez/tmp/local/lib/python2.4/site-packages/numpy/distutils/system_info.py']
Choose a task (^D to quit, Enter to continue with setup):
##### msg:
( library_dirs = /usr/local/lib:/usr/lib )
FOUND:
libraries = ['f77blas', 'cblas', 'atlas']
library_dirs = ['/usr/lib/sse2']
language = c
define_macros = [('NO_ATLAS_INFO', 2)]
=================
I tried to fix it, but the call sequence in that code is convoluted
enough that after a few 'import traceback;traceback.print_stack()'
tries I sort of gave up. That code is rather (how can I say this
nicely) pasta-like :), and thoroughly uncommented, so I'm afraid I
won't be able to contribute a cleanup here.
I think this tool should run by default in a mode with NO attempt to
fire a command-line subsystem of its own, so users can simply run
python /path/to/system_info > system_info.log
for further analysis.
|
|
From: David M. C. <co...@ph...> - 2006-06-08 23:06:48
|
On Thu, 8 Jun 2006 16:48:27 -0600 "Fernando Perez" <fpe...@gm...> wrote: [snip] > I tried to fix it, but the call sequence in that code is convoluted > enough that after a few 'import traceback;traceback.print_stack()' > tries I sort of gave up. That code is rather (how can I say this > nicely) pasta-like :), and thoroughly uncommented, so I'm afraid I > won't be able to contribute a cleanup here. I think the whole numpy.distutils could use a good cleanup ... > I think this tool should run by default in a mode with NO attempt to > fire a command-line subsystem of its own, so users can simply run > > python /path/to/system_info > system_info.log > > for further analysis. Agree; I'll look at it. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |co...@ph... |
|
From: Fernando P. <fpe...@gm...> - 2006-06-08 23:12:01
|
On 6/8/06, David M. Cooke <co...@ph...> wrote: > Agree; I'll look at it. Many thanks. I'm sorry not to help, but I have a really big fish to fry right now, and can't commit to the diversion this would mean. Cheers, f |
|
From: Dan C. <jd...@uw...> - 2006-06-09 01:23:22
|
I don't know if it's related, but I've found on my Debian system that whenever I want to compile something that uses the atlas library, I need to put -L/usr/lib/sse2 on the gcc line, even though everything seems to indicate that the linker has been told to look there already. It could be that Ubuntu has a similar issue, and that it is affecting your build. Dan |
|
From: Fernando P. <fpe...@gm...> - 2006-06-09 01:39:46
|
On 6/8/06, Dan Christensen <jd...@uw...> wrote: > I don't know if it's related, but I've found on my Debian system that > whenever I want to compile something that uses the atlas library, I > need to put -L/usr/lib/sse2 on the gcc line, even though everything > seems to indicate that the linker has been told to look there already. > It could be that Ubuntu has a similar issue, and that it is affecting > your build. mmh, given how green I am in the ubuntu world, you may well be right. But my original question went before any linking happens, since I was just posting the messages from numpy's system_info, which doesn't attempt to link at anything, it just does a static filesystem analysis. So perhaps there is more than one issue here. I'm just trying to clarify, from the given messages (which I found a bit confusing) whether all the atlas/sse2 stuff is actually being picked up or not, at least as far as numpy thinks it is. Cheers, f |
|
From: Simon B. <si...@ar...> - 2006-06-09 02:09:25
|
On Thu, 8 Jun 2006 16:48:27 -0600 "Fernando Perez" <fpe...@gm...> wrote: > > In summary, I don't really know if this is actually finding what it > wants or not, given the two messages. I just went through this on debian sarge which is similar. I put this in site.cgf: [atlas] library_dirs = /usr/lib/atlas/ atlas_libs = lapack, blas Then I needed to set LD_LIBRARY_PATH to point to /usr/lib/atlas/sse2. $ env LD_LIBRARY_PATH=/usr/lib/atlas/sse2 python2.4 Python 2.4.3 (#4, Jun 5 2006, 19:07:06) [GCC 3.4.1 (Debian 3.4.1-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> [1]+ Stopped env LD_LIBRARY_PATH=/usr/lib/atlas/sse2 python2.4 Look in /proc/PID/maps for the relevant libs: $ ps -a|grep python ... 16953 pts/64 00:00:00 python2.4 $ grep atlas /proc/16953/maps b6fa7000-b750e000 r-xp 00000000 00:0c 1185402 /usr/lib/atlas/sse2/libblas.so.3.0 b750e000-b7513000 rwxp 00567000 00:0c 1185402 /usr/lib/atlas/sse2/libblas.so.3.0 b7513000-b7a58000 r-xp 00000000 00:0c 1185401 /usr/lib/atlas/sse2/liblapack.so.3.0 b7a58000-b7a5b000 rwxp 00545000 00:0c 1185401 /usr/lib/atlas/sse2/liblapack.so.3.0 $ But to really test this is working I ran python under gdb and set a break point on cblas_dgemm. Then a call to numpy.dot should break inside the sse2/liblapack.so.3.0. (also it's a lot faster with the sse2 dgemm) $ env LD_LIBRARY_PATH=/usr/lib/atlas/sse2 gdb python2.4 GNU gdb 6.1-debian Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-linux"...Using host libthread_db library "/lib/tls/libthread_db.so.1". (gdb) break cblas_dgemm Function "cblas_dgemm" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (cblas_dgemm) pending. (gdb) run Starting program: /home/users/simonb/bin/python2.4 [Thread debugging using libthread_db enabled] [New Thread -1210476000 (LWP 17557)] Python 2.4.3 (#4, Jun 5 2006, 19:07:06) [GCC 3.4.1 (Debian 3.4.1-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Breakpoint 2 at 0xb7549db0 Pending breakpoint "cblas_dgemm" resolved <------- import numpy is in my pythonstartup >>> a=numpy.empty((1024,1024),'d') >>> b=numpy.empty((1024,1024),'d') >>> numpy.dot(a,b) [Switching to Thread -1210476000 (LWP 17557)] Breakpoint 2, 0xb7549db0 in cblas_dgemm () from /usr/lib/atlas/sse2/liblapack.so.3 (gdb) bingo. Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com |
|
From: Fernando P. <fpe...@gm...> - 2006-06-09 02:26:01
|
On 6/8/06, Simon Burton <si...@ar...> wrote:
> On Thu, 8 Jun 2006 16:48:27 -0600
> "Fernando Perez" <fpe...@gm...> wrote:
>
> >
> > In summary, I don't really know if this is actually finding what it
> > wants or not, given the two messages.
>
> I just went through this on debian sarge which is similar.
>
> I put this in site.cgf:
>
> [atlas]
> library_dirs = /usr/lib/atlas/
> atlas_libs = lapack, blas
>
> Then I needed to set LD_LIBRARY_PATH to point to /usr/lib/atlas/sse2.
[...]
> But to really test this is working I ran python under gdb and set
> a break point on cblas_dgemm. Then a call to numpy.dot should
> break inside the sse2/liblapack.so.3.0.
>
> (also it's a lot faster with the sse2 dgemm)
>
> $ env LD_LIBRARY_PATH=/usr/lib/atlas/sse2 gdb python2.4
OK, thanks a LOT for that gdb trick: it provides a very nice way to
understand what's actually going on. self.note("really, learn better
use of gdb")
Using that, though, it would then seem as if the build DID
successfully find everything without any further action on my part:
longs[dist]> gdb python
GNU gdb 6.4-debian
...
(gdb) break cblas_dgemm
Function "cblas_dgemm" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (cblas_dgemm) pending.
(gdb) run
Starting program: /usr/bin/python
...
Python 2.4.3 (#2, Apr 27 2006, 14:43:58)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(no debugging symbols found)
>>> import numpy
Breakpoint 2 at 0x40429860
Pending breakpoint "cblas_dgemm" resolved
>>> a=numpy.empty((1024,1024),'d')
>>> b=numpy.empty((1024,1024),'d')
>>> numpy.dot(a,b)
[Switching to Thread 1075428416 (LWP 3919)]
Breakpoint 2, 0x40429860 in cblas_dgemm () from /usr/lib/sse2/libcblas.so.3
======================================================
Note that on my system, LD_LIBRARY_PATH does NOT contain that dir:
longs[dist]> env | grep LD_LIB
LD_LIBRARY_PATH=/usr/local/lf9560/lib:/usr/local/intel/mkl/8.0.2/lib/32:/usr/local/intel/compiler90/lib:/home/fperez/usr/lib:/home/fperez/usr/local/lib:
and I built everything with a plain
setup.py install --prefix=~/tmp/local
without /any/ tweaks to site.cfg, no LD_LIBRARY_PATH modifications or
anything else. I just installed atlas-sse2* and lapack3*, but NOT
refblas3*.
Basically it seems that the build process does the right thing out of
the box, and the warning is spurious. Since I was being extra-careful
in this build, I didn't want to let any warning of that kind go
unchecked.
It might still be worth fixing that warning to prevent others from
going on a similar wild goose chase, but I'm not comfortable touching
that code (I don't know if anyone besides Pearu is).
Thanks for the help!
Cheers,
f
|
|
From: Stephan T. <st...@si...> - 2006-06-09 08:06:35
|
> ====================================
> atlas_info:
> ( library_dirs = /usr/local/lib:/usr/lib )
> ( paths: /usr/lib/atlas,/usr/lib/sse2 )
> looking libraries f77blas,cblas,atlas in /usr/local/lib but found None
> looking libraries f77blas,cblas,atlas in /usr/local/lib but found None
(.. more of these...)
Some of these and similar spurious warnings can be eliminated by
replacing the calls to check_libs in system_info.py with calls to
check_libs2. Currently these warnings are generated for each file
extension that is tested (".so", ".a"...) Alternatively, the warnings
could be made more informative.
Many of the other warnings could be eliminated by consolidating the
various BLAS/LAPACK options.
If anyone is manipulating the build system, could he please apply the
patch from #114 fixing the Windows build?
> I tried to fix it, but the call sequence in that code is convoluted
> enough that after a few 'import traceback;traceback.print_stack()'
> tries I sort of gave up. That code is rather (how can I say this
> nicely) pasta-like :), and thoroughly uncommented, so I'm afraid I
> won't be able to contribute a cleanup here.
Even if you spent enough time to understand the existing code, you
probably wouldn't have a chance to clean up the code because any small
change could break some obscure platform/compiler/library combination.
Moreover, changes could break the build of scipy and other libraries
depending on Numpy-distutils.
If you really wanted to rewrite the build code, you'd need to specify a
minimum set of supported platform and library combinations, have each of
them available for testing and deliberately risk breaking any other
platform.
Regards,
Stephan
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X