Thread: [Pypar-developers] PyPar installation
Brought to you by:
uniomni
Menu
▾
▴
pypar-developers
|
From: Ole N. <ole...@gm...> - 2009-12-18 05:14:47
|
Can anyone help out? I don't understand where the -lopa comes from. Cheers and thanks Ole ---------- Forwarded message ---------- From: Romesh Abeysuriya <rom...@gm...> Date: Fri, Dec 18, 2009 at 11:26 AM Subject: PyPar installation To: ole...@gm... Cc: "Dr. Jay Larson (ANU)" <jay...@an...> Hi Ole, We are trying to build PyPar on a couple of platforms, and are encountering the same problem. The two architectures are: 64-bit intels (one Linux, the other OS X Snow Leopard). The common error we encounter in both is associated with this message: gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so sh: -c: line 0: unexpected EOF while looking for matching `'' sh: -c: line 1: syntax error: unexpected end of file _exec_command_posix failed (status=512) /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' /usr/bin/bash: -c: line 1: syntax error: unexpected end of file /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' /usr/bin/bash: -c: line 1: syntax error: unexpected end of file error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit status 2 The main problem stems from this -lopa' element. Grepping around your source for "lopa" or "opa" didn't give us a clue how to fix this problem. Additionally, on Snow Leopard, problems arise because PyPar is trying to create a Universal Binary, when only x86_64 MPICH libraries have been compiled, so we are looking for a way to remove the extra -arch switches. We really would like to use PyPar in our project, so if you could please advise us how to fix these problems, we'd be really grateful. For reference, here's some version information on our software stack we're using to build PyPar: MPICH2-1.2.1 Python 2.6.1 (OS X) Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X) GCC 4.1.2 (Linux), 4.2.1 (OS X) PyPar 2.1.0.66 Thanks and all the best... Romesh Abeysuriya Jay Larson |
|
From: Ole N. <ole...@gm...> - 2009-12-19 01:27:02
|
Hi Romesh When I build pypar on our 64 bit Intel cluster (Ubuntu 9.04) with OpenMPI, the compile string looks like this: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -I/usr/lib/python2.6/dist-packages/numpy/core/include -I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o -fPIC gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lopen-pal -ldl -lnsl -lutil -lm -o build/lib.linux-x86_64-2.6/pypar/mpiext.so and all is well. The same is true on a 32 bit Ubuntu 9.10 system. Could I ask you to try to run python compile_pypar_locally.py That will compile the mpiext.so using mpicc directly (bypassing distutils) and the compile output should look something like this. nielso@alamba:~/sandpit/pypar/source$ python compile_pypar_locally.py mpicc -c mpiext.c -I/usr/include/python2.6 -I/usr/lib/python2.6/dist-packages/numpy/core/include -o mpiext.o -Wall -fPIC mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium MPI initialised OK Then you should be able to run the unit tests mpirun -np 2 python test_pypar.py Please let us know how you go. Cheers Ole un...@in... On Fri, Dec 18, 2009 at 3:26 PM, Romesh Abeysuriya <rom...@gm...>wrote: > Hi Ole, > > We are trying to build PyPar on a couple of platforms, and are encountering > the same problem. > > The two architectures are: 64-bit intels (one Linux, the other OS X Snow > Leopard). The common > error we encounter in both is associated with this message: > > gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o > -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt > -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so > sh: -c: line 0: unexpected EOF while looking for matching `'' > sh: -c: line 1: syntax error: unexpected end of file > _exec_command_posix failed (status=512) > /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' > /usr/bin/bash: -c: line 1: syntax error: unexpected end of file > /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' > /usr/bin/bash: -c: line 1: syntax error: unexpected end of file > error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o > -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt > -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit > status 2 > > The main problem stems from this -lopa' element. Grepping around your > source for "lopa" or "opa" didn't give us a clue how to fix this problem. > > Additionally, on Snow Leopard, problems arise because PyPar is trying to > create a Universal Binary, when only x86_64 MPICH libraries have been > compiled, so we are looking for a way to remove the extra -arch switches. > > We really would like to use PyPar in our project, so if you could please > advise us how to fix these problems, we'd be really grateful. > > For reference, here's some version information on our software stack we're > using to build PyPar: > > MPICH2-1.2.1 > Python 2.6.1 (OS X) > Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X) > GCC 4.1.2 (Linux), 4.2.1 (OS X) > PyPar 2.1.0.66 > > > Thanks and all the best... > > Romesh Abeysuriya > Jay Larson > > > |
|
From: Jay L. <jw...@rs...> - 2009-12-19 06:19:34
|
Hi Ole,
Thanks for the rapid response...and on the weekend--wow!
Since we are all in Canberra, we should meet up sometime and I will buy you some drinks.
All the Best,
Jay
Sent from Jay's BlackBerry®
-----Original Message-----
From: Ole Nielsen <ole...@gm...>
Date: Sat, 19 Dec 2009 12:26:50
To: Romesh Abeysuriya<rom...@gm...>
Cc: Dr. Jay Larson (ANU)<jay...@an...>; Pypar-developers<Pyp...@li...>
Subject: Re: PyPar installation
Hi Romesh
When I build pypar on our 64 bit Intel cluster (Ubuntu 9.04) with OpenMPI,
the compile string looks like this:
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -I/usr/lib/openmpi/include
-I/usr/lib/openmpi/include/openmpi
-I/usr/lib/python2.6/dist-packages/numpy/core/include
-I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o
-fPIC
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib/openmpi/lib -lmpi -lopen-rte
-lopen-pal -ldl -lnsl -lutil -lm -o
build/lib.linux-x86_64-2.6/pypar/mpiext.so
and all is well. The same is true on a 32 bit Ubuntu 9.10 system.
Could I ask you to try to run
python compile_pypar_locally.py
That will compile the mpiext.so using mpicc directly (bypassing distutils)
and the compile output should look something like this.
nielso@alamba:~/sandpit/pypar/source$ python compile_pypar_locally.py
mpicc -c mpiext.c -I/usr/include/python2.6
-I/usr/lib/python2.6/dist-packages/numpy/core/include -o mpiext.o -Wall
-fPIC
mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium
MPI initialised OK
Then you should be able to run the unit tests
mpirun -np 2 python test_pypar.py
Please let us know how you go.
Cheers
Ole
un...@in...
On Fri, Dec 18, 2009 at 3:26 PM, Romesh Abeysuriya <rom...@gm...>wrote:
> Hi Ole,
>
> We are trying to build PyPar on a couple of platforms, and are encountering
> the same problem.
>
> The two architectures are: 64-bit intels (one Linux, the other OS X Snow
> Leopard). The common
> error we encounter in both is associated with this message:
>
> gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o
> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt
> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so
> sh: -c: line 0: unexpected EOF while looking for matching `''
> sh: -c: line 1: syntax error: unexpected end of file
> _exec_command_posix failed (status=512)
> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file
> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file
> error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o
> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt
> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit
> status 2
>
> The main problem stems from this -lopa' element. Grepping around your
> source for "lopa" or "opa" didn't give us a clue how to fix this problem.
>
> Additionally, on Snow Leopard, problems arise because PyPar is trying to
> create a Universal Binary, when only x86_64 MPICH libraries have been
> compiled, so we are looking for a way to remove the extra -arch switches.
>
> We really would like to use PyPar in our project, so if you could please
> advise us how to fix these problems, we'd be really grateful.
>
> For reference, here's some version information on our software stack we're
> using to build PyPar:
>
> MPICH2-1.2.1
> Python 2.6.1 (OS X)
> Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X)
> GCC 4.1.2 (Linux), 4.2.1 (OS X)
> PyPar 2.1.0.66
>
>
> Thanks and all the best...
>
> Romesh Abeysuriya
> Jay Larson
>
>
>
|
|
From: Romesh A. <rom...@gm...> - 2009-12-19 03:08:36
|
Hi Ole,
That seems to fix compilation on the OS X machine- compiling gives
romesh: source > python compile_pypar_locally.py
mpicc -c mpiext.c
-I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
-I/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/core/include
-o mpiext.o -Wall
mpiext.c: In function ‘type_map’:
mpiext.c:127: warning: return makes integer from pointer without a cast
mpiext.c: In function ‘op_map’:
mpiext.c:168: warning: return makes integer from pointer without a cast
mpicc -bundle -flat_namespace -undefined suppress mpiext.o -o mpiext.so
MPI initialised OK
However, test_pypar.py does not run correctly. An mpd session is opened on
the local machine with
romesh: source > mpdboot --totalnum=1 --ncpus=2
And then test_pypar.py is run using
romesh: source > mpirun -np 2 python test_pypar.py
Pypar (version 2.1.0) initialised MPI OK with 2 processors
I am processor 0 of 2 on node Romesh-MBP-3.local
I am processor 1 of 2 on node Romesh-MBP-3.local
Raw communication of numeric integer arrays OK
...
...
...
Raw reduce using pypar.BXOR OK
Traceback (most recent call last):
File "test_pypar.py", line 953, in <module>
assert status.error == 0
AssertionError
rank 0 in job 1 Romesh-MBP-3.local_49875 caused collective abort of all
ranks
exit status of rank 0: return code 1
with the error originating at line 953, and status variable being set in
line 941
941 B, status = pypar.receive(numproc-1, return_status = True)
953 assert status.error == 0
Any ideas on what might be causing this?
We will try this on the Linux machine on Monday
Thanks for the quick reply!
Romesh
2009/12/19 Ole Nielsen <ole...@gm...>
> Hi Romesh
>
> When I build pypar on our 64 bit Intel cluster (Ubuntu 9.04) with OpenMPI,
> the compile string looks like this:
>
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -I/usr/lib/openmpi/include
> -I/usr/lib/openmpi/include/openmpi
> -I/usr/lib/python2.6/dist-packages/numpy/core/include
> -I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o
> -fPIC
> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib/openmpi/lib -lmpi -lopen-rte
> -lopen-pal -ldl -lnsl -lutil -lm -o
> build/lib.linux-x86_64-2.6/pypar/mpiext.so
>
> and all is well. The same is true on a 32 bit Ubuntu 9.10 system.
>
>
> Could I ask you to try to run
> python compile_pypar_locally.py
>
> That will compile the mpiext.so using mpicc directly (bypassing distutils)
> and the compile output should look something like this.
>
> nielso@alamba:~/sandpit/pypar/source$ python compile_pypar_locally.py
> mpicc -c mpiext.c -I/usr/include/python2.6
> -I/usr/lib/python2.6/dist-packages/numpy/core/include -o mpiext.o -Wall
> -fPIC
> mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium
> MPI initialised OK
>
>
> Then you should be able to run the unit tests
> mpirun -np 2 python test_pypar.py
>
> Please let us know how you go.
>
> Cheers
> Ole
>
>
>
>
> un...@in...
>
>
> On Fri, Dec 18, 2009 at 3:26 PM, Romesh Abeysuriya <rom...@gm...>wrote:
>
>> Hi Ole,
>>
>> We are trying to build PyPar on a couple of platforms, and are
>> encountering the same problem.
>>
>> The two architectures are: 64-bit intels (one Linux, the other OS X Snow
>> Leopard). The common
>> error we encounter in both is associated with this message:
>>
>> gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o
>> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt
>> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so
>> sh: -c: line 0: unexpected EOF while looking for matching `''
>> sh: -c: line 1: syntax error: unexpected end of file
>> _exec_command_posix failed (status=512)
>> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
>> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file
>> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
>> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file
>> error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o
>> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt
>> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit
>> status 2
>>
>> The main problem stems from this -lopa' element. Grepping around your
>> source for "lopa" or "opa" didn't give us a clue how to fix this problem.
>>
>> Additionally, on Snow Leopard, problems arise because PyPar is trying to
>> create a Universal Binary, when only x86_64 MPICH libraries have been
>> compiled, so we are looking for a way to remove the extra -arch switches.
>>
>> We really would like to use PyPar in our project, so if you could please
>> advise us how to fix these problems, we'd be really grateful.
>>
>> For reference, here's some version information on our software stack we're
>> using to build PyPar:
>>
>> MPICH2-1.2.1
>> Python 2.6.1 (OS X)
>> Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X)
>> GCC 4.1.2 (Linux), 4.2.1 (OS X)
>> PyPar 2.1.0.66
>>
>>
>> Thanks and all the best...
>>
>> Romesh Abeysuriya
>> Jay Larson
>>
>>
>>
>
|
|
From: Ole N. <ole...@gm...> - 2009-12-19 18:08:46
|
OK - good that it compiles. I too noticed a recent problem with reduce: It has worked for many years, but with OpenMPI version 1.3.2-3ubuntu1 I too had failed tests to do with reduce. Everything else is OK, though. I'll investigate. Beer in Canberra sounds great. Are having drinkies at the ANUSF before Xmas? Cheers Ole On Sat, Dec 19, 2009 at 2:08 PM, Romesh Abeysuriya <rom...@gm...>wrote: > Hi Ole, > > That seems to fix compilation on the OS X machine- compiling gives > > romesh: source > python compile_pypar_locally.py > mpicc -c mpiext.c > -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 > -I/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/core/include > -o mpiext.o -Wall > mpiext.c: In function ‘type_map’: > mpiext.c:127: warning: return makes integer from pointer without a cast > mpiext.c: In function ‘op_map’: > mpiext.c:168: warning: return makes integer from pointer without a cast > mpicc -bundle -flat_namespace -undefined suppress mpiext.o -o mpiext.so > MPI initialised OK > > However, test_pypar.py does not run correctly. An mpd session is opened on > the local machine with > > romesh: source > mpdboot --totalnum=1 --ncpus=2 > > And then test_pypar.py is run using > > romesh: source > mpirun -np 2 python test_pypar.py > Pypar (version 2.1.0) initialised MPI OK with 2 processors > I am processor 0 of 2 on node Romesh-MBP-3.local > I am processor 1 of 2 on node Romesh-MBP-3.local > Raw communication of numeric integer arrays OK > ... > ... > ... > Raw reduce using pypar.BXOR OK > Traceback (most recent call last): > File "test_pypar.py", line 953, in <module> > assert status.error == 0 > AssertionError > rank 0 in job 1 Romesh-MBP-3.local_49875 caused collective abort of all > ranks > exit status of rank 0: return code 1 > > > with the error originating at line 953, and status variable being set in > line 941 > > 941 B, status = pypar.receive(numproc-1, return_status = True) > 953 assert status.error == 0 > > Any ideas on what might be causing this? > > We will try this on the Linux machine on Monday > > Thanks for the quick reply! > > Romesh > > 2009/12/19 Ole Nielsen <ole...@gm...> > > Hi Romesh >> >> When I build pypar on our 64 bit Intel cluster (Ubuntu 9.04) with OpenMPI, >> the compile string looks like this: >> >> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall >> -Wstrict-prototypes -fPIC -I/usr/lib/openmpi/include >> -I/usr/lib/openmpi/include/openmpi >> -I/usr/lib/python2.6/dist-packages/numpy/core/include >> -I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o >> -fPIC >> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions >> build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib/openmpi/lib -lmpi -lopen-rte >> -lopen-pal -ldl -lnsl -lutil -lm -o >> build/lib.linux-x86_64-2.6/pypar/mpiext.so >> >> and all is well. The same is true on a 32 bit Ubuntu 9.10 system. >> >> >> Could I ask you to try to run >> python compile_pypar_locally.py >> >> That will compile the mpiext.so using mpicc directly (bypassing distutils) >> and the compile output should look something like this. >> >> nielso@alamba:~/sandpit/pypar/source$ python compile_pypar_locally.py >> mpicc -c mpiext.c -I/usr/include/python2.6 >> -I/usr/lib/python2.6/dist-packages/numpy/core/include -o mpiext.o -Wall >> -fPIC >> mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium >> MPI initialised OK >> >> >> Then you should be able to run the unit tests >> mpirun -np 2 python test_pypar.py >> >> Please let us know how you go. >> >> Cheers >> Ole >> >> >> >> >> un...@in... >> >> >> On Fri, Dec 18, 2009 at 3:26 PM, Romesh Abeysuriya <rom...@gm... >> > wrote: >> >>> Hi Ole, >>> >>> We are trying to build PyPar on a couple of platforms, and are >>> encountering the same problem. >>> >>> The two architectures are: 64-bit intels (one Linux, the other OS X Snow >>> Leopard). The common >>> error we encounter in both is associated with this message: >>> >>> gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o >>> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt >>> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so >>> sh: -c: line 0: unexpected EOF while looking for matching `'' >>> sh: -c: line 1: syntax error: unexpected end of file >>> _exec_command_posix failed (status=512) >>> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' >>> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file >>> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' >>> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file >>> error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o >>> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt >>> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit >>> status 2 >>> >>> The main problem stems from this -lopa' element. Grepping around your >>> source for "lopa" or "opa" didn't give us a clue how to fix this problem. >>> >>> Additionally, on Snow Leopard, problems arise because PyPar is trying to >>> create a Universal Binary, when only x86_64 MPICH libraries have been >>> compiled, so we are looking for a way to remove the extra -arch switches. >>> >>> We really would like to use PyPar in our project, so if you could please >>> advise us how to fix these problems, we'd be really grateful. >>> >>> For reference, here's some version information on our software stack >>> we're using to build PyPar: >>> >>> MPICH2-1.2.1 >>> Python 2.6.1 (OS X) >>> Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X) >>> GCC 4.1.2 (Linux), 4.2.1 (OS X) >>> PyPar 2.1.0.66 >>> >>> >>> Thanks and all the best... >>> >>> Romesh Abeysuriya >>> Jay Larson >>> >>> >>> >> > |
Thanks for helping keep SourceForge clean.
X