pypar-developers Mailing List for pypar - parallel programming with Python
Brought to you by:
uniomni
Menu
▾
▴
pypar-developers — Discussions about development of pypar
You can subscribe to this list here.
| 2007 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(7) |
Aug
(2) |
Sep
(1) |
Oct
|
Nov
|
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2008 |
Jan
(2) |
Feb
(4) |
Mar
(1) |
Apr
(4) |
May
(4) |
Jun
(1) |
Jul
(2) |
Aug
|
Sep
(1) |
Oct
(7) |
Nov
(1) |
Dec
|
| 2009 |
Jan
|
Feb
|
Mar
(2) |
Apr
|
May
(4) |
Jun
(4) |
Jul
(2) |
Aug
|
Sep
(1) |
Oct
(6) |
Nov
|
Dec
(16) |
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
|
From: Ole N. <ole...@gm...> - 2009-12-22 13:14:26
|
Hi everyone Sparked by the recent pypar activity and growing annoyances with SF I decided to move Pypar to Google Code: The new home is: http://code.google.com/p/pypar I have synced the repository so that all changesets are still available. There is also a new mailing list: pyp...@go... which can be searched by Google and viewed at http://groups.google.com/group/pypar-discuss I will try to forward the old mail archives to this list so that they can be searched. Please let me know if you wish to join this new mailing list and also if you wish to re-join the pypar project. Cheers everyone Ole Nielsen ---------- Forwarded message ---------- From: <no...@go...> Date: Tue, Dec 22, 2009 at 11:25 PM Subject: Google Groups: You've created the new group pypar-discuss To: ole...@gm... Hello ole...@gm..., Congratulations: you've successfully created your Google Group, pypar-discuss. Here are the essentials: * Group name: pypar-discuss * Group home page: http://groups.google.com.au/group/pypar-discuss * Group email address pyp...@go... And here are links to a few more Google Group-related goodies: * Change group settings: http://groups.google.com.au/group/pypar-discuss/manage * Invite more users: http://groups.google.com.au/group/pypar-discuss/members_invite If you have questions about this or any other group, please visit the Google Groups Help Centre at http://groups.google.com.au/support?hl=en-GB. Enjoy your group and make us proud! The Google Groups Team |
|
From: Ole N. <ole...@gm...> - 2009-12-22 07:25:37
|
OK - try to use the attached setup.py and let me know how it goes.
Thanks
Ole
On Tue, Dec 22, 2009 at 6:31 AM, Romesh Abeysuriya <rom...@gm...>wrote:
> Hi Ole,
>
> Here is the latest from compiling on my OS X machine
>
> pypar-2.1.0_66 : has the -lopa' problem, but if I run
> compile_pypar_locally, it installs and runs (but fails tests as discussed
> previously)
>
> pypar-2.1.2_87 : 'python setup.py install' runs and finishes without
> errors, although it complains about the library architecture (as previously
> discussed re. building universal binary as opposed to a single
> architecture). However, if I try to import pypar in python, I get this:
>
> >>> import pypar
> ERROR: C extension mpiext could not be imported.
> Please compile mpiext.c e.g. by running
> python compile_pypar_manually.py
> in the pypar directory, or by using
> python setup.py install
>
> WARNING: MPI library could not be initialised - running sequentially
> >>>
>
> By the way, should this error message read
> "Please compile mpiext.c e.g. by running
>
> python compile_pypar_locally.py"
>
> rather than "python compile_pypar_manually.py" ?
>
> If I run compile_pypar_locally, it compiles. However, Python then does
> this:
>
> >>> import pypar
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "pypar.py", line 863, in <module>
> mpi = CDLL('libmpi.so.0', RTLD_GLOBAL)
> File
> "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ctypes/__init__.py",
> line 345, in __init__
> self._handle = _dlopen(self._name, mode)
> OSError: dlopen(libmpi.so.0, 10): image not found
> >>>
>
> pypar-2.1.3_91 : 'python setup.py install' quits with the following error
> message
>
> ...
> running build_ext
> building 'pypar.mpiext' extension
> creating build/temp.macosx-10.6-universal-2.6
> gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os
> -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64
> -pipe
> -I/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/core/include
> -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
> -c mpiext.c -o build/temp.macosx-10.6-universal-2.6/mpiext.o
> gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -arch i386 -arch ppc
> -arch x86_64 build/temp.macosx-10.6-universal-2.6/mpiext.o -L/usr/opt/mpi
> -lmpi -lelan -o build/lib.macosx-10.6-universal-2.6/pypar/mpiext.so
> ld: library not found for -lelan
>
> collect2: ld returned 1 exit status
> ld: library not found for -lelan
>
> collect2: ld returned 1 exit status
> ld: library not found for -lelan
>
> collect2: ld returned 1 exit status
> lipo: can't open input file:
> /var/folders/eW/eWXFeb8cGoKDvOMgR+R4Q++++TI/-Tmp-//cciaX7u5.out (No such
> file or directory)
> error: command 'gcc-4.2' failed with exit status 1
>
> Running compile_pypar_locally gives exactly the same error message in
> python as 2.1.2_87
>
> Thanks for the speedy updates!
>
> Romesh
>
> 2009/12/22 Jay Larson <jw...@rs...>
>
> Hi Ole,
>>
>> The -lopa' problem is certainly the same.
>>
>> Jay
>>
>> Sent from Jay's BlackBerry®
>> ------------------------------
>> *From: * Ole Nielsen <ole...@gm...>
>> *Date: *Mon, 21 Dec 2009 23:37:01 +1100
>> *To: *Dr. Jay Larson (ANU)<jay...@an...>; Pypar-developers<
>> Pyp...@li...>; Romesh Abeysuriya<
>> rom...@gm...>; Jussi Lehtola<jus...@fe...>
>> *Subject: *Fwd: Problems with pypar & MPICH2
>>
>> Hi guys
>>
>> This sounds like the problem you had?
>>
>> Is this something with recent MPICH versions perhaps?
>>
>> Cheers
>> Ole
>>
>> ---------- Forwarded message ----------
>> From: Jussi Lehtola <jus...@fe...>
>> Date: Mon, Dec 21, 2009 at 11:34 PM
>> Subject: Problems with pypar & MPICH2
>> To: ole...@gm...
>>
>>
>> Hi,
>>
>>
>> I'm trying to add MPICH2 support to the Fedora package of pypar, however
>> the compilation fails with
>>
>> mpicc -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
>> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
>> -D_GNU_SOURCE -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
>> -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64
>> -mtune=generic -fPIC -I/usr/include/mpich2-x86_64
>> -I/usr/lib64/python2.6/site-packages/numpy/core/include
>> -I/usr/include/python2.6 -c mpiext.c -o
>> build/temp.linux-x86_64-2.6/mpiext.o -fPIC
>> mpiext.c: In function 'type_map':
>> mpiext.c:127: warning: return makes integer from pointer without a cast
>> mpiext.c: In function 'op_map':
>> mpiext.c:168: warning: return makes integer from pointer without a cast
>> gcc -pthread -shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
>> -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64
>> -mtune=generic build/temp.linux-x86_64-2.6/mpiext.o
>> -L/usr/lib64/mpich2/lib -L/usr/lib64 -lmpich -lopa -lpthread -lrt -lopa'
>> -lpython2.6 -o build/lib.linux-x86_64-2.6/pypar/mpiext.so
>> /usr/bin/ld: cannot find -lopa'
>> collect2: ld returned 1 exit status
>>
>> so it seems that an extra ' is picked up somewhere. Do you have any idea
>> where this might come out from?
>>
>> Also, although I've defined CC=mpicc, the file is compiled with gcc..
>>
>> --
>> Jussi Lehtola
>> Fedora Project Contributor
>> jus...@fe...
>>
>>
>>
>
|
|
From: Romesh A. <rom...@gm...> - 2009-12-21 23:32:17
|
Hi Ole,
Here is the latest from compiling on my OS X machine
pypar-2.1.0_66 : has the -lopa' problem, but if I run compile_pypar_locally,
it installs and runs (but fails tests as discussed previously)
pypar-2.1.2_87 : 'python setup.py install' runs and finishes without errors,
although it complains about the library architecture (as previously
discussed re. building universal binary as opposed to a single
architecture). However, if I try to import pypar in python, I get this:
>>> import pypar
ERROR: C extension mpiext could not be imported.
Please compile mpiext.c e.g. by running
python compile_pypar_manually.py
in the pypar directory, or by using
python setup.py install
WARNING: MPI library could not be initialised - running sequentially
>>>
By the way, should this error message read
"Please compile mpiext.c e.g. by running
python compile_pypar_locally.py"
rather than "python compile_pypar_manually.py" ?
If I run compile_pypar_locally, it compiles. However, Python then does this:
>>> import pypar
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pypar.py", line 863, in <module>
mpi = CDLL('libmpi.so.0', RTLD_GLOBAL)
File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ctypes/__init__.py",
line 345, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(libmpi.so.0, 10): image not found
>>>
pypar-2.1.3_91 : 'python setup.py install' quits with the following error
message
...
running build_ext
building 'pypar.mpiext' extension
creating build/temp.macosx-10.6-universal-2.6
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os
-Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64
-pipe
-I/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/core/include
-I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
-c mpiext.c -o build/temp.macosx-10.6-universal-2.6/mpiext.o
gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -arch i386 -arch ppc -arch
x86_64 build/temp.macosx-10.6-universal-2.6/mpiext.o -L/usr/opt/mpi -lmpi
-lelan -o build/lib.macosx-10.6-universal-2.6/pypar/mpiext.so
ld: library not found for -lelan
collect2: ld returned 1 exit status
ld: library not found for -lelan
collect2: ld returned 1 exit status
ld: library not found for -lelan
collect2: ld returned 1 exit status
lipo: can't open input file:
/var/folders/eW/eWXFeb8cGoKDvOMgR+R4Q++++TI/-Tmp-//cciaX7u5.out (No such
file or directory)
error: command 'gcc-4.2' failed with exit status 1
Running compile_pypar_locally gives exactly the same error message in python
as 2.1.2_87
Thanks for the speedy updates!
Romesh
2009/12/22 Jay Larson <jw...@rs...>
> Hi Ole,
>
> The -lopa' problem is certainly the same.
>
> Jay
>
> Sent from Jay's BlackBerry®
> ------------------------------
> *From: * Ole Nielsen <ole...@gm...>
> *Date: *Mon, 21 Dec 2009 23:37:01 +1100
> *To: *Dr. Jay Larson (ANU)<jay...@an...>; Pypar-developers<
> Pyp...@li...>; Romesh Abeysuriya<
> rom...@gm...>; Jussi Lehtola<jus...@fe...>
> *Subject: *Fwd: Problems with pypar & MPICH2
>
> Hi guys
>
> This sounds like the problem you had?
>
> Is this something with recent MPICH versions perhaps?
>
> Cheers
> Ole
>
> ---------- Forwarded message ----------
> From: Jussi Lehtola <jus...@fe...>
> Date: Mon, Dec 21, 2009 at 11:34 PM
> Subject: Problems with pypar & MPICH2
> To: ole...@gm...
>
>
> Hi,
>
>
> I'm trying to add MPICH2 support to the Fedora package of pypar, however
> the compilation fails with
>
> mpicc -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
> -D_GNU_SOURCE -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64
> -mtune=generic -fPIC -I/usr/include/mpich2-x86_64
> -I/usr/lib64/python2.6/site-packages/numpy/core/include
> -I/usr/include/python2.6 -c mpiext.c -o
> build/temp.linux-x86_64-2.6/mpiext.o -fPIC
> mpiext.c: In function 'type_map':
> mpiext.c:127: warning: return makes integer from pointer without a cast
> mpiext.c: In function 'op_map':
> mpiext.c:168: warning: return makes integer from pointer without a cast
> gcc -pthread -shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64
> -mtune=generic build/temp.linux-x86_64-2.6/mpiext.o
> -L/usr/lib64/mpich2/lib -L/usr/lib64 -lmpich -lopa -lpthread -lrt -lopa'
> -lpython2.6 -o build/lib.linux-x86_64-2.6/pypar/mpiext.so
> /usr/bin/ld: cannot find -lopa'
> collect2: ld returned 1 exit status
>
> so it seems that an extra ' is picked up somewhere. Do you have any idea
> where this might come out from?
>
> Also, although I've defined CC=mpicc, the file is compiled with gcc..
>
> --
> Jussi Lehtola
> Fedora Project Contributor
> jus...@fe...
>
>
>
|
|
From: Jay L. <jw...@rs...> - 2009-12-21 21:33:49
|
Hi Ole, The -lopa' problem is certainly the same. Jay Sent from Jay's BlackBerry® -----Original Message----- From: Ole Nielsen <ole...@gm...> Date: Mon, 21 Dec 2009 23:37:01 To: Dr. Jay Larson (ANU)<jay...@an...>; Pypar-developers<Pyp...@li...>; Romesh Abeysuriya<rom...@gm...>; Jussi Lehtola<jus...@fe...> Subject: Fwd: Problems with pypar & MPICH2 Hi guys This sounds like the problem you had? Is this something with recent MPICH versions perhaps? Cheers Ole ---------- Forwarded message ---------- From: Jussi Lehtola <jus...@fe...> Date: Mon, Dec 21, 2009 at 11:34 PM Subject: Problems with pypar & MPICH2 To: ole...@gm... Hi, I'm trying to add MPICH2 support to the Fedora package of pypar, however the compilation fails with mpicc -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC -I/usr/include/mpich2-x86_64 -I/usr/lib64/python2.6/site-packages/numpy/core/include -I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o -fPIC mpiext.c: In function 'type_map': mpiext.c:127: warning: return makes integer from pointer without a cast mpiext.c: In function 'op_map': mpiext.c:168: warning: return makes integer from pointer without a cast gcc -pthread -shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib64/mpich2/lib -L/usr/lib64 -lmpich -lopa -lpthread -lrt -lopa' -lpython2.6 -o build/lib.linux-x86_64-2.6/pypar/mpiext.so /usr/bin/ld: cannot find -lopa' collect2: ld returned 1 exit status so it seems that an extra ' is picked up somewhere. Do you have any idea where this might come out from? Also, although I've defined CC=mpicc, the file is compiled with gcc.. -- Jussi Lehtola Fedora Project Contributor jus...@fe... |
|
From: Ole N. <ole...@gm...> - 2009-12-21 18:33:40
|
Or as it is, it appears that with MPICH the command to get the compiler string is mpicc -compile_info -link_info Could I bother you to paste that in? Thanks Ole On Tue, Dec 22, 2009 at 5:29 AM, Ole Nielsen <ole...@gm...>wrote: > OK - then the problem might be with mpicc --showme > Could I bother you to issue that command (using mpicc from MPICH2) and > paste the results in. > With OpenMPI it is > > o@desktop:~/sandpit/pypar$ mpicc --showme > gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread > -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic > -lnsl -lutil -lm -ldl > > and this works fine. > > Cheers > Ole > > > On Mon, Dec 21, 2009 at 11:42 PM, Jussi Lehtola < > jus...@fe...> wrote: > >> On Mon, 2009-12-21 at 23:38 +1100, Ole Nielsen wrote: >> > Hi Jussi - this compiles fine where I am, but I am using OpenMPI. >> > The compiler flags are generated mpicc. >> >> I'm, too, a Open MPI user myself. >> >> > Does it work if you run python compile_pypar_locally.py - just as a >> > test. >> >> Yes, it works just fine with MPICH2. >> >> -- >> Jussi Lehtola >> Fedora Project Contributor >> jus...@fe... >> >> > |
|
From: Ole N. <ole...@gm...> - 2009-12-21 18:29:53
|
OK - then the problem might be with mpicc --showme Could I bother you to issue that command (using mpicc from MPICH2) and paste the results in. With OpenMPI it is o@desktop:~/sandpit/pypar$ mpicc --showme gcc -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl and this works fine. Cheers Ole On Mon, Dec 21, 2009 at 11:42 PM, Jussi Lehtola < jus...@fe...> wrote: > On Mon, 2009-12-21 at 23:38 +1100, Ole Nielsen wrote: > > Hi Jussi - this compiles fine where I am, but I am using OpenMPI. > > The compiler flags are generated mpicc. > > I'm, too, a Open MPI user myself. > > > Does it work if you run python compile_pypar_locally.py - just as a > > test. > > Yes, it works just fine with MPICH2. > > -- > Jussi Lehtola > Fedora Project Contributor > jus...@fe... > > |
|
From: Ole N. <ole...@gm...> - 2009-12-21 12:45:41
|
Hi guys This sounds like the problem you had? Is this something with recent MPICH versions perhaps? Cheers Ole ---------- Forwarded message ---------- From: Jussi Lehtola <jus...@fe...> Date: Mon, Dec 21, 2009 at 11:34 PM Subject: Problems with pypar & MPICH2 To: ole...@gm... Hi, I'm trying to add MPICH2 support to the Fedora package of pypar, however the compilation fails with mpicc -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC -I/usr/include/mpich2-x86_64 -I/usr/lib64/python2.6/site-packages/numpy/core/include -I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o -fPIC mpiext.c: In function 'type_map': mpiext.c:127: warning: return makes integer from pointer without a cast mpiext.c: In function 'op_map': mpiext.c:168: warning: return makes integer from pointer without a cast gcc -pthread -shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib64/mpich2/lib -L/usr/lib64 -lmpich -lopa -lpthread -lrt -lopa' -lpython2.6 -o build/lib.linux-x86_64-2.6/pypar/mpiext.so /usr/bin/ld: cannot find -lopa' collect2: ld returned 1 exit status so it seems that an extra ' is picked up somewhere. Do you have any idea where this might come out from? Also, although I've defined CC=mpicc, the file is compiled with gcc.. -- Jussi Lehtola Fedora Project Contributor jus...@fe... |
|
From: Ole N. <ole...@gm...> - 2009-12-19 18:08:46
|
OK - good that it compiles. I too noticed a recent problem with reduce: It has worked for many years, but with OpenMPI version 1.3.2-3ubuntu1 I too had failed tests to do with reduce. Everything else is OK, though. I'll investigate. Beer in Canberra sounds great. Are having drinkies at the ANUSF before Xmas? Cheers Ole On Sat, Dec 19, 2009 at 2:08 PM, Romesh Abeysuriya <rom...@gm...>wrote: > Hi Ole, > > That seems to fix compilation on the OS X machine- compiling gives > > romesh: source > python compile_pypar_locally.py > mpicc -c mpiext.c > -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 > -I/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/core/include > -o mpiext.o -Wall > mpiext.c: In function ‘type_map’: > mpiext.c:127: warning: return makes integer from pointer without a cast > mpiext.c: In function ‘op_map’: > mpiext.c:168: warning: return makes integer from pointer without a cast > mpicc -bundle -flat_namespace -undefined suppress mpiext.o -o mpiext.so > MPI initialised OK > > However, test_pypar.py does not run correctly. An mpd session is opened on > the local machine with > > romesh: source > mpdboot --totalnum=1 --ncpus=2 > > And then test_pypar.py is run using > > romesh: source > mpirun -np 2 python test_pypar.py > Pypar (version 2.1.0) initialised MPI OK with 2 processors > I am processor 0 of 2 on node Romesh-MBP-3.local > I am processor 1 of 2 on node Romesh-MBP-3.local > Raw communication of numeric integer arrays OK > ... > ... > ... > Raw reduce using pypar.BXOR OK > Traceback (most recent call last): > File "test_pypar.py", line 953, in <module> > assert status.error == 0 > AssertionError > rank 0 in job 1 Romesh-MBP-3.local_49875 caused collective abort of all > ranks > exit status of rank 0: return code 1 > > > with the error originating at line 953, and status variable being set in > line 941 > > 941 B, status = pypar.receive(numproc-1, return_status = True) > 953 assert status.error == 0 > > Any ideas on what might be causing this? > > We will try this on the Linux machine on Monday > > Thanks for the quick reply! > > Romesh > > 2009/12/19 Ole Nielsen <ole...@gm...> > > Hi Romesh >> >> When I build pypar on our 64 bit Intel cluster (Ubuntu 9.04) with OpenMPI, >> the compile string looks like this: >> >> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall >> -Wstrict-prototypes -fPIC -I/usr/lib/openmpi/include >> -I/usr/lib/openmpi/include/openmpi >> -I/usr/lib/python2.6/dist-packages/numpy/core/include >> -I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o >> -fPIC >> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions >> build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib/openmpi/lib -lmpi -lopen-rte >> -lopen-pal -ldl -lnsl -lutil -lm -o >> build/lib.linux-x86_64-2.6/pypar/mpiext.so >> >> and all is well. The same is true on a 32 bit Ubuntu 9.10 system. >> >> >> Could I ask you to try to run >> python compile_pypar_locally.py >> >> That will compile the mpiext.so using mpicc directly (bypassing distutils) >> and the compile output should look something like this. >> >> nielso@alamba:~/sandpit/pypar/source$ python compile_pypar_locally.py >> mpicc -c mpiext.c -I/usr/include/python2.6 >> -I/usr/lib/python2.6/dist-packages/numpy/core/include -o mpiext.o -Wall >> -fPIC >> mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium >> MPI initialised OK >> >> >> Then you should be able to run the unit tests >> mpirun -np 2 python test_pypar.py >> >> Please let us know how you go. >> >> Cheers >> Ole >> >> >> >> >> un...@in... >> >> >> On Fri, Dec 18, 2009 at 3:26 PM, Romesh Abeysuriya <rom...@gm... >> > wrote: >> >>> Hi Ole, >>> >>> We are trying to build PyPar on a couple of platforms, and are >>> encountering the same problem. >>> >>> The two architectures are: 64-bit intels (one Linux, the other OS X Snow >>> Leopard). The common >>> error we encounter in both is associated with this message: >>> >>> gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o >>> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt >>> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so >>> sh: -c: line 0: unexpected EOF while looking for matching `'' >>> sh: -c: line 1: syntax error: unexpected end of file >>> _exec_command_posix failed (status=512) >>> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' >>> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file >>> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' >>> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file >>> error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o >>> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt >>> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit >>> status 2 >>> >>> The main problem stems from this -lopa' element. Grepping around your >>> source for "lopa" or "opa" didn't give us a clue how to fix this problem. >>> >>> Additionally, on Snow Leopard, problems arise because PyPar is trying to >>> create a Universal Binary, when only x86_64 MPICH libraries have been >>> compiled, so we are looking for a way to remove the extra -arch switches. >>> >>> We really would like to use PyPar in our project, so if you could please >>> advise us how to fix these problems, we'd be really grateful. >>> >>> For reference, here's some version information on our software stack >>> we're using to build PyPar: >>> >>> MPICH2-1.2.1 >>> Python 2.6.1 (OS X) >>> Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X) >>> GCC 4.1.2 (Linux), 4.2.1 (OS X) >>> PyPar 2.1.0.66 >>> >>> >>> Thanks and all the best... >>> >>> Romesh Abeysuriya >>> Jay Larson >>> >>> >>> >> > |
|
From: Jay L. <jw...@rs...> - 2009-12-19 06:19:34
|
Hi Ole,
Thanks for the rapid response...and on the weekend--wow!
Since we are all in Canberra, we should meet up sometime and I will buy you some drinks.
All the Best,
Jay
Sent from Jay's BlackBerry®
-----Original Message-----
From: Ole Nielsen <ole...@gm...>
Date: Sat, 19 Dec 2009 12:26:50
To: Romesh Abeysuriya<rom...@gm...>
Cc: Dr. Jay Larson (ANU)<jay...@an...>; Pypar-developers<Pyp...@li...>
Subject: Re: PyPar installation
Hi Romesh
When I build pypar on our 64 bit Intel cluster (Ubuntu 9.04) with OpenMPI,
the compile string looks like this:
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -I/usr/lib/openmpi/include
-I/usr/lib/openmpi/include/openmpi
-I/usr/lib/python2.6/dist-packages/numpy/core/include
-I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o
-fPIC
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib/openmpi/lib -lmpi -lopen-rte
-lopen-pal -ldl -lnsl -lutil -lm -o
build/lib.linux-x86_64-2.6/pypar/mpiext.so
and all is well. The same is true on a 32 bit Ubuntu 9.10 system.
Could I ask you to try to run
python compile_pypar_locally.py
That will compile the mpiext.so using mpicc directly (bypassing distutils)
and the compile output should look something like this.
nielso@alamba:~/sandpit/pypar/source$ python compile_pypar_locally.py
mpicc -c mpiext.c -I/usr/include/python2.6
-I/usr/lib/python2.6/dist-packages/numpy/core/include -o mpiext.o -Wall
-fPIC
mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium
MPI initialised OK
Then you should be able to run the unit tests
mpirun -np 2 python test_pypar.py
Please let us know how you go.
Cheers
Ole
un...@in...
On Fri, Dec 18, 2009 at 3:26 PM, Romesh Abeysuriya <rom...@gm...>wrote:
> Hi Ole,
>
> We are trying to build PyPar on a couple of platforms, and are encountering
> the same problem.
>
> The two architectures are: 64-bit intels (one Linux, the other OS X Snow
> Leopard). The common
> error we encounter in both is associated with this message:
>
> gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o
> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt
> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so
> sh: -c: line 0: unexpected EOF while looking for matching `''
> sh: -c: line 1: syntax error: unexpected end of file
> _exec_command_posix failed (status=512)
> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file
> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file
> error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o
> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt
> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit
> status 2
>
> The main problem stems from this -lopa' element. Grepping around your
> source for "lopa" or "opa" didn't give us a clue how to fix this problem.
>
> Additionally, on Snow Leopard, problems arise because PyPar is trying to
> create a Universal Binary, when only x86_64 MPICH libraries have been
> compiled, so we are looking for a way to remove the extra -arch switches.
>
> We really would like to use PyPar in our project, so if you could please
> advise us how to fix these problems, we'd be really grateful.
>
> For reference, here's some version information on our software stack we're
> using to build PyPar:
>
> MPICH2-1.2.1
> Python 2.6.1 (OS X)
> Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X)
> GCC 4.1.2 (Linux), 4.2.1 (OS X)
> PyPar 2.1.0.66
>
>
> Thanks and all the best...
>
> Romesh Abeysuriya
> Jay Larson
>
>
>
|
|
From: Romesh A. <rom...@gm...> - 2009-12-19 03:08:36
|
Hi Ole,
That seems to fix compilation on the OS X machine- compiling gives
romesh: source > python compile_pypar_locally.py
mpicc -c mpiext.c
-I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
-I/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy/core/include
-o mpiext.o -Wall
mpiext.c: In function ‘type_map’:
mpiext.c:127: warning: return makes integer from pointer without a cast
mpiext.c: In function ‘op_map’:
mpiext.c:168: warning: return makes integer from pointer without a cast
mpicc -bundle -flat_namespace -undefined suppress mpiext.o -o mpiext.so
MPI initialised OK
However, test_pypar.py does not run correctly. An mpd session is opened on
the local machine with
romesh: source > mpdboot --totalnum=1 --ncpus=2
And then test_pypar.py is run using
romesh: source > mpirun -np 2 python test_pypar.py
Pypar (version 2.1.0) initialised MPI OK with 2 processors
I am processor 0 of 2 on node Romesh-MBP-3.local
I am processor 1 of 2 on node Romesh-MBP-3.local
Raw communication of numeric integer arrays OK
...
...
...
Raw reduce using pypar.BXOR OK
Traceback (most recent call last):
File "test_pypar.py", line 953, in <module>
assert status.error == 0
AssertionError
rank 0 in job 1 Romesh-MBP-3.local_49875 caused collective abort of all
ranks
exit status of rank 0: return code 1
with the error originating at line 953, and status variable being set in
line 941
941 B, status = pypar.receive(numproc-1, return_status = True)
953 assert status.error == 0
Any ideas on what might be causing this?
We will try this on the Linux machine on Monday
Thanks for the quick reply!
Romesh
2009/12/19 Ole Nielsen <ole...@gm...>
> Hi Romesh
>
> When I build pypar on our 64 bit Intel cluster (Ubuntu 9.04) with OpenMPI,
> the compile string looks like this:
>
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -I/usr/lib/openmpi/include
> -I/usr/lib/openmpi/include/openmpi
> -I/usr/lib/python2.6/dist-packages/numpy/core/include
> -I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o
> -fPIC
> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib/openmpi/lib -lmpi -lopen-rte
> -lopen-pal -ldl -lnsl -lutil -lm -o
> build/lib.linux-x86_64-2.6/pypar/mpiext.so
>
> and all is well. The same is true on a 32 bit Ubuntu 9.10 system.
>
>
> Could I ask you to try to run
> python compile_pypar_locally.py
>
> That will compile the mpiext.so using mpicc directly (bypassing distutils)
> and the compile output should look something like this.
>
> nielso@alamba:~/sandpit/pypar/source$ python compile_pypar_locally.py
> mpicc -c mpiext.c -I/usr/include/python2.6
> -I/usr/lib/python2.6/dist-packages/numpy/core/include -o mpiext.o -Wall
> -fPIC
> mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium
> MPI initialised OK
>
>
> Then you should be able to run the unit tests
> mpirun -np 2 python test_pypar.py
>
> Please let us know how you go.
>
> Cheers
> Ole
>
>
>
>
> un...@in...
>
>
> On Fri, Dec 18, 2009 at 3:26 PM, Romesh Abeysuriya <rom...@gm...>wrote:
>
>> Hi Ole,
>>
>> We are trying to build PyPar on a couple of platforms, and are
>> encountering the same problem.
>>
>> The two architectures are: 64-bit intels (one Linux, the other OS X Snow
>> Leopard). The common
>> error we encounter in both is associated with this message:
>>
>> gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o
>> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt
>> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so
>> sh: -c: line 0: unexpected EOF while looking for matching `''
>> sh: -c: line 1: syntax error: unexpected end of file
>> _exec_command_posix failed (status=512)
>> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
>> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file
>> /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
>> /usr/bin/bash: -c: line 1: syntax error: unexpected end of file
>> error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o
>> -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt
>> -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit
>> status 2
>>
>> The main problem stems from this -lopa' element. Grepping around your
>> source for "lopa" or "opa" didn't give us a clue how to fix this problem.
>>
>> Additionally, on Snow Leopard, problems arise because PyPar is trying to
>> create a Universal Binary, when only x86_64 MPICH libraries have been
>> compiled, so we are looking for a way to remove the extra -arch switches.
>>
>> We really would like to use PyPar in our project, so if you could please
>> advise us how to fix these problems, we'd be really grateful.
>>
>> For reference, here's some version information on our software stack we're
>> using to build PyPar:
>>
>> MPICH2-1.2.1
>> Python 2.6.1 (OS X)
>> Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X)
>> GCC 4.1.2 (Linux), 4.2.1 (OS X)
>> PyPar 2.1.0.66
>>
>>
>> Thanks and all the best...
>>
>> Romesh Abeysuriya
>> Jay Larson
>>
>>
>>
>
|
|
From: Ole N. <ole...@gm...> - 2009-12-19 01:27:02
|
Hi Romesh When I build pypar on our 64 bit Intel cluster (Ubuntu 9.04) with OpenMPI, the compile string looks like this: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -I/usr/lib/python2.6/dist-packages/numpy/core/include -I/usr/include/python2.6 -c mpiext.c -o build/temp.linux-x86_64-2.6/mpiext.o -fPIC gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/mpiext.o -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lopen-pal -ldl -lnsl -lutil -lm -o build/lib.linux-x86_64-2.6/pypar/mpiext.so and all is well. The same is true on a 32 bit Ubuntu 9.10 system. Could I ask you to try to run python compile_pypar_locally.py That will compile the mpiext.so using mpicc directly (bypassing distutils) and the compile output should look something like this. nielso@alamba:~/sandpit/pypar/source$ python compile_pypar_locally.py mpicc -c mpiext.c -I/usr/include/python2.6 -I/usr/lib/python2.6/dist-packages/numpy/core/include -o mpiext.o -Wall -fPIC mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium MPI initialised OK Then you should be able to run the unit tests mpirun -np 2 python test_pypar.py Please let us know how you go. Cheers Ole un...@in... On Fri, Dec 18, 2009 at 3:26 PM, Romesh Abeysuriya <rom...@gm...>wrote: > Hi Ole, > > We are trying to build PyPar on a couple of platforms, and are encountering > the same problem. > > The two architectures are: 64-bit intels (one Linux, the other OS X Snow > Leopard). The common > error we encounter in both is associated with this message: > > gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o > -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt > -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so > sh: -c: line 0: unexpected EOF while looking for matching `'' > sh: -c: line 1: syntax error: unexpected end of file > _exec_command_posix failed (status=512) > /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' > /usr/bin/bash: -c: line 1: syntax error: unexpected end of file > /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' > /usr/bin/bash: -c: line 1: syntax error: unexpected end of file > error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o > -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt > -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit > status 2 > > The main problem stems from this -lopa' element. Grepping around your > source for "lopa" or "opa" didn't give us a clue how to fix this problem. > > Additionally, on Snow Leopard, problems arise because PyPar is trying to > create a Universal Binary, when only x86_64 MPICH libraries have been > compiled, so we are looking for a way to remove the extra -arch switches. > > We really would like to use PyPar in our project, so if you could please > advise us how to fix these problems, we'd be really grateful. > > For reference, here's some version information on our software stack we're > using to build PyPar: > > MPICH2-1.2.1 > Python 2.6.1 (OS X) > Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X) > GCC 4.1.2 (Linux), 4.2.1 (OS X) > PyPar 2.1.0.66 > > > Thanks and all the best... > > Romesh Abeysuriya > Jay Larson > > > |
|
From: Ole N. <ole...@gm...> - 2009-12-18 05:14:47
|
Can anyone help out? I don't understand where the -lopa comes from. Cheers and thanks Ole ---------- Forwarded message ---------- From: Romesh Abeysuriya <rom...@gm...> Date: Fri, Dec 18, 2009 at 11:26 AM Subject: PyPar installation To: ole...@gm... Cc: "Dr. Jay Larson (ANU)" <jay...@an...> Hi Ole, We are trying to build PyPar on a couple of platforms, and are encountering the same problem. The two architectures are: 64-bit intels (one Linux, the other OS X Snow Leopard). The common error we encounter in both is associated with this message: gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so sh: -c: line 0: unexpected EOF while looking for matching `'' sh: -c: line 1: syntax error: unexpected end of file _exec_command_posix failed (status=512) /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' /usr/bin/bash: -c: line 1: syntax error: unexpected end of file /usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `'' /usr/bin/bash: -c: line 1: syntax error: unexpected end of file error: Command "gcc -pthread -shared build/temp.linux-x86_64-2.4/mpiext.o -L/users/prl/rga112/mpich2-install-scucomp1/lib -lmpich -lopa -lpthread -lrt -lopa' -o build/lib.linux-x86_64-2.4/pypar/mpiext.so" failed with exit status 2 The main problem stems from this -lopa' element. Grepping around your source for "lopa" or "opa" didn't give us a clue how to fix this problem. Additionally, on Snow Leopard, problems arise because PyPar is trying to create a Universal Binary, when only x86_64 MPICH libraries have been compiled, so we are looking for a way to remove the extra -arch switches. We really would like to use PyPar in our project, so if you could please advise us how to fix these problems, we'd be really grateful. For reference, here's some version information on our software stack we're using to build PyPar: MPICH2-1.2.1 Python 2.6.1 (OS X) Numpy 1.0.1 (current rpmforge repository version), latest SVN (OS X) GCC 4.1.2 (Linux), 4.2.1 (OS X) PyPar 2.1.0.66 Thanks and all the best... Romesh Abeysuriya Jay Larson |
|
From: Ole N. <ole...@gm...> - 2009-10-05 01:55:44
|
Hi again Jordan I can't comment on this error at this stage but I know that all tests normally pass so it must be something about your version of MPICH perhaps. However, if the problem only arises with reduce, I'd ignore it for now unless, of course, you plan to rely on reduce. In my (and many other's) opinion the most efficient MPI programs use a very small subset of MPI (as in send and receive). So I'd just get cracking with the parallel Python program :-) Cheers Ole On Mon, Oct 5, 2009 at 1:08 AM, Jordan Atlas <jc...@co...> wrote: > Hi Ole, > > When you have a chance, I have a question about tests that seem to be > failing in pypar_tests.py. > > Most of the tests run fine. When it gets to the section for testing > reduce, all the assert statements return False. the reduce function seems > to be encountering the error "Input array and buffer must be of the same > type". According to python the arrays being passed in here have the same > type. So I'm not sure what would be causing mpiext to interpret these as > different types. Any ideas? > > Otherwise things seem to be working and I'm looking forward to using > pypar! > > Thank you, > --Jordan Atlas > > |
|
From: Ole N. <ole...@gm...> - 2009-10-05 01:51:52
|
Summary from Jordan Atlas in regard to running Pypar on Windows using MPICH.
---------- Forwarded message ----------
From: Jordan Atlas <jc...@co...>
Date: Mon, Oct 5, 2009 at 12:24 AM
Subject: Re: multicore machines with pypar
To: Ole Nielsen <ole...@gm...>
Hi Ole,
Thank you for attending to my questions even though you are in transit!
I think I have found the answer to my problems... Basically it's to pass in
the path to python when running pypar based scripts. You suggested it
earlier, but I wasn't careful enough to make sure I was passing in the
correct path (on windows there is no 'bin'). Sorry about that.
There are some other details related to the email you sent most
recently, which are below in case you are interested. Also, the actual
testing of pypar fails on a particular test so I'll probably email you again
after I've investigated that a bit.
*1) I found network_timing.c (earlier I had just searched for "timings"
with an s so I missed it).
*
FYI, it took a while to get network_timing.exe to run properly. It would
crash every time I ran it with a stack overflow exception. I was able to
prevent this by changing the constant "MAXM" to 50,000 instead of 500,000.
Just letting you know in case this is unexpected.
I can now execute the network_timing example (in C) with this output:
MAXM = 50000, number of processors = 2
Measurements are repeated 10 times for reliability
I am process 0 on jca33-305.
I am process 1 on jca33-305.
Timing overhead is 0.000000 seconds
Run 1 of 10
Run 2 of 10
Run 3 of 10
Run 4 of 10
Run 5 of 10
Run 6 of 10
Run 7 of 10
Run 8 of 10
Run 9 of 10
Run 10 of 10
Bytes transferred time (micro seconds)
min avg max
----------------------------------------------
8 12 19 41
40008 70 97 118
80008 90 133 155
120008 85 160 186
160008 126 206 280
200008 115 225 259
240008 132 259 346
280008 105 250 322
320008 156 269 374
360008 161 293 398
Linear regression on best timings (t = t_l + t_b * bytes):
t_b = 0.000324
t_l = 47.273603
Estimated relative variance = 0.000000001
Estimated bandwith (1/t_b): 3082.216 Mb/s
Estimated latency: 12 micro s
*2) When I try "mpirun -np 2 network_timing.py", I get a CreateProcess
error:
*
Unable to launch
'C:\Python25\Lib\site-packages\pypar\pypar_2.1.0_66\source\network_timing.py',
erro
r 8: CreateProcess failed: Not enough storage is available to process this
command.
This is a different error than I get when just trying to open python with
mpirun (though fixing this one might just leave me with the old error).
Thinking that this might also be related to MAXM, I tried changing the value
of MAXM in network_timing.py, and it did NOT fix the error.
3) *One question is how MPICH allows the environment variables to passed on
to the MPI processes.*
In the process of trying to figure out how to pass this information to
MPICH, I tried to pass pythons path in on the command line as you suggested
earlier. This is when I realized I had used the wrong path earlier (it's
C:\python25\python as opposed to C:\python25\bin\python). Once I fixed this
I was able to run the tests with 'mpirun -np 4 C:\python25\python
test_pypar.py' (except for one failure towards the end). Also,
network_timings.py works as well!
Thanks!
--Jordan
Ole Nielsen wrote:
OK
I you are right the first thing that must be resolved is for mpich to launch
Python.
But many othyers on the mailing list have used MPICH so it seems strange.
I am in transit in Indonesia at the moment, so everything I say is from
memory.
There is a c program that ships with pypar called eiter timings.c or
network_timimngs.c or something like that. It is a pure C program that
measures bandwidth using MPI. There is also a python version doing the same
thing using pypart.
As for the option -x it might be -X (capital). This is the case in OPENMPI
at least.
If is doesn;'t have gthat option, I think one question is how MPICH allows
the environment variables to passed on to the MPI processes.
Cheers and please let us know if you find the problem.
Ole
PS Otherwise, switch to Linux and OpenMPI :-)
On Sun, Oct 4, 2009 at 1:16 AM, Jordan Atlas <jc...@co...> wrote:
> mpirun does work with executables. For example I'm able to use it with
> cpi.c, which is included with MPICH. I don't know about timings.c (I don't
> seem to have that file)...
>
> mpirun cannot launch python itself, however, I can run Python normally.
>
> My mpirun doesn't have a -x option. I looked through the manual for mpich
> and couldn't find a similar option to what you describe.
>
> mpirun -np 4 C:Python25\bin\python.exe generates the same CreateProcess
> error.
>
> If you have any other suggestions I would be happy to hear them.
> Otherwise, this seems to be more of a mpich/python problem than a pypar
> problem, so I will followup with the mpich support team...
>
> Thank you for your time,
> --Jordan
>
>
> Ole Nielsen wrote:
>
> This is a step forward. Can I just confirm:
> mpirun works with executables (e.g. timings.c)
> mpirun cannot launch python itself
> however, you can run Python normally.
>
> Is ths correct?
>
> If so, I think the problem would relate to your system path (i.e. how it
> can find Python)and how mpirun interprets it. I seem to remember, that the
> -x option with MPIRUN does something to make sure every process is aware of
> environment variables such as PATH.
>
> So perhaps you could try something like
>
> mpirun -x PATH -np 4 python
>
>
> or failing that, juse the full pathname to Python (from memory)
>
> mpirun -np 4 C:Python25\bin\python.exe
>
>
> Let us know how you go
>
> Cheers
> Ole
> On Sat, Oct 3, 2009 at 5:21 AM, Jordan Atlas <jc...@co...> wrote:
>
>> Ole,
>>
>> I should add that this program doesn't seem to be specific to pypar.
>> Rather, I can't seem to run anything pythonic. For example, if I type
>> 'mpirun -np 4 python' alone, it fails with the same error. I can, however,
>> run parallel .exe files using mpirun.
>>
>> I just wanted to say this in case it provides a clue as to what I may
>> be doing incorrectly.
>>
>> Thanks,
>>
>> --Jordan
>>
>> Ole Nielsen wrote:
>>
>> Could you try
>>
>> C:\Python25\Lib\site-packages\pypar>mpirun -np 4 python test_pypar.py
>>
>> and let me know if that gives the same error?
>>
>> Has anyone else on the list seen this?
>>
>> Cheers and thanks
>> Ole
>>
>>
>> On Thu, Oct 1, 2009 at 2:49 AM, Jordan Atlas <jc...@co...> wrote:
>>
>>> Hi Ole,
>>>
>>> Thanks for your response. Following your advice, I was able to
>>> verify that I can get MPICH to run multiple processes that utilizes both of
>>> my machines' processors.
>>>
>>> Now I'm having trouble getting the pypar tests to run. If I type:
>>>
>>> C:\Python25\Lib\site-packages\pypar>mpirun -np 4 test_pypar.py
>>>
>>> Then I get the error:
>>>
>>> Unable to launch 'C:\Python25\Lib\site-packages\pypar\test_pypar.py',
>>> error 8: CreateProcess failed:
>>> Not enough storage is available to process this command.
>>>
>>> I assume it is talking about "RAM" storage, but my machine has over 2GB
>>> of RAM free right now, so I don't understand the problem. Am I calling the
>>> program correctly?
>>>
>>> Note that if I run test_pypar without calling mpirun, I get the output:
>>>
>>> C:\Python25\Lib\site-packages\pypar>test_pypar.py
>>> Pypar (version 2.1.0) initialised MPI OK with 1 processors
>>> I am processor 0 of 1 on node jca33-305.
>>>
>>> Please let me know if my question is unclear.
>>>
>>> Thanks,
>>>
>>> --Jordan
>>>
>>>
>>> Ole Nielsen wrote:
>>>
>>> Hi Jordan
>>>
>>> Great that you got pypar installed!
>>>
>>> As for the number of processors it is really a matter for the underlying
>>> MPI implementation.
>>> If you run a C program, such as the include timing measurement, using MPI
>>> you would have the same issue.
>>>
>>> Depending on the MPI implementation there are different ways of telling
>>> how many CPUs you got.
>>> With LAM MPI, there is a .machines file, with OPENMPI it is a matter for
>>> mpirun to specify.
>>>
>>> Typically, you run you parallel program on e.g. 8 processes using
>>>
>>> mpirun -np 8 <parallel_program>
>>>
>>>
>>> Hope this helps
>>>
>>> Cheers
>>> Ole
>>>
>>>
>>> PS - everything is better if you use Linux, though :-)
>>>
>>>
>>> On Thu, Sep 24, 2009 at 4:16 AM, Jordan Atlas <jc...@co...> wrote:
>>>
>>>> Hi Ole,
>>>>
>>>> After carefully following the readme instuctions and making a couple
>>>> changes to the compile files, I was able to get pypar installed on my
>>>> machine. However, it reports that my machine only has one processor, even
>>>> though it has two.
>>>>
>>>> Is there anything in particular I need to do to make sure pypar is
>>>> aware of multiple processors in windows?
>>>>
>>>> Thanks,
>>>>
>>>> --Jordan
>>>>
>>>> --
>>>> *Jordan Atlas*
>>>> Email: jc...@co...
>>>> Chemical and Biomolecular Engineering
>>>> Cornell University
>>>>
>>>
>>>
>>>
>>
>>
>
>
|
|
From: Jordan A. <jc...@co...> - 2009-10-03 14:17:04
|
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
mpirun does work with executables. For example I'm able to use it with
cpi.c, which is included with MPICH. I don't know about timings.c (I
don't seem to have that file)...<br>
<br>
mpirun cannot launch python itself, however, I can run Python normally.<br>
<br>
My mpirun doesn't have a -x option. I looked through the manual for
mpich and couldn't find a similar option to what you describe.<br>
<br>
mpirun -np 4 C:Python25\bin\python.exe generates the same CreateProcess
error.<br>
<br>
If you have any other suggestions I would be happy to hear them.
Otherwise, this seems to be more of a mpich/python problem than a pypar
problem, so I will followup with the mpich support team...<br>
<br>
Thank you for your time,<br>
--Jordan<br>
<br>
<br>
Ole Nielsen wrote:
<blockquote
cite="mid:b4a...@ma..."
type="cite">This is a step forward. Can I just confirm:<br>
mpirun works with executables (e.g. timings.c)<br>
mpirun cannot launch python itself<br>
however, you can run Python normally.<br>
<br>
Is ths correct?<br>
<br>
If so, I think the problem would relate to your system path (i.e. how
it can find Python)and how mpirun interprets it. I seem to remember,
that the -x option with MPIRUN does something to make sure every
process is aware of environment variables such as PATH.<br>
<br>
So perhaps you could try something like<br>
<br>
mpirun -x PATH -np 4 python<br>
<br>
<br>
or failing that, juse the full pathname to Python (from memory)<br>
<br>
mpirun -np 4 C:Python25\bin\python.exe <br>
<br>
<br>
Let us know how you go<br>
<br>
Cheers<br>
Ole<br>
<div class="gmail_quote">On Sat, Oct 3, 2009 at 5:21 AM, Jordan Atlas
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:jc...@co...">jc...@co...</a>></span> wrote:<br>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">Ole,<br>
<br>
I should add that this program doesn't seem to be specific to
pypar. Rather, I can't seem to run anything pythonic. For example, if
I type 'mpirun -np 4 python' alone, it fails with the same error. I
can, however, run parallel .exe files using mpirun.<br>
<br>
I just wanted to say this in case it provides a clue as to what I
may be doing incorrectly.
<div>
<div class="h5"><br>
<br>
Thanks,<br>
<br>
--Jordan<br>
<br>
Ole Nielsen wrote:
<blockquote type="cite">Could you try<br>
<br>
C:\Python25\Lib\site-packages\pypar>mpirun -np 4 python test_pypar.py<br>
<br>
and let me know if that gives the same error?<br>
<br>
Has anyone else on the list seen this?<br>
<br>
Cheers and thanks<br>
Ole<br>
<br>
<br>
<div class="gmail_quote">On Thu, Oct 1, 2009 at 2:49 AM, Jordan
Atlas <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:jc...@co..." target="_blank">jc...@co...</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">Hi Ole,<br>
<br>
Thanks for your response. Following your advice, I was able to
verify that I can get MPICH to run multiple processes that utilizes
both of my machines' processors.<br>
<br>
Now I'm having trouble getting the pypar tests to run. If I type:<br>
<br>
C:\Python25\Lib\site-packages\pypar>mpirun -np 4 test_pypar.py<br>
<br>
Then I get the error:<br>
<br>
Unable to launch 'C:\Python25\Lib\site-packages\pypar\test_pypar.py',
error 8: CreateProcess failed:<br>
Not enough storage is available to process this command.<br>
<br>
I assume it is talking about "RAM" storage, but my machine has over 2GB
of RAM free right now, so I don't understand the problem. Am I calling
the program correctly?<br>
<br>
Note that if I run test_pypar without calling mpirun, I get the output:<br>
<br>
C:\Python25\Lib\site-packages\pypar>test_pypar.py<br>
Pypar (version 2.1.0) initialised MPI OK with 1 processors<br>
I am processor 0 of 1 on node jca33-305.<br>
<br>
Please let me know if my question is unclear.<br>
<br>
Thanks,<br>
<font color="#888888"><br>
--Jordan</font>
<div>
<div><br>
<br>
<br>
Ole Nielsen wrote:
<blockquote type="cite">Hi Jordan<br>
<br>
Great that you got pypar installed!<br>
<br>
As for the number of processors it is really a matter for the
underlying MPI implementation.<br>
If you run a C program, such as the include timing measurement, using
MPI you would have the same issue.<br>
<br>
Depending on the MPI implementation there are different ways of telling
how many CPUs you got.<br>
With LAM MPI, there is a .machines file, with OPENMPI it is a matter
for mpirun to specify.<br>
<br>
Typically, you run you parallel program on e.g. 8 processes using<br>
<br>
mpirun -np 8 <parallel_program><br>
<br>
<br>
Hope this helps<br>
<br>
Cheers<br>
Ole<br>
<br>
<br>
PS - everything is better if you use Linux, though :-)<br>
<br>
<br>
<div class="gmail_quote">On Thu, Sep 24, 2009 at 4:16 AM,
Jordan
Atlas <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:jc...@co..." target="_blank">jc...@co...</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">Hi Ole,<br>
<br>
After carefully following the readme instuctions and making a
couple changes to the compile files, I was able to get pypar installed
on my machine. However, it reports that my machine only has one
processor, even though it has two.<br>
<br>
Is there anything in particular I need to do to make sure pypar is
aware of multiple processors in windows? <br>
<br>
Thanks,<br>
<br>
--Jordan<br>
<font color="#888888"><br>
<div>-- <br>
<b>Jordan Atlas</b> <br>
Email: <a moz-do-not-send="true" href="mailto:jc...@co..."
target="_blank">jc...@co...</a> <br>
Chemical and Biomolecular Engineering <br>
Cornell University </div>
</font></div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</body>
</html>
|
|
From: Ole N. <ole...@gm...> - 2009-10-03 02:39:46
|
This is a step forward. Can I just confirm: mpirun works with executables (e.g. timings.c) mpirun cannot launch python itself however, you can run Python normally. Is ths correct? If so, I think the problem would relate to your system path (i.e. how it can find Python)and how mpirun interprets it. I seem to remember, that the -x option with MPIRUN does something to make sure every process is aware of environment variables such as PATH. So perhaps you could try something like mpirun -x PATH -np 4 python or failing that, juse the full pathname to Python (from memory) mpirun -np 4 C:Python25\bin\python.exe Let us know how you go Cheers Ole On Sat, Oct 3, 2009 at 5:21 AM, Jordan Atlas <jc...@co...> wrote: > Ole, > > I should add that this program doesn't seem to be specific to pypar. > Rather, I can't seem to run anything pythonic. For example, if I type > 'mpirun -np 4 python' alone, it fails with the same error. I can, however, > run parallel .exe files using mpirun. > > I just wanted to say this in case it provides a clue as to what I may > be doing incorrectly. > > > Thanks, > > --Jordan > > Ole Nielsen wrote: > > Could you try > > C:\Python25\Lib\site-packages\pypar>mpirun -np 4 python test_pypar.py > > and let me know if that gives the same error? > > Has anyone else on the list seen this? > > Cheers and thanks > Ole > > > On Thu, Oct 1, 2009 at 2:49 AM, Jordan Atlas <jc...@co...> wrote: > >> Hi Ole, >> >> Thanks for your response. Following your advice, I was able to verify >> that I can get MPICH to run multiple processes that utilizes both of my >> machines' processors. >> >> Now I'm having trouble getting the pypar tests to run. If I type: >> >> C:\Python25\Lib\site-packages\pypar>mpirun -np 4 test_pypar.py >> >> Then I get the error: >> >> Unable to launch 'C:\Python25\Lib\site-packages\pypar\test_pypar.py', >> error 8: CreateProcess failed: >> Not enough storage is available to process this command. >> >> I assume it is talking about "RAM" storage, but my machine has over 2GB of >> RAM free right now, so I don't understand the problem. Am I calling the >> program correctly? >> >> Note that if I run test_pypar without calling mpirun, I get the output: >> >> C:\Python25\Lib\site-packages\pypar>test_pypar.py >> Pypar (version 2.1.0) initialised MPI OK with 1 processors >> I am processor 0 of 1 on node jca33-305. >> >> Please let me know if my question is unclear. >> >> Thanks, >> >> --Jordan >> >> >> Ole Nielsen wrote: >> >> Hi Jordan >> >> Great that you got pypar installed! >> >> As for the number of processors it is really a matter for the underlying >> MPI implementation. >> If you run a C program, such as the include timing measurement, using MPI >> you would have the same issue. >> >> Depending on the MPI implementation there are different ways of telling >> how many CPUs you got. >> With LAM MPI, there is a .machines file, with OPENMPI it is a matter for >> mpirun to specify. >> >> Typically, you run you parallel program on e.g. 8 processes using >> >> mpirun -np 8 <parallel_program> >> >> >> Hope this helps >> >> Cheers >> Ole >> >> >> PS - everything is better if you use Linux, though :-) >> >> >> On Thu, Sep 24, 2009 at 4:16 AM, Jordan Atlas <jc...@co...> wrote: >> >>> Hi Ole, >>> >>> After carefully following the readme instuctions and making a couple >>> changes to the compile files, I was able to get pypar installed on my >>> machine. However, it reports that my machine only has one processor, even >>> though it has two. >>> >>> Is there anything in particular I need to do to make sure pypar is >>> aware of multiple processors in windows? >>> >>> Thanks, >>> >>> --Jordan >>> >>> -- >>> *Jordan Atlas* >>> Email: jc...@co... >>> Chemical and Biomolecular Engineering >>> Cornell University >>> >> >> >> > > |
|
From: Jordan A. <jc...@co...> - 2009-10-02 19:05:26
|
Hi Ole and all, I get: C:\Python25\Lib\site-packages\pypar>mpirun -np 4 python test_pypar.py Unable to launch 'C:\Python25\Lib\site-packages\pypar\python', error 2: CreateProcess failed: The sy stem cannot find the file specified. So the same error. It's unclear to me where (i.e. in what function) the error is even being generated... I really appreciate any suggestions you may have. --Jordan-- On Wed, Sep 30, 2009 at 10:28 PM, Ole Nielsen <ole...@gm...>wrote: > Could you try > > C:\Python25\Lib\site-packages\pypar>mpirun -np 4 python test_pypar.py > > and let me know if that gives the same error? > > Has anyone else on the list seen this? > > Cheers and thanks > Ole > > > > On Thu, Oct 1, 2009 at 2:49 AM, Jordan Atlas <jc...@co...> wrote: > >> Hi Ole, >> >> Thanks for your response. Following your advice, I was able to verify >> that I can get MPICH to run multiple processes that utilizes both of my >> machines' processors. >> >> Now I'm having trouble getting the pypar tests to run. If I type: >> >> C:\Python25\Lib\site-packages\pypar>mpirun -np 4 test_pypar.py >> >> Then I get the error: >> >> Unable to launch 'C:\Python25\Lib\site-packages\pypar\test_pypar.py', >> error 8: CreateProcess failed: >> Not enough storage is available to process this command. >> >> I assume it is talking about "RAM" storage, but my machine has over 2GB of >> RAM free right now, so I don't understand the problem. Am I calling the >> program correctly? >> >> Note that if I run test_pypar without calling mpirun, I get the output: >> >> C:\Python25\Lib\site-packages\pypar>test_pypar.py >> Pypar (version 2.1.0) initialised MPI OK with 1 processors >> I am processor 0 of 1 on node jca33-305. >> >> Please let me know if my question is unclear. >> >> Thanks, >> >> --Jordan >> >> >> >> Ole Nielsen wrote: >> >> Hi Jordan >> >> Great that you got pypar installed! >> >> As for the number of processors it is really a matter for the underlying >> MPI implementation. >> If you run a C program, such as the include timing measurement, using MPI >> you would have the same issue. >> >> Depending on the MPI implementation there are different ways of telling >> how many CPUs you got. >> With LAM MPI, there is a .machines file, with OPENMPI it is a matter for >> mpirun to specify. >> >> Typically, you run you parallel program on e.g. 8 processes using >> >> mpirun -np 8 <parallel_program> >> >> >> Hope this helps >> >> Cheers >> Ole >> >> >> PS - everything is better if you use Linux, though :-) >> >> >> On Thu, Sep 24, 2009 at 4:16 AM, Jordan Atlas <jc...@co...> wrote: >> >>> Hi Ole, >>> >>> After carefully following the readme instuctions and making a couple >>> changes to the compile files, I was able to get pypar installed on my >>> machine. However, it reports that my machine only has one processor, even >>> though it has two. >>> >>> Is there anything in particular I need to do to make sure pypar is >>> aware of multiple processors in windows? >>> >>> Thanks, >>> >>> --Jordan >>> >>> -- >>> *Jordan Atlas* >>> Email: jc...@co... >>> Chemical and Biomolecular Engineering >>> Cornell University >>> >> >> >> > |
|
From: Ole N. <ole...@gm...> - 2009-10-01 02:28:17
|
Could you try C:\Python25\Lib\site-packages\pypar>mpirun -np 4 python test_pypar.py and let me know if that gives the same error? Has anyone else on the list seen this? Cheers and thanks Ole On Thu, Oct 1, 2009 at 2:49 AM, Jordan Atlas <jc...@co...> wrote: > Hi Ole, > > Thanks for your response. Following your advice, I was able to verify > that I can get MPICH to run multiple processes that utilizes both of my > machines' processors. > > Now I'm having trouble getting the pypar tests to run. If I type: > > C:\Python25\Lib\site-packages\pypar>mpirun -np 4 test_pypar.py > > Then I get the error: > > Unable to launch 'C:\Python25\Lib\site-packages\pypar\test_pypar.py', error > 8: CreateProcess failed: > Not enough storage is available to process this command. > > I assume it is talking about "RAM" storage, but my machine has over 2GB of > RAM free right now, so I don't understand the problem. Am I calling the > program correctly? > > Note that if I run test_pypar without calling mpirun, I get the output: > > C:\Python25\Lib\site-packages\pypar>test_pypar.py > Pypar (version 2.1.0) initialised MPI OK with 1 processors > I am processor 0 of 1 on node jca33-305. > > Please let me know if my question is unclear. > > Thanks, > > --Jordan > > > > Ole Nielsen wrote: > > Hi Jordan > > Great that you got pypar installed! > > As for the number of processors it is really a matter for the underlying > MPI implementation. > If you run a C program, such as the include timing measurement, using MPI > you would have the same issue. > > Depending on the MPI implementation there are different ways of telling how > many CPUs you got. > With LAM MPI, there is a .machines file, with OPENMPI it is a matter for > mpirun to specify. > > Typically, you run you parallel program on e.g. 8 processes using > > mpirun -np 8 <parallel_program> > > > Hope this helps > > Cheers > Ole > > > PS - everything is better if you use Linux, though :-) > > > On Thu, Sep 24, 2009 at 4:16 AM, Jordan Atlas <jc...@co...> wrote: > >> Hi Ole, >> >> After carefully following the readme instuctions and making a couple >> changes to the compile files, I was able to get pypar installed on my >> machine. However, it reports that my machine only has one processor, even >> though it has two. >> >> Is there anything in particular I need to do to make sure pypar is >> aware of multiple processors in windows? >> >> Thanks, >> >> --Jordan >> >> -- >> *Jordan Atlas* >> Email: jc...@co... >> Chemical and Biomolecular Engineering >> Cornell University >> > > > |
|
From: Ole N. <ole...@gm...> - 2009-09-23 21:45:09
|
---------- Forwarded message ---------- From: Ole Nielsen <ole...@gm...> Date: Thu, Sep 24, 2009 at 7:36 AM Subject: Re: multicore machines with pypar To: Jordan Atlas <jc...@co...> Hi Jordan Great that you got pypar installed! As for the number of processors it is really a matter for the underlying MPI implementation. If you run a C program, such as the include timing measurement, using MPI you would have the same issue. Depending on the MPI implementation there are different ways of telling how many CPUs you got. With LAM MPI, there is a .machines file, with OPENMPI it is a matter for mpirun to specify. Typically, you run you parallel program on e.g. 8 processes using mpirun -np 8 <parallel_program> Hope this helps Cheers Ole PS - everything is better if you use Linux, though :-) On Thu, Sep 24, 2009 at 4:16 AM, Jordan Atlas <jc...@co...> wrote: > Hi Ole, > > After carefully following the readme instuctions and making a couple > changes to the compile files, I was able to get pypar installed on my > machine. However, it reports that my machine only has one processor, even > though it has two. > > Is there anything in particular I need to do to make sure pypar is > aware of multiple processors in windows? > > Thanks, > > --Jordan > > -- > *Jordan Atlas* > Email: jc...@co... > Chemical and Biomolecular Engineering > Cornell University > |
|
From: Ole N. <ole...@gm...> - 2009-07-06 12:52:06
|
For your information Cheers Ole ---------- Forwarded message ---------- From: Jussi Lehtola <jus...@fe...> Date: Mon, Jul 6, 2009 at 8:49 AM Subject: Pypar finally in Fedora To: Ole Nielsen <ole...@gm...> Hi, thought you might want to know: pypar is finally in Fedora 10, 11 and EPEL for RHEL 5 and can be installed by # yum -y install pypar -- Jussi Lehtola Fedora Project Contributor jus...@fe... |
|
From: Ole N. <ole...@gm...> - 2009-07-01 23:05:07
|
gcc: unrecognized option '-showme' /usr/lib/gcc/i486-linux-gnu/4.3.3/../../../../lib/crt1.o: In function `_start': /build/buildd/glibc-2.9/csu/../sysdeps/i386/elf/start.S:115: undefined reference to `main' collect2: ld returned 1 exit status /home/mmckerns/bin/mpicc: line 248: echo: write error: Broken pipe /home/mmckerns/bin/mpicc: line 248: echo: write error: Broken pipe + '[' no = no ']' + unset 'allargs[0]' ++ expr 0 + 1 + argno=1 + for arg in '"$@"' + addarg=yes + case "$arg" in + linking=no + '[' yes = no ']' ++ expr 1 + 1 + argno=2 + for arg in '"$@"' + addarg=yes + case "$arg" in + '[' yes = no ']' ++ expr 2 + 1 + argno=3 + for arg in '"$@"' + addarg=yes + case "$arg" in + '[' yes = no ']' ++ expr 3 + 1 + argno=4 + for arg in '"$@"' + addarg=yes + case "$arg" in + '[' yes = no ']' ++ expr 4 + 1 + argno=5 + '[' 5 -eq 0 ']' + '[' no = yes ']' + mpilibs='-lmpich -lopa' + CFLAGS= + LDFLAGS=' ' + '[' -n '' ']' + '[' no = yes ']' + gcc -c /tmp/tmp6pVYfi.c -o /tmp/tmp6pVYfi.o -I/home/mmckerns/include + rc=0 + exit 0 gcc -I/home/mmckerns/include -L/home/mmckerns/lib -L/home/mmckerns/lib -Wl,-rpath,/home/mmckerns/lib -lmpich -lopa -lpthread -lpthread -lrt + '[' no = no ']' + unset 'allargs[0]' ++ expr 0 + 1 + argno=1 + for arg in '"$@"' + addarg=yes + case "$arg" in + linking=no + '[' yes = no ']' ++ expr 1 + 1 + argno=2 + for arg in '"$@"' + addarg=yes + case "$arg" in + '[' yes = no ']' ++ expr 2 + 1 + argno=3 + for arg in '"$@"' + addarg=yes + case "$arg" in + '[' yes = no ']' ++ expr 3 + 1 + argno=4 + for arg in '"$@"' + addarg=yes + case "$arg" in + '[' yes = no ']' ++ expr 4 + 1 + argno=5 + '[' 5 -eq 0 ']' + '[' no = yes ']' + mpilibs='-lmpich -lopa' + CFLAGS= + LDFLAGS=' ' + '[' -n '' ']' + '[' no = yes ']' + gcc -c /tmp/tmp6pVYfi.c -o /tmp/tmp6pVYfi.o -I/home/mmckerns/include + rc=0 + exit 0 running build running build_py creating build creating build/lib.linux-i686-2.5 creating build/lib.linux-i686-2.5/pypar copying compile_pypar_locally.py -> build/lib.linux-i686-2.5/pypar copying pypar_balancer.py -> build/lib.linux-i686-2.5/pypar copying network_timing.py -> build/lib.linux-i686-2.5/pypar copying pypar.py -> build/lib.linux-i686-2.5/pypar copying test_pypar.py -> build/lib.linux-i686-2.5/pypar copying compile.py -> build/lib.linux-i686-2.5/pypar copying test_init.py -> build/lib.linux-i686-2.5/pypar copying __init__.py -> build/lib.linux-i686-2.5/pypar copying __metadata__.py -> build/lib.linux-i686-2.5/pypar running build_ext building 'pypar.mpiext' extension creating build/temp.linux-i686-2.5 gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mmckerns/include -I/home/mmckerns/lib/python2.5/site-packages/numpy/core/include -I/home/mmckerns/include/python2.5 -c mpiext.c -o build/temp.linux-i686-2.5/mpiext.o mpiext.c: In function 'type_map': mpiext.c:127: warning: return makes integer from pointer without a cast mpiext.c: In function 'op_map': mpiext.c:168: warning: return makes integer from pointer without a cast gcc -pthread -shared build/temp.linux-i686-2.5/mpiext.o -L/home/mmckerns/lib -lmpich -lopa -lpthread -lrt -lopa' -o build/lib.linux-i686-2.5/pypar/mpiext.so /usr/bin/ld: cannot find -lopa' collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1 |
|
From: <Ole...@ga...> - 2009-06-11 21:00:35
|
Has anyone else had this problem?
Cheers
Ole
-----Original Message-----
From: Jean-Baptiste Juin [mailto:jb...@as...]
Sent: Friday, 12 June 2009 12:16 AM
To: Nielsen Ole
Subject: Re: python-C API compatibility problem with pypar [SEC=UNCLASSIFIED]
mpich2 use python scripts as daemon and to run mpi programs: mpd,
mpiexec, etc. Anyway after few try mpich2 recognized my python 2.5.4
installation. Then I built pypar with the same python 2.5.4. Same
error.
I will try again and eventually try another python parallel module.
Cheers,
Jean-Baptiste
Le Thu, 11 Jun 2009 11:56:27 +1000,
<Ole...@ga...> a écrit :
>
> I have no idea, sorry, but I am surprised that mpich2 has any
> reference to Python at all. If this causes trouble, you might
> consider switching to OpenMPI instead.
>
> Thoughts anyone?
>
> Cheers
> Ole
>
> -----Original Message-----
> From: Jean-Baptiste Juin [mailto:jb...@as...]
> Sent: Thursday, 11 June 2009 6:16 AM
> To: Nielsen Ole
> Subject: Re: python-C API compatibility problem with pypar
> [SEC=UNCLASSIFIED]
>
> Hi Ole,
>
> you are right: mpich2 use python2.4.
>
> Do you know how to force the python location when compiling mpich2 ?
> I installed python2.5.4 and 2.6.2 in my home and they are not found by
> the mpich2 configure script.
>
> Cheers,
>
> JB
>
>
> Le Wed, 10 Jun 2009 09:57:26 +1000,
> <Ole...@ga...> a écrit :
>
> > Hi Jean-Baptiste
> >
> > Thanks for you mail.
> >
> > I don't know what may be the matter so I forwarded your request onto
> > the pypar mailing list with a few thoughts of my own.
> >
> > Are you sure you are using the same version of Python for compiling
> > Pypar as the one you use to execute your MPI programs?
> >
> > Cheers
> > Ole
> >
> > -----Original Message-----
> > From: Jean-Baptiste Juin [mailto:jb...@as...]
> > Sent: Wednesday, 10 June 2009 7:54 AM
> > To: Nielsen Ole
> > Subject: python-C API compatibility problem with pypar
> >
> > Dear Ole,
> >
> > I'm using pypar for astronomy data analysis and I installed it on
> > two laptop for the moment wihtout any problem.
> >
> > I tried on a new machine with a functionnal mpich2 installation
> > (1.0.8p1). The compilation is fine, the problem comes when running a
> > mpi program, I got that error:
> >
> > /home/jbjuin/sciSofts/pypar_2.1.0_66/source/pypar.py:731:
> > RuntimeWarning: Python C API version mismatch for module mpiext:
> > This Python has API version 1012, module mpiext has version 1013.
> > import
> > mpiext /home/jbjuin/sciSofts/pypar_2.1.0_66/source/pypar.py:731:
> > RuntimeWarning: Python C API version mismatch for module mpiext:
> > This Python has API version 1012, module mpiext has version 1013.
> > import mpiext
> >
> > The python version I use is the 2.5.4,
> > pypar version is 2.1.0_66.
> >
> > Any idea to solve that issue ?
> >
> > Thanks for that wonderful module,
> >
> > Jean-Baptiste Juin
> >
> >
> >
> > --
> >
> > jb...@as...
> >
> > Astronomy Department
> > P. Universidad Catolica
> >
> >
> >
> >
> >
> >
>
>
> --
>
> jb...@as...
>
> Astronomy Department
> P. Universidad Catolica
>
>
>
>
>
>
--
jb...@as...
Astronomy Department
P. Universidad Catolica
|
|
From: <Ole...@ga...> - 2009-06-11 02:03:23
|
I have no idea, sorry, but I am surprised that mpich2 has any reference to Python at all. If this causes trouble, you might consider switching to OpenMPI instead.
Thoughts anyone?
Cheers
Ole
-----Original Message-----
From: Jean-Baptiste Juin [mailto:jb...@as...]
Sent: Thursday, 11 June 2009 6:16 AM
To: Nielsen Ole
Subject: Re: python-C API compatibility problem with pypar [SEC=UNCLASSIFIED]
Hi Ole,
you are right: mpich2 use python2.4.
Do you know how to force the python location when compiling mpich2 ?
I installed python2.5.4 and 2.6.2 in my home and they are not found by
the mpich2 configure script.
Cheers,
JB
Le Wed, 10 Jun 2009 09:57:26 +1000,
<Ole...@ga...> a écrit :
> Hi Jean-Baptiste
>
> Thanks for you mail.
>
> I don't know what may be the matter so I forwarded your request onto
> the pypar mailing list with a few thoughts of my own.
>
> Are you sure you are using the same version of Python for compiling
> Pypar as the one you use to execute your MPI programs?
>
> Cheers
> Ole
>
> -----Original Message-----
> From: Jean-Baptiste Juin [mailto:jb...@as...]
> Sent: Wednesday, 10 June 2009 7:54 AM
> To: Nielsen Ole
> Subject: python-C API compatibility problem with pypar
>
> Dear Ole,
>
> I'm using pypar for astronomy data analysis and I installed it on two
> laptop for the moment wihtout any problem.
>
> I tried on a new machine with a functionnal mpich2 installation
> (1.0.8p1). The compilation is fine, the problem comes when running a
> mpi program, I got that error:
>
> /home/jbjuin/sciSofts/pypar_2.1.0_66/source/pypar.py:731:
> RuntimeWarning: Python C API version mismatch for module mpiext: This
> Python has API version 1012, module mpiext has version 1013. import
> mpiext /home/jbjuin/sciSofts/pypar_2.1.0_66/source/pypar.py:731:
> RuntimeWarning: Python C API version mismatch for module mpiext: This
> Python has API version 1012, module mpiext has version 1013. import
> mpiext
>
> The python version I use is the 2.5.4,
> pypar version is 2.1.0_66.
>
> Any idea to solve that issue ?
>
> Thanks for that wonderful module,
>
> Jean-Baptiste Juin
>
>
>
> --
>
> jb...@as...
>
> Astronomy Department
> P. Universidad Catolica
>
>
>
>
>
>
--
jb...@as...
Astronomy Department
P. Universidad Catolica
|
|
From: Prabhu R. <pr...@ae...> - 2009-06-10 16:23:10
|
On 06/10/09 05:26, Ole...@ga... wrote: > Has anyone seen this warning before? > It appears that the compiled mpiext is newer than the Python he is running. > But, since the compile script is using Python itself, I don't understand what might be happening here. As you mention, the only reason this may happen is that the mpiext is compiled with a different Python version. I would clean out the build directory and rebuild/reinstall pypar to get this working. I think it is simply a matter of the mpiext.so file already existing from an earlier install or build. cheers, prabhu |
|
From: <Ole...@ga...> - 2009-06-09 23:56:28
|
Has anyone seen this warning before?
It appears that the compiled mpiext is newer than the Python he is running.
But, since the compile script is using Python itself, I don't understand what might be happening here.
Any ideas anyone?
Cheers and many thanks
Ole Nielsen
-----Original Message-----
From: Jean-Baptiste Juin [mailto:jb...@as...]
Sent: Wednesday, 10 June 2009 7:54 AM
To: Nielsen Ole
Subject: python-C API compatibility problem with pypar
Dear Ole,
I'm using pypar for astronomy data analysis and I installed it on two
laptop for the moment wihtout any problem.
I tried on a new machine with a functionnal mpich2 installation
(1.0.8p1). The compilation is fine, the problem comes when running a
mpi program, I got that error:
/home/jbjuin/sciSofts/pypar_2.1.0_66/source/pypar.py:731:
RuntimeWarning: Python C API version mismatch for module mpiext: This
Python has API version 1012, module mpiext has version 1013. import
mpiext /home/jbjuin/sciSofts/pypar_2.1.0_66/source/pypar.py:731:
RuntimeWarning: Python C API version mismatch for module mpiext: This
Python has API version 1012, module mpiext has version 1013. import
mpiext
The python version I use is the 2.5.4,
pypar version is 2.1.0_66.
Any idea to solve that issue ?
Thanks for that wonderful module,
Jean-Baptiste Juin
--
jb...@as...
Astronomy Department
P. Universidad Catolica
|
6 messages has been excluded from this view by a project administrator.
