pypar-developers — Discussions about development of pypar

[Pypar-developers] Fwd: pypar and reduce

[Ole N. <ole...@gm...>]

Would anyone be so kind as to provide Norbert with an example of how to use
the reduce function.
Thanks
Ole Nielsen

---------- Forwarded message ----------
From: Norbert Marwan <ma...@pi...>
Date: Fri, May 8, 2009 at 6:14 PM
Subject: pypar and reduce
To: ole...@gm...


Dear Ole,

I have found your very nice pypar, but I'm still rather new with Python.
I'm not sure how i can use the reduce() within pypar. It would be nice
to have examples in the documentation of pypar. Could you perhaps
provide me with a short example on how to use reduce()? This would be
really great.

Thank you very much!
Best regards
Norbert

Re: [Pypar-developers] question about installing pypar [SEC=UNCLASSIFIED]

[<Ole...@ga...>]

Excellent, Thomas.

Thanks for that. Since I am not using Windows myself for this sort of thing, I'll leave it to the community to decide what the setup.py file (and compile.py) should ultimately look like in case of the win32 architecture.
I continue to be happy to incorporate updates as you make them available, but I'd like the opinion from the rest of you before I do so.

Thanks again
Ole Nielsen

-----Original Message-----
From: Tho...@cs... [mailto:Tho...@cs...]
Sent: Friday, 8 May 2009 3:07 PM
To: Nielsen Ole
Cc: pyp...@li...
Subject: RE: question about installing pypar [SEC=UNCLASSIFIED]

Hi Ole,

Thanks for that. I finally managed to get it all working OK on Windows with Visual Studio 2003 and MPICH2.
For info, here's what I had to do

I changed setup.py line 112 to replace
output = "gcc -L$MPICH_DIR\SDK.gcc\lib -lmpich -I$MPICH_DIR\SDK.gcc\include"
with
output = "gcc -L$MPICH_DIR\lib -lmpi -I$MPICH_DIR\include"
since using MPICH2 and not MPICH.

Then in compile.py line 112 (as well!?!) I changed the definition of python_include:
            python_include = os.path.join(sys.exec_prefix, 'include')

Then when running setup.py the compile and link commands printed were slightly wrong. I then ran them manually with those corrections:

"C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe" /c /nologo /Ox /MD /W3 /GX /DNDEBUG /I%MPICH_DIR%\include /IC:\Python25\lib\site-packages\numpy\core\include /IC:\Python25\include /IC:\Python25\PC /Tcmpiext.c /Fobuild\temp.win32-2.5\Release\mpiext.obj

"C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link.exe" /DLL /nologo /INCREMENTAL:NO /LIBPATH:%MPICH_DIR%\lib /LIBPATH:C:\Python25\libs /LIBPATH:C:\Python25\PCBuild mpi.lib /EXPORT:initmpiext build\temp.win32-2.5\Release\mpiext.obj /OUT:build\lib.win32-2.5\pypar\mpiext.pyd /IMPLIB:build\temp.win32-2.5\Release\mpiext.lib

After that, setup.py ran fine with the "build" and "install" targets, and I could run a demo example OK.
Thanks,

Thomas.

________________________________
From: Ole...@ga... [mailto:Ole...@ga...]
Sent: Thursday, 7 May 2009 10:21 AM
To: Poulet, Thomas (E&M, Kensington)
Cc: pyp...@li...
Subject: RE: question about installing pypar [SEC=UNCLASSIFIED]

Hi Thomas

>From what you are saying the problem lies with your MPI system.
The first thing to do is to test a simple C or Fortran based MPI program such as network_timing.c or other examples like that.
If that can't run, there is no point in trying pypar.

Perhaps you could try OpenMPI which seems to be the most developed MPI implementation at present.

I have CC'd this to the pypar mailing list so that others can chime in.

Cheers
Ole

-----Original Message-----
From: Tho...@cs... [mailto:Tho...@cs...]
Sent: Wednesday, 6 May 2009 7:31 PM
To: Nielsen Ole
Subject: question about installing pypar

Dear Oli,

I'm trying to install pypar on a windows machine with MPICH and visual studio 2003. I'm not having much luck though
mpirun -np 2 demo.py consistently returns


"C:\projects\pypar_2.1.0_66\demos>mpirun -np 2 demo.py
Unable to launch 'C:\projects\pypar_2.1.0_66\demos\demo.py', error 8: CreateProcess failed: Not enough storage is available to process this command."

This appears to have something to do with the thread stack size, but I'm not really sure.

I fell back to trying to compile network_timings.c (to try to test the mpich instalation) with the commands

cl network_timing.c /MD "/Ic:/program files/mpich/SDK/include" /Foa.obj
link a.obj "/LIBPATH:c:/program files/mpich/SDK/lib" mpich.lib

but this just results in the error message

C:\Documents and Settings\pou036\Desktop\shared>mpirun -localonly -np 2 a.exe
Wait for process 0 to write port to temporary file timed out

I was just wondering if you had any suggestions...
Cheers,

Thomas.

PS: as you can see I'm not really familiar with MPI on Windows and compilation issues...

Re: [Pypar-developers] question about installing pypar [SEC=UNCLASSIFIED]

[<Tho...@cs...>]

Hi Ole,

Thanks for that. I finally managed to get it all working OK on Windows with Visual Studio 2003 and MPICH2.
For info, here's what I had to do

I changed setup.py line 112 to replace
output = "gcc -L$MPICH_DIR\SDK.gcc\lib -lmpich -I$MPICH_DIR\SDK.gcc\include"
with
output = "gcc -L$MPICH_DIR\lib -lmpi -I$MPICH_DIR\include"
since using MPICH2 and not MPICH.

Then in compile.py line 112 (as well!?!) I changed the definition of python_include:
            python_include = os.path.join(sys.exec_prefix, 'include')

Then when running setup.py the compile and link commands printed were slightly wrong. I then ran them manually with those corrections:

"C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe" /c /nologo /Ox /MD /W3 /GX /DNDEBUG /I%MPICH_DIR%\include /IC:\Python25\lib\site-packages\numpy\core\include /IC:\Python25\include /IC:\Python25\PC /Tcmpiext.c /Fobuild\temp.win32-2.5\Release\mpiext.obj

"C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link.exe" /DLL /nologo /INCREMENTAL:NO /LIBPATH:%MPICH_DIR%\lib /LIBPATH:C:\Python25\libs /LIBPATH:C:\Python25\PCBuild mpi.lib /EXPORT:initmpiext build\temp.win32-2.5\Release\mpiext.obj /OUT:build\lib.win32-2.5\pypar\mpiext.pyd /IMPLIB:build\temp.win32-2.5\Release\mpiext.lib

After that, setup.py ran fine with the "build" and "install" targets, and I could run a demo example OK.
Thanks,

Thomas.

________________________________
From: Ole...@ga... [mailto:Ole...@ga...]
Sent: Thursday, 7 May 2009 10:21 AM
To: Poulet, Thomas (E&M, Kensington)
Cc: pyp...@li...
Subject: RE: question about installing pypar [SEC=UNCLASSIFIED]

Hi Thomas

>From what you are saying the problem lies with your MPI system.
The first thing to do is to test a simple C or Fortran based MPI program such as network_timing.c or other examples like that.
If that can't run, there is no point in trying pypar.

Perhaps you could try OpenMPI which seems to be the most developed MPI implementation at present.

I have CC'd this to the pypar mailing list so that others can chime in.

Cheers
Ole

-----Original Message-----
From: Tho...@cs... [mailto:Tho...@cs...]
Sent: Wednesday, 6 May 2009 7:31 PM
To: Nielsen Ole
Subject: question about installing pypar

Dear Oli,

I'm trying to install pypar on a windows machine with MPICH and visual studio 2003. I'm not having much luck though
mpirun -np 2 demo.py consistently returns


"C:\projects\pypar_2.1.0_66\demos>mpirun -np 2 demo.py
Unable to launch 'C:\projects\pypar_2.1.0_66\demos\demo.py', error 8: CreateProcess failed: Not enough storage is available to process this command."

This appears to have something to do with the thread stack size, but I'm not really sure.

I fell back to trying to compile network_timings.c (to try to test the mpich instalation) with the commands

cl network_timing.c /MD "/Ic:/program files/mpich/SDK/include" /Foa.obj
link a.obj "/LIBPATH:c:/program files/mpich/SDK/lib" mpich.lib

but this just results in the error message

C:\Documents and Settings\pou036\Desktop\shared>mpirun -localonly -np 2 a.exe
Wait for process 0 to write port to temporary file timed out

I was just wondering if you had any suggestions...
Cheers,

Thomas.

PS: as you can see I'm not really familiar with MPI on Windows and compilation issues...

Re: [Pypar-developers] question about installing pypar [SEC=UNCLASSIFIED]

[<Ole...@ga...>]

Hi Thomas

>From what you are saying the problem lies with your MPI system.
The first thing to do is to test a simple C or Fortran based MPI program such as network_timing.c or other examples like that.
If that can't run, there is no point in trying pypar.

Perhaps you could try OpenMPI which seems to be the most developed MPI implementation at present.

I have CC'd this to the pypar mailing list so that others can chime in.

Cheers
Ole

-----Original Message-----
From: Tho...@cs... [mailto:Tho...@cs...]
Sent: Wednesday, 6 May 2009 7:31 PM
To: Nielsen Ole
Subject: question about installing pypar

Dear Oli,

I'm trying to install pypar on a windows machine with MPICH and visual studio 2003. I'm not having much luck though
mpirun -np 2 demo.py consistently returns


"C:\projects\pypar_2.1.0_66\demos>mpirun -np 2 demo.py
Unable to launch 'C:\projects\pypar_2.1.0_66\demos\demo.py', error 8: CreateProcess failed: Not enough storage is available to process this command."

This appears to have something to do with the thread stack size, but I'm not really sure.

I fell back to trying to compile network_timings.c (to try to test the mpich instalation) with the commands

cl network_timing.c /MD "/Ic:/program files/mpich/SDK/include" /Foa.obj
link a.obj "/LIBPATH:c:/program files/mpich/SDK/lib" mpich.lib

but this just results in the error message

C:\Documents and Settings\pou036\Desktop\shared>mpirun -localonly -np 2 a.exe
Wait for process 0 to write port to temporary file timed out

I was just wondering if you had any suggestions...
Cheers,

Thomas.

PS: as you can see I'm not really familiar with MPI on Windows and compilation issues...

Re: [Pypar-developers] PyPar needs to be fixed to work with OpenMPI 1.3

[Ole <un...@in...>]

Hi Felix

Thanks for the mail and the links.
I am using pypar with OpenMPI, but need to check which version I use.

What you describe and what the links hint at sound nasty and I am not 
sure at this stage what to do about it.

If anyone can see the way forward in this regard, I'd love to hear from 
you and will be very happy to implement any patches to the pypar 
installer or pypar itself.

Cheers
Ole


Felix Richter wrote:
> Hi,
>
> it seems PyPar won't work with OpenMPI 1.3 any more. The problem I found is 
> the same as described here:
>
> http://www.open-mpi.org/community/lists/users/2009/02/8135.php
>
> Solutions are proposed here (an other thread):
>
> http://www.open-mpi.org/community/lists/users/2009/02/8186.php
>
> I'm afraid I am of no help in implementing the proposed workaround.
>
>
> Felix
>
> ------------------------------------------------------------------------------
> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based development
> software that enables intelligent coding and step-through debugging.
> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> _______________________________________________
> Pypar-developers mailing list
> Pyp...@li...
> https://lists.sourceforge.net/lists/listinfo/pypar-developers
>
>   

[Pypar-developers] PyPar needs to be fixed to work with OpenMPI 1.3

[Felix R. <fe...@ph...>]

Hi,

it seems PyPar won't work with OpenMPI 1.3 any more. The problem I found is 
the same as described here:

http://www.open-mpi.org/community/lists/users/2009/02/8135.php

Solutions are proposed here (an other thread):

http://www.open-mpi.org/community/lists/users/2009/02/8186.php

I'm afraid I am of no help in implementing the proposed workaround.


Felix

[Pypar-developers] help pypar

[guilherme b. <gui...@ya...>]

hi,

    I m student of master degree at fereral University of Rio
Grande do sul - brazil. And I need work with pypar, but i m having some problem
with installation on the cluster, so can you  help me with some
information about platform and version (mpi, python, pypar) work well on debian system?

Guilherme Dal Bianco

Computer engineer 


      Veja quais são os assuntos do momento no Yahoo! +Buscados
http://br.maisbuscados.yahoo.com

[Pypar-developers] Error when trying to import PyPar

[Craig F. <cf...@ie...>]

I get the following error message when I try "import pypar" at the Python command line, and when I attempt to submit a PyPar job through our job submission system:

[rdma_iba_init.c:91] Error initializing MVAPICH2 malloc library
Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(230): Initialization failed
MPID_Init(81)........: channel initialization failed


It seems like PyPar is having trouble starting the MVAPICH2 library, but I don't know why.  I checked with our system admin, and he verified that I was submitting the jobs correctly.  We verified that my "mpirun" executable is the same version of MPI that I used to build the PyPar extension.  I don't know where to go from here, because the error message isn't very helpful.  Does anyone have any suggestions about what might be causing the problem, or how I might get a more detailed error message?  Thanks!

NOTE: PyPar builds and runs fine on my dual-core desktop PC (gcc and openmpi).  I am having trouble on a RedHat Enterprise cluster, which uses the Intel C and Fortran compilers and mvapich2-0.9.8-15.  We also have mvapich-0.9.9 and openmpi-1.2.2-1 which I might try next.

    Craig Finch


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Pypar-developers] Successful compilation on x86_64 cluster

[Craig F. <oa...@ya...>]

I resolved this issue. It turned out that it was not a problem with the Python path.  The example script I was using was based on demo3.py from the pypar_demos tarball.  This script has the "import" statements embedded in a try/except block, which has the unfortunate side effect of suppressing useful error messages.  I put the import statements outside the try block, and received a more detailed error message.  Numpy was trying to load the Lapack shared library from the Intel Math Kernel Library, but it was failing.  It turned out that the MKL libraries had been moved to a new location after a recent upgrade of our cluster, and I had to change my LD_LIBRARY_PATH environment variable to point to the new location.

PyPar is still not running, but I will start a new thread for that issue.

    Craig Finch



----- Original Message ----
From: Ole Nielsen <ole...@gm...>
To: Craig Finch <cf...@ie...>; Pypar-developers <Pyp...@li...>
Sent: Saturday, October 18, 2008 1:16:27 AM
Subject: [Pypar-developers] Successful compilation on x86_64 cluster

Hello Craig and tanks for your interest in pypar and the compilation
information.

Regarding Python finding its packages when invoked using mpirun I
think it is a matter of exporting the environment variable PYTHONPATH
using something like this

mpirun -x PYTHONPATH .....

I have made an alias taking care of that.

Please let me know if this addresses you problem.

Cheers
Ole

> From: Craig Finch <cf...@ie...>
> To: pyp...@li...
> Date: Fri, 17 Oct 2008 14:55:15 -0700 (PDT)
> Subject: Successful compilation on x86_64 cluster
> Here' s a short summary of what I did to compile pypar and install it in my home directory on our Intel Xeon 64 bit cluster with Intel compilers and MPI:
>
> Compile and install pypar:
> python setup.py config --compiler=intel install --home=~
>
> The mpiext library compiles and installs without error, but it can't be loaded by Python.  I compiled the binary manually with the following command:
>
> icc -fPIC -i-dynamic -D_EM64T_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS -D_SHMEM_COLL_ -DRDMA_CM -I/usr/include/python2.4 -I/home/cfinch/lib64/python/numpy/core/include -I/usr/include -O2 -I/usr/mpi/intel/mvapich2-0.9.8-15/include -L/usr/mpi/intel/mvapich2-0.9.8-15/lib -Wl,-rpath -Wl,/usr/mpi/intel/mvapich2-0.9.8-15/lib -lmpich -L/usr/lib64 -lrdmacm -libverbs -libumad -lpthread -lrt -I/usr/include/python2.4 -c mpiext.c -o mpiext.o
>
> I then linked it using mpicc:
> mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium
>
> And copied it to
> cp mpiext.so ~/lib64/python/pypar/
>
> I don't claim to understand the options I had to use to get icc to compile the extension correctly.  I basically looked at the files setup.py and compile.py, and kludged together options until something worked.  It would be great if someone could explain how this process works and include it in the docs for future reference.
>
> I do have one question.  When I submit a pypar job using mpirun, python can't seem to find the Python packages (numpy, pypar, etc). that are installed in my home directory (~/lib64/python).  I verified that ~/lib64/python is in the Python path for the Python processes that are started by mpirun, but they still can't find my packages.  Any suggestions how to make them available?  I don't think our cluster admin has any experience with parallel Python.
>
>   Craig Finch
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
>
> ---------- Forwarded message ----------
> From: pyp...@li...
> To:
> Date:
> Subject: confirm 469f2ca2493a178d1ef6720486793ce4522b0f2f
> If you reply to this message, keeping the Subject: header intact,
> Mailman will discard the held message.  Do this if the message is
> spam.  If you reply to this message and include an Approved: header
> with the list password in it, the message will be approved for posting
> to the list.  The Approved: header can also appear in the first line
> of the body of the reply.
>

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Pypar-developers mailing list
Pyp...@li...
https://lists.sourceforge.net/lists/listinfo/pypar-developers


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Pypar-developers] pypar

[Ole N. <ole...@gm...>]

Thanks for your mail

I am glad that you find pypar useful.

To answer your question, I have no plans to add the extensions you
mention, but would be very happy to assist in incorporating them if
someone else were to implement and test them.

Best regards
Ole Nielsen


On Mon, Oct 13, 2008 at 4:32 PM, Reynaldo Gil <gi...@gm...> wrote:
> Hellow. I thanks for you  flexible python mpi implementation. Do you
> plan to support in pypar mpi2 extensións like dynamic process
> creation?
> Thanks
> Gil
>

[Pypar-developers] Successful compilation on x86_64 cluster

[Ole N. <ole...@gm...>]

Hello Craig and tanks for your interest in pypar and the compilation
information.

Regarding Python finding its packages when invoked using mpirun I
think it is a matter of exporting the environment variable PYTHONPATH
using something like this

mpirun -x PYTHONPATH .....

I have made an alias taking care of that.

Please let me know if this addresses you problem.

Cheers
Ole

> From: Craig Finch <cf...@ie...>
> To: pyp...@li...
> Date: Fri, 17 Oct 2008 14:55:15 -0700 (PDT)
> Subject: Successful compilation on x86_64 cluster
> Here' s a short summary of what I did to compile pypar and install it in my home directory on our Intel Xeon 64 bit cluster with Intel compilers and MPI:
>
> Compile and install pypar:
> python setup.py config --compiler=intel install --home=~
>
> The mpiext library compiles and installs without error, but it can't be loaded by Python.  I compiled the binary manually with the following command:
>
> icc -fPIC -i-dynamic -D_EM64T_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS -D_SHMEM_COLL_ -DRDMA_CM -I/usr/include/python2.4 -I/home/cfinch/lib64/python/numpy/core/include -I/usr/include -O2 -I/usr/mpi/intel/mvapich2-0.9.8-15/include -L/usr/mpi/intel/mvapich2-0.9.8-15/lib -Wl,-rpath -Wl,/usr/mpi/intel/mvapich2-0.9.8-15/lib -lmpich -L/usr/lib64 -lrdmacm -libverbs -libumad -lpthread -lrt -I/usr/include/python2.4 -c mpiext.c -o mpiext.o
>
> I then linked it using mpicc:
> mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium
>
> And copied it to
> cp mpiext.so ~/lib64/python/pypar/
>
> I don't claim to understand the options I had to use to get icc to compile the extension correctly.  I basically looked at the files setup.py and compile.py, and kludged together options until something worked.  It would be great if someone could explain how this process works and include it in the docs for future reference.
>
> I do have one question.  When I submit a pypar job using mpirun, python can't seem to find the Python packages (numpy, pypar, etc). that are installed in my home directory (~/lib64/python).  I verified that ~/lib64/python is in the Python path for the Python processes that are started by mpirun, but they still can't find my packages.  Any suggestions how to make them available?  I don't think our cluster admin has any experience with parallel Python.
>
>   Craig Finch
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
>
> ---------- Forwarded message ----------
> From: pyp...@li...
> To:
> Date:
> Subject: confirm 469f2ca2493a178d1ef6720486793ce4522b0f2f
> If you reply to this message, keeping the Subject: header intact,
> Mailman will discard the held message.  Do this if the message is
> spam.  If you reply to this message and include an Approved: header
> with the list password in it, the message will be approved for posting
> to the list.  The Approved: header can also appear in the first line
> of the body of the reply.
>

[Pypar-developers] Successful compilation on x86_64 cluster

[Craig F. <cf...@ie...>]

Here' s a short summary of what I did to compile pypar and install it in my home directory on our Intel Xeon 64 bit cluster with Intel compilers and MPI:

Compile and install pypar:
python setup.py config --compiler=intel install --home=~

The mpiext library compiles and installs without error, but it can't be loaded by Python.  I compiled the binary manually with the following command:

icc -fPIC -i-dynamic -D_EM64T_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS -D_SHMEM_COLL_ -DRDMA_CM -I/usr/include/python2.4 -I/home/cfinch/lib64/python/numpy/core/include -I/usr/include -O2 -I/usr/mpi/intel/mvapich2-0.9.8-15/include -L/usr/mpi/intel/mvapich2-0.9.8-15/lib -Wl,-rpath -Wl,/usr/mpi/intel/mvapich2-0.9.8-15/lib -lmpich -L/usr/lib64 -lrdmacm -libverbs -libumad -lpthread -lrt -I/usr/include/python2.4 -c mpiext.c -o mpiext.o

I then linked it using mpicc:
mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium

And copied it to 
cp mpiext.so ~/lib64/python/pypar/

I don't claim to understand the options I had to use to get icc to compile the extension correctly.  I basically looked at the files setup.py and compile.py, and kludged together options until something worked.  It would be great if someone could explain how this process works and include it in the docs for future reference.

I do have one question.  When I submit a pypar job using mpirun, python can't seem to find the Python packages (numpy, pypar, etc). that are installed in my home directory (~/lib64/python).  I verified that ~/lib64/python is in the Python path for the Python processes that are started by mpirun, but they still can't find my packages.  Any suggestions how to make them available?  I don't think our cluster admin has any experience with parallel Python.

   Craig Finch


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Pypar-developers] Fwd: Pypar 2.1.0_53 Windows compilation

[Prabhu R. <pr...@ae...>]

Ole Nielsen wrote:
> I have been trying to compile/install pypar on my XP+MSYS/MINGW
> machine, but I have not been able to. I'm able to build it just fine
> using "python setup.py build -c mingw32" but I'm not able to install
[...]
> I have also tried to manually copy the pypar build directory into
> site-packages, which works from ipython but will not run the demo.py
> files from the cooamn-line using mpirun.
> 
> Any advice would greatly be appreciated.

Perhaps this thread will help:

http://thread.gmane.org/gmane.comp.python.enthought.devel/14644/focus=14713

cheers,
prabhu

[Pypar-developers] Fwd: Pypar 2.1.0_53 Windows compilation

[Ole N. <ole...@gm...>]

Can someone help Chiraj please?
Ole

---------- Forwarded message ----------
From: Chiraj <som...@gm...>
Date: Mon, Oct 6, 2008 at 8:37 PM
Subject: Pypar 2.1.0_53 Windows compilation
To: ole...@gm...


Hi Ole,

I have been trying to compile/install pypar on my XP+MSYS/MINGW
machine, but I have not been able to. I'm able to build it just fine
using "python setup.py build -c mingw32" but I'm not able to install
it into the python2.5 site-packages. The command I used for
installation was exactly the same as the above but with install. It
replies with the following message:
---
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
  or: setup.py --help [cmd1 cmd2 ...]
  or: setup.py --help-commands
  or: setup.py cmd --help

error: invalid command 'mingw32'
---
or with this if I remove the -c option:
---
running install
running build
running build_py
running build_ext
error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.
---

I have also tried to manually copy the pypar build directory into
site-packages, which works from ipython but will not run the demo.py
files from the cooamn-line using mpirun.

Any advice would greatly be appreciated.

Regards
Chiraj

Re: [Pypar-developers] Pypar: success and failure

[Ole N. <ole...@gm...>]

Hello Craig

Thanks for your mail and interest in pypar.
The software runs on a large number of different facilities - and
although some of the docu is out of date, the package itself should be
fine.
Please let me know what obstacles you have encountered and also cc the
mailing list so that we can sort it out for you.

Further on, you experiences would feed into the documentation.

Cheers and thanks
Ole Nielsen

On Sat, Sep 20, 2008 at 7:03 AM, Craig Finch <cf...@ie...> wrote:
> Ole,
> I tried to send you an email at the address Ole...@an... that I found in the PyPar docs, but I guess that's out of date.  If you receive this, and you are interested in hearing about my "lessons learned" and suggestions for PyPar, please reply!
>
>   Craig Finch
>
>
>
>
>

Re: [Pypar-developers] pypar: receiving without specifying source?

[Ole N. <ole...@gm...>]

By the way

The recently contributed pypar_balancer.py from Felix Richter implements a
general master-slave framework and you can see how pypar.any_source is used
in conjunction with status.source.

You can see that code at

http://pypar.svn.sourceforge.net/viewvc/pypar/source/pypar_balancer.py?view=markup

Cheers
Ole

On Fri, Jul 25, 2008 at 6:34 PM, Ole Nielsen
<ole...@gm...>wrote:

> Hi Magnar
>
> Thanks for your interest in pypar.
>
> It is possible to receive messages from any source in the MPI standard and
> this is also the case in pypar as in this snippet taken from the bundled
> mandel brot example mandel_parallel_dynamic.py
>
> R, status = pypar.receive(pypar.any_source,
>                                          tag=result_tag,
>                                          return_status=True)
>
> The status object contains various information and in particular you can
> get the source for the message just received through
>
> status.source
>
>
> Hope this helps
> Cheers
> Ole
>
> On Mon, Jul 21, 2008 at 6:12 PM, <mag...@st...> wrote:
>
>> Hi,
>>
>> Is it possible to receive a message without having specified a source in
>> pypar, but with just specifying e.g. a tag? This would come in handy when
>> a master node distributes jobs and waits for results, to send out a new
>> job each time a slave is done. The master would like to first receive the
>> results from the first slave who finishes his job, but we cannot know
>> beforehand which slave that will be.
>>
>> Magnar K. Bugge
>>
>>
>>
>

Re: [Pypar-developers] pypar: receiving without specifying source?

[Ole N. <ole...@gm...>]

Hi Magnar

Thanks for your interest in pypar.

It is possible to receive messages from any source in the MPI standard and
this is also the case in pypar as in this snippet taken from the bundled
mandel brot example mandel_parallel_dynamic.py

R, status = pypar.receive(pypar.any_source,
                                         tag=result_tag,
                                         return_status=True)

The status object contains various information and in particular you can get
the source for the message just received through

status.source


Hope this helps
Cheers
Ole

On Mon, Jul 21, 2008 at 6:12 PM, <mag...@st...> wrote:

> Hi,
>
> Is it possible to receive a message without having specified a source in
> pypar, but with just specifying e.g. a tag? This would come in handy when
> a master node distributes jobs and waits for results, to send out a new
> job each time a slave is done. The master would like to first receive the
> results from the first slave who finishes his job, but we cannot know
> beforehand which slave that will be.
>
> Magnar K. Bugge
>
>
>

Re: [Pypar-developers] Pypar problem: segfaults on me :(

[Ole N. <ole...@gm...>]

Hey Guy

I'll be installing Ubuntu Hardy in the next few weeks. I'll obviously try to
install pypar on that and get back to you with the results.
Pypar has installed and worked on a wide range of platforms for the last 7
years, so I can't imagine this is anything serious.

Have you tried to run the test suite:
mpirun -np 4 python test_pypar.py

Please remind me if I forget to reply and also please get back to me if you
figure out what might be the problem.

Cheers
Ole Nielsen

On Mon, Jun 2, 2008 at 4:03 PM, Guy K. Kloss <G....@ma...> wrote:

> Forgot to mention ...
>
> I'm running on an Ubuntu Hardy system, gcc 4.2.3, on an Intel Core 2 Duo
> ...
>
> Guy
>
> --
> Guy K. Kloss
> Institute of Information and Mathematical Sciences
> Te Kura Putaiao o Mohiohio me Pangarau
> Room 2.63, Quad Block A Building
> Massey University, Auckland, Albany
> Private Bag 102 904, North Shore Mail Centre
> voice: +64 9 414-0800 ext. 9585   fax: +64 9 441-8181
> eMail: G....@ma...  http://www.massey.ac.nz/~gkloss/<http://www.massey.ac.nz/%7Egkloss/>
>

Re: [Pypar-developers] Simple Pypar Load Balancing class

[Ole N. <ole...@gm...>]

Hi Felix

Thanks for your posting and your contribution to pypar. I think that the
load balance class is a novel idea that is likely to be very useful to
parallel Python applications. I have tested it and added it to the pypar
repository (revision 51) where it'll become part of next release. Currently,
it sits in the pypar source directory along with pypar.py itself and
mpiext.c - do you have any preferences or suggestions for where it should
live and also how to best install it along with the rest of pypar?
One option, of course, would be to include the class within pypar itself. I
am open to suggestions.

Cheers and thanks
Ole

On Tue, May 13, 2008 at 9:38 PM, Felix Richter <fe...@ph...>
wrote:

> Dear Pypar friends,
>
> here is a simple load balancer I wrote based on demo3.py. We currently use
> it
> for a number of our computations.
>
> The idea is to write a simple class that describes the work and to pass it
> to
> the balancer which feeds the hungry nodes.
>
> This might also be interesting to Jonas.
>
> Best regards,
>
>        Felix
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Pypar-developers mailing list
> Pyp...@li...
> https://lists.sourceforge.net/lists/listinfo/pypar-developers
>
>

[Pypar-developers] Simple Pypar Load Balancing class

[Felix R. <fe...@ph...>]

Dear Pypar friends,

here is a simple load balancer I wrote based on demo3.py. We currently use it 
for a number of our computations. 

The idea is to write a simple class that describes the work and to pass it to 
the balancer which feeds the hungry nodes.

This might also be interesting to Jonas.

Best regards,

	Felix

Re: [Pypar-developers] pypar bug+patch

[Ole N. <ole...@gm...>]

Hi Jim

Thanks again for taking interest in pypar. I looked closer at you
suggestion, incorporated it and ran all the tests and examples.
It is now part of the latest release  (pypar_2.1.0_49). It incorporates


   - Race-condition patch from Jim Bosch
   - Functionality and demo for Bsend by C Makassikis
   - Deprecated obsolete functions
   - Migrated Mandelbrot demos to numpy
   - Fixed demo3.py for version 2 thanks to Felix Richter

Let me know how it goes.
Cheers and thanks
Ole Nielsen
Canberra, Australia



On Fri, Apr 18, 2008 at 6:53 AM, Jim Bosch <jf...@uc...> wrote:

> I've found (and fixed, I think, with the attached patch) a bug in pypar:
> in the current version (SVN), when sending a generic python object, pypar
> actually passes two MPI messages (the "control info" message, and the actual
> object).  However, it does not require that both of these come from the same
> node, so if a third node sends a message that is received between the
> "control info" message and the data message, pypar gets confused.
>
> The attached patch adds an option to "receive_control_info" that causes it
> to return the source node, and this is used to require that the data message
> come from the same node.  It also contains some pickle error trapping and
> re-raising, which I found useful to add a little more information to aid in
> debugging those sorts of errors.
>
> Thanks!
>
> Jim Bosch
>
>
>
>

[Pypar-developers] New pypar release

[Ole N. <ole...@gm...>]

Dear all who are interested in pypar or parallel programming with Python.


I have released a new pypar version (2.1.0_49). It incorporates


   - Race-condition patch from Jim Bosch
   - Functionality and demo for Bsend by C Makassikis
   - Deprecated obsolete functions
   - Migrated Mandelbrot demos to numpy
   - Fixed demo3.py for version 2 thanks to Felix Richter


I also verified that all tests and demos work on a Linux 32 bit platform.

Thanks to everyone for all the contributions and looking forward to hearing
how it goes.



Best regards
Ole Nielsen
Canberra, Australia

Re: [Pypar-developers] pypar examples

[Ole N. <ole...@gm...>]

By the way, you might like to go through these lectures. They are
aimed at building three different parallel versions of computing the
Mandelbrot fractal using pypar.

http://datamining.anu.edu.au/~ole/work/teaching/summerschool2005/

The code is also available under demos/mandelbrot_example in the pypar
distribution.

Good luck
Ole




On Fri, Apr 25, 2008 at 10:03 PM, "Boström, Jonas"
<Jon...@as...> wrote:
> Hi Ole,
>  I'm planning to make some of my python codes to run in parallel by using your pypar module.
>  However,  I'm a "learn-by-looking-at-other-peoples-codes" kind of guy...and I when I google "pypar examples/tutorials" I do not find much.
>  Would you mind terribly sending a few of your (basic) example pypar codes? Or point me to other codes?
>  All the best
>  Jonas
>  ps. Credit were credit is due, of course!
>
>  #############################
>  Jonas Boström, Assoc. Professor
>  Computational Chemistry
>  Lead Discovery Department
>  AstraZeneca R&D Mölndal
>  Pepperedsleden 1
>  S-431 50 Mölndal
>  email: jon...@as...
>  phone: +46 31 706 52 51
>  #############################
>
>

[Pypar-developers] Fwd: pypar examples

[Ole N. <ole...@gm...>]

Can anyone help this guy with some additional examples (and please CC
this mailing list)?
Best regards
Ole


---------- Forwarded message ----------
From: "Boström, Jonas" <Jon...@as...>
Date: Fri, Apr 25, 2008 at 10:03 PM
Subject: pypar examples
To: ole...@gm...


Hi Ole,
 I'm planning to make some of my python codes to run in parallel by
using your pypar module.
 However,  I'm a "learn-by-looking-at-other-peoples-codes" kind of
guy...and I when I google "pypar examples/tutorials" I do not find
much.
 Would you mind terribly sending a few of your (basic) example pypar
codes? Or point me to other codes?
 All the best
 Jonas
 ps. Credit were credit is due, of course!

 #############################
 Jonas Boström, Assoc. Professor
 Computational Chemistry
 Lead Discovery Department
 AstraZeneca R&D Mölndal
 Pepperedsleden 1
 S-431 50 Mölndal
 email: jon...@as...
 phone: +46 31 706 52 51
 #############################

[Pypar-developers] Non-blocking communication (reply to justus)

[Ole N. <ole...@gm...>]

Oh by the way - the four calls (mpi_isend, mpi_irecv, mpi_wait and
mpi_test) you need for non-blocking communication are listed here:

http://www.amath.washington.edu/~beowulf/mpi/noblock.html

Cheers and good lock


On Fri, Apr 11, 2008 at 10:13 AM, Justus Schwabedal
<jus...@gm...> wrote:
> Hi there.
>  In my programs a pypar.receive statement is cpu blocking. Is there a
>  way to not make it block my cpu and just wait until a massage comes in
>  or is it inherent to mpi?
>  Yours, Justus
>
>
>  On Apr 7, 2008, at 8:23 PM, Ole Nielsen wrote:
>
>  > Ah, I see.
>  > Your idea is great. The only question is what system to base it on.
>  >
>  > Let me know how it goes.
>  >
>  > Cheers
>  > Ole
>  >
>  > On Mon, Mar 31, 2008 at 2:25 AM, Justus Schwabedal
>  > <jus...@gm...> wrote:
>  >>
>  >> For what I know you can just help seti by letting them use your
>  >> computer.
>  >> But you cannot commit jobs yourself, which is what I was thinking of.
>  >>
>  >>
>  >>
>  >>
>  >> On Mar 30, 2008, at 4:39 AM, Ole Nielsen wrote:
>  >> Hi again Justus
>  >>
>  >> I think what you are saying is possible, although I suspect other
>  >> frameworks
>  >> migh be better suited for loosely parallel jobs such as SETI@home
>  >> type
>  >> frameworks. MPI (and pypar)  is specifically geared towards tightly
>  >> coupled
>  >> parallel jobs on dedicated clusters.
>  >> I guess, I am suggesting that you scope the options before settling
>  >> on
>  >> pypar.
>  >> If you go ahead, though, I'll be very interested in following the
>  >> progress
>  >> and commenting on designs.
>  >>
>  >> Best regards
>  >> Ole
>  >>
>  >>
>  >> On Fri, Mar 21, 2008 at 1:33 PM, Justus Schwabedal
>  >> <jus...@gm...> wrote:
>  >>>
>  >>>
>  >>> Good Morning Ole!
>  >>> I'd be glad to contribute a little although I'm more on the novice
>  >>> then on
>  >> the expert side concerning python. To mpi I'm pretty new, but I
>  >> started to
>  >> actively work with it recently and I'm farely interested.
>  >>>  You can probably guess the potentials lying in code distribution.
>  >>> Do you
>  >> think something like a community parallelization would be possible
>  >> with
>  >> python? I'm thinking of something similar to the seti project, but
>  >> such that
>  >> everybody can commit jobs and they are distributed intelligentely.
>  >> What are
>  >> your thoughts? I'm not quite sure about the technical problems
>  >> occuring.
>  >> Afterall it would be absolutely conformal with gnu/linux ideas. One
>  >> would
>  >> have to control cpu and memory usage of the "external processes"
>  >> and the
>  >> rights they are given on the machine. Well, I guess that's beyond
>  >> the scope
>  >> of pypar but also beyond the possible work one could get done in a
>  >> year or
>  >> so.
>  >>> Yours, Justus
>  >>>
>  >>>
>  >>>
>  >>>
>  >>> On Mar 20, 2008, at 2:10 PM, Ole Nielsen wrote:
>  >>>
>  >>> Hi Justus
>  >>>
>  >>> Thank you very much for your mail and your interest in pypar.
>  >>> Pypar has steadily grown and served us and several others over the
>  >>> past
>  >> 6-7 years.
>  >>> Many people have made contributions over the years and the whole
>  >>> idea is
>  >> that pypar should be maintained by a community.
>  >>>
>  >>> I think your ideas are great and it wouldn't hurt to throw them
>  >>> out there
>  >> for some feedback. If you are keen, I could make an account for you
>  >> in the
>  >> subversion repository where pypar lives and your pypar-tools could
>  >> evolve
>  >> from there. That'd be great actually.
>  >>>
>  >>> As for the adhoc implementation, that'll be fine initially with the
>  >> understanding that we'd like to see the code base grow into robust,
>  >> general
>  >> and flexible tools in the future.
>  >>> One thing that I have found crucial is the use of a good unit test
>  >>> suite
>  >> from the word go as well as version control. I can assist with
>  >> either.
>  >>>
>  >>> The idea of distributing the code as (executable I assume) strings
>  >>> is
>  >> definitely novel and interesting. I assume this would be useful for
>  >> distributed systems that don't have a common NFS mounted filesystem
>  >> where
>  >> source code is accessible by all computer nodes. Is that correct?
>  >>>
>  >>> Distributing code objecs could be done, but the underlying
>  >>> implementation
>  >> would serialise the objects as strings using the underlying MPI
>  >> calls.
>  >> Distributing strings is straight forward and probably more efficient.
>  >>>
>  >>> I am not aware of anyone else doing this, but it'd be worth a
>  >>> quick search
>  >> or post to find out.
>  >>>
>  >>> Looking forward to hear more
>  >>>
>  >>> Cheers
>  >>> Ole Nielsen
>  >>> Canberra, Australia
>  >>>
>  >>>
>  >>>
>  >>> On Mon, Mar 17, 2008 at 1:28 AM, Justus Schwabedal
>  >> <jus...@gm...> wrote:
>  >>>
>  >>>> Cheers,
>  >>>> I found out about pypar a couple of weeks  ago. I instantly
>  >>>> started to
>  >>>> develop a distributed system which distributes source code in the
>  >>>> form
>  >>>> of strings. I would like to contribute this into the project by
>  >>>> maybe
>  >>>> opening something like pypar-tools. My implementation is kindof
>  >>>> adhoc
>  >>>> and I would like to ask if I can get some tips concerning this
>  >>>> issue.
>  >>>> Can I distribute code objects instead of strings? Would that be
>  >>>> more
>  >>>> efficient? Questions like that are of my concern. Is somebody doing
>  >>>> something similar?
>  >>>> Yours, Justus
>  >>>>
>  >>>>
>  >> -------------------------------------------------------------------------
>  >>>> This SF.net email is sponsored by: Microsoft
>  >>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>  >>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>  >>>> _______________________________________________
>  >>>> Pypar-developers mailing list
>  >>>> Pyp...@li...
>  >>>> https://lists.sourceforge.net/lists/listinfo/pypar-developers
>  >>>>
>  >>>
>  >>>
>  >>>
>  >>
>  >>
>  >>
>
>
>  -------------------------------------------------------------------------
>  This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>  Don't miss this year's exciting event. There's still time to save $100.
>  Use priority code J8TL2D2.
>  http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
>  _______________________________________________
>  Pypar-developers mailing list
>  Pyp...@li...
>  https://lists.sourceforge.net/lists/listinfo/pypar-developers
>