Re: [Pypar-developers] Successful compilation on x86_64 cluster

[Craig F. <oa...@ya...>]

I resolved this issue. It turned out that it was not a problem with the Python path.  The example script I was using was based on demo3.py from the pypar_demos tarball.  This script has the "import" statements embedded in a try/except block, which has the unfortunate side effect of suppressing useful error messages.  I put the import statements outside the try block, and received a more detailed error message.  Numpy was trying to load the Lapack shared library from the Intel Math Kernel Library, but it was failing.  It turned out that the MKL libraries had been moved to a new location after a recent upgrade of our cluster, and I had to change my LD_LIBRARY_PATH environment variable to point to the new location.

PyPar is still not running, but I will start a new thread for that issue.

    Craig Finch



----- Original Message ----
From: Ole Nielsen <ole...@gm...>
To: Craig Finch <cf...@ie...>; Pypar-developers <Pyp...@li...>
Sent: Saturday, October 18, 2008 1:16:27 AM
Subject: [Pypar-developers] Successful compilation on x86_64 cluster

Hello Craig and tanks for your interest in pypar and the compilation
information.

Regarding Python finding its packages when invoked using mpirun I
think it is a matter of exporting the environment variable PYTHONPATH
using something like this

mpirun -x PYTHONPATH .....

I have made an alias taking care of that.

Please let me know if this addresses you problem.

Cheers
Ole

> From: Craig Finch <cf...@ie...>
> To: pyp...@li...
> Date: Fri, 17 Oct 2008 14:55:15 -0700 (PDT)
> Subject: Successful compilation on x86_64 cluster
> Here' s a short summary of what I did to compile pypar and install it in my home directory on our Intel Xeon 64 bit cluster with Intel compilers and MPI:
>
> Compile and install pypar:
> python setup.py config --compiler=intel install --home=~
>
> The mpiext library compiles and installs without error, but it can't be loaded by Python.  I compiled the binary manually with the following command:
>
> icc -fPIC -i-dynamic -D_EM64T_ -D_SMP_ -DUSE_HEADER_CACHING -DONE_SIDED -DMPID_USE_SEQUENCE_NUMBERS -D_SHMEM_COLL_ -DRDMA_CM -I/usr/include/python2.4 -I/home/cfinch/lib64/python/numpy/core/include -I/usr/include -O2 -I/usr/mpi/intel/mvapich2-0.9.8-15/include -L/usr/mpi/intel/mvapich2-0.9.8-15/lib -Wl,-rpath -Wl,/usr/mpi/intel/mvapich2-0.9.8-15/lib -lmpich -L/usr/lib64 -lrdmacm -libverbs -libumad -lpthread -lrt -I/usr/include/python2.4 -c mpiext.c -o mpiext.o
>
> I then linked it using mpicc:
> mpicc -shared mpiext.o -o mpiext.so -mcmodel=medium
>
> And copied it to
> cp mpiext.so ~/lib64/python/pypar/
>
> I don't claim to understand the options I had to use to get icc to compile the extension correctly.  I basically looked at the files setup.py and compile.py, and kludged together options until something worked.  It would be great if someone could explain how this process works and include it in the docs for future reference.
>
> I do have one question.  When I submit a pypar job using mpirun, python can't seem to find the Python packages (numpy, pypar, etc). that are installed in my home directory (~/lib64/python).  I verified that ~/lib64/python is in the Python path for the Python processes that are started by mpirun, but they still can't find my packages.  Any suggestions how to make them available?  I don't think our cluster admin has any experience with parallel Python.
>
>   Craig Finch
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
>
> ---------- Forwarded message ----------
> From: pyp...@li...
> To:
> Date:
> Subject: confirm 469f2ca2493a178d1ef6720486793ce4522b0f2f
> If you reply to this message, keeping the Subject: header intact,
> Mailman will discard the held message.  Do this if the message is
> spam.  If you reply to this message and include an Approved: header
> with the list password in it, the message will be approved for posting
> to the list.  The Approved: header can also appear in the first line
> of the body of the reply.
>

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Pypar-developers mailing list
Pyp...@li...
https://lists.sourceforge.net/lists/listinfo/pypar-developers


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com