PySCeS can now do parallel parameter scans on a cluster or locally on an SMP machine. Latest code available from SVN.
From the SVN log:
NEW FEATURE:
Added PySCeS parallel parameter scanning abilities using IPython.
IPython has great parallel computing functions and makes it easy to set up an IPython cluster, either locally on an SMP machine or in a networked cluster (http://ipython.scipy.org/doc/stable/html/parallel/index.html). It includes a MultiEngineClient to distribute out tasks onto the IPython engines, and a load-balanced TaskClient.
This functionality has been utilized to subclass Scanner into ParScanner. The new class has 2 main methods:
- Run() - runs the scan on the load-balanced TaskClient by splitting up the ScanSpace into a number of tasks using the self.scans_per_run parameter (defaults to 100). Optimal values will depend on the length of ScanSpace as well as the number of ipengines available. Visual feedback on the progress (number of tasks completed) is given.
- RunScatter() - distributes the ScanSpace evenly over all ipengines and collects all results when done. Unfortunately no feedback on progress is possible with this method.
Importantly the Scanner and ParScanner classes share exactly the same API in terms of instantiation, adding Scan Parameters and User Outputs. It is therefore a trivial matter to adapt your scripts :-)
Depending on the computational intensity of the tasks (e.g. MCA), speedups in excess of 85% of the theoretical maximum have been observed on a 15-CPU linux cluster.
A number of example scripts are included in the "examples" directory to illustrate the new functionality:
- testRunTc.py - script to test the Run() method which uses IPython's load-balanced TaskClient
- testRunScatter.py - script to test the RunScatter() method which uses the IPython's MultiEngineClient
- testparscanner.py - script combining the tests for the different parallel scanning methods
- testinvalidstate.py - script to test handling of invalid steady states between the serial and parallel scanning methods
- benchmark.ipy - IPython script (needs to be run from within IPython!) to benchmark the different serial and parallel scanning methods
- parallelscan.ipy - IPython script to demonstrate how to parallelise a simple 1-D scan (Scan1 method of PySCeS) in 5 lines of code (THIS DOES NOT NEED ANY OF THE NEW FUNCTIONALITY!)
The test scripts make use of the model "isola2a.psc" which is included in the "pscmodels" directory.
Happy testing, benchmarking and bugreporting!
-johann
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
PySCeS can now do parallel parameter scans on a cluster or locally on an SMP machine. Latest code available from SVN.
From the SVN log:
NEW FEATURE:
Added PySCeS parallel parameter scanning abilities using IPython.
IPython has great parallel computing functions and makes it easy to set up an IPython cluster, either locally on an SMP machine or in a networked cluster (http://ipython.scipy.org/doc/stable/html/parallel/index.html). It includes a MultiEngineClient to distribute out tasks onto the IPython engines, and a load-balanced TaskClient.
This functionality has been utilized to subclass Scanner into ParScanner. The new class has 2 main methods:
- Run() - runs the scan on the load-balanced TaskClient by splitting up the ScanSpace into a number of tasks using the self.scans_per_run parameter (defaults to 100). Optimal values will depend on the length of ScanSpace as well as the number of ipengines available. Visual feedback on the progress (number of tasks completed) is given.
- RunScatter() - distributes the ScanSpace evenly over all ipengines and collects all results when done. Unfortunately no feedback on progress is possible with this method.
Importantly the Scanner and ParScanner classes share exactly the same API in terms of instantiation, adding Scan Parameters and User Outputs. It is therefore a trivial matter to adapt your scripts :-)
Depending on the computational intensity of the tasks (e.g. MCA), speedups in excess of 85% of the theoretical maximum have been observed on a 15-CPU linux cluster.
A number of example scripts are included in the "examples" directory to illustrate the new functionality:
- testRunTc.py - script to test the Run() method which uses IPython's load-balanced TaskClient
- testRunScatter.py - script to test the RunScatter() method which uses the IPython's MultiEngineClient
- testparscanner.py - script combining the tests for the different parallel scanning methods
- testinvalidstate.py - script to test handling of invalid steady states between the serial and parallel scanning methods
- benchmark.ipy - IPython script (needs to be run from within IPython!) to benchmark the different serial and parallel scanning methods
- parallelscan.ipy - IPython script to demonstrate how to parallelise a simple 1-D scan (Scan1 method of PySCeS) in 5 lines of code (THIS DOES NOT NEED ANY OF THE NEW FUNCTIONALITY!)
The test scripts make use of the model "isola2a.psc" which is included in the "pscmodels" directory.
Happy testing, benchmarking and bugreporting!
-johann
Cool just got a new quad core laptop, perfect to try this out on …. will let you know how it goes.