dispy / News: Recent posts

dispy 4.6.0 release

dispy version 4.6.0 has been released. The changes since last release are:

  • Added service_start, service_stop and service_end options to dispynode to allow a node to run jobs during specific hours of the day.
  • Added save_config and config optoins to dispynode to save given options in a file and use that file to start dispynode in all machines in a cluster with those options (instead of having to list options for each run).
  • Added submit_node option to JobCluster and SharedJobCluster to schedule a job to given node. This method, along with cluster_status callback can be used to implement custom job schedulers in the client itself.
  • Added exclusive option to SharedJobCluster. If this option is set to True, that cluster is executed exclusively, without sharing nodes with other clusters.
  • An issue since 4.5.3 release with cluster not detecting nodes (apparently seems to occur with Google Cloud) has been fixed; thanks to Stylianos Kyriacou.
Posted by Giridhar Pemmasani 2015-10-28

dispy 4.5.5 release

dispy version 4.5.5 has been released. This version fixes Issue #21 - Using same cluster more than once.

Posted by Giridhar Pemmasani 2015-09-08

dispy 4.5.4 release

dispy version 4.5.4 has been released.

This version supports setup and cleanup parameters under Windows with some limitations (compared to Linux, OS X and other Posix systems, where there are no limitations); these should be Python functions or partial functions. See JobCluster and node_setup.py and node_shvars.py examples for more details on limitations with Windows.

Posted by Giridhar Pemmasani 2015-08-13

dispy 4.5.3 release

dispy version 4.5.3 has been released. This version supports passing class instances in client to servers with Python 3 under Windows. Processes started with multiprocessing in Python 3 under Windows use __mp_main__ namespace, so user defined code is executed in that namespace so unpickling/deserialization of objects works.

Posted by Giridhar Pemmasani 2015-08-03

dispy 4.5.2 release

dispy version 4.5.2 has been released.

Version 4.5 introduced a feature to keep a computation's global variables in a (dictionary) variable in that computation. This feature works with Linux, OS X and other Unix variants, but not with Windows. The implementation of this feature broke Windows, as the variable couldn't be sent as argument to multiprocessing.Process. Version 4.5.2 fixes this issue so dispynode works with Windows again.

Posted by Giridhar Pemmasani 2015-07-23

dispy 4.5.1 release

dispy version 4.5.1 release fixes issue in 4.5 version with loading httpd module in dispyscheduler.

Posted by Giridhar Pemmasani 2015-07-08

dispy 4.5 release

Short summary of changes in version 4.5:

  • Fixed dispyscheduler to import httpd module.
  • dispynode stores global variables of each computation in its own namespace. With this, if more than once cluster (from same scheduler) is being executed, global variables initialized by one computation doesn't interfere with those of another. Moreover, a computation doesn't need to worry about having to remove global variables in 'cleanup' - they will be thrown away when computation is closed.
  • Fixed 'del_cluster' to remove cluster from all the nodes before removing cluster. Otherwise, because 'yield' is used to close node, if multiple clusters are used, a removed cluster can be accessed by scheduler through node's list of clusters. Thanks to Nikola Knezevic.
Posted by Giridhar Pemmasani 2015-06-30

dispy 4.4 release

Short summary of changes in version 4.4:

  • 'setup' and 'cleanup' can be partial (Python) functions (in addition to being normal functions). This way nodes can be setup with different functions (i.e.,. partial functions can be called with different arguments for different nodes).
  • httpd module and web pages support multiple clusters in the same program. The web pages show each cluster separately, as well as a special cluster that combines information for all clusters in a cluster named '* Combined'.
  • dispyscheduler program supports '--httpd' option. When this option is given, httpd server is started so all the clusters currently using the scheduler can be monitored in web browser. The names of clusters have ' @ <client IP="">' appended.
  • Added 'timeout' and 'terminate_pending' to 'recover_jobs' function in dispy so fault recovery can be used to close the nodes, if necessary.
  • 'nodes' parameter for JobCluster and SharedJobCluster can be given as a list of 'NodeAllocate' objects. This gives more flexibility to allocate nodes, as well as customize allocation (for example, to use fewer CPUs during specific time interval).
  • dispynode and dispyscheduler programs have '--msg_timeout' option to adjust socket timeout used to send/receive messages. Default is 5 seconds. If necessary, for example, to send large files over slow network, the timeout can be adjusted with this option. dispy module also has MsgTimeout variable that can be adjusted in the client program appropriately.
Posted by Giridhar Pemmasani 2015-06-07

dispy 4.3 released

dispy version 4.3 has been released. Most of the changes since last release are fixes. The one change is simplification of fault recovery. Now the clients store recovery information always and in case of crash, function recover_jobs in dispy module can be used to retrieve the results of jobs and release the nodes. See Fualt Recovery for more details.

Posted by Giridhar Pemmasani 2015-05-25

dispy 4.2 released

dispy version 4.2 has been released. The changes are:

  • Fixes issue with distributing computations as programs (Bug 11)[https://sourceforge.net/p/dispy/bugs/11/].
  • Added 'close_node' function in dispy module so client can close currently running computations on the node after recovering from lost connection / crash. See doc strings for 'close_node' on how to use it.
  • Added '--clean' option to dispynode and dispyscheduler. If this option is given, dispynode and dispyscheduler remove any files that are left behind by previous runs. The files are now stored by dispynode in /tmp/dispy/node and by dispyscheduler in /tmp/dispy/scheduler.
Posted by Giridhar Pemmasani 2015-05-07

dispy 4.1 released

dispy version 4.1 has been released with following changes from version 4.0:

  • httpd module now includes interface to manage cluster (changing CPUs when JobCluster is used, adding nodes), monitoring jobs on nodes, terminating jobs etc.
  • httpd can be used with SSL so https protocol can be used to monitor and manage cluster with web browsers.
  • Using httpd module is slightly different from the way it was in version 4.0; see httpd module and example.
Posted by Giridhar Pemmasani 2015-05-04

dispy 4.0 released

dispy version 4.0 has been released. In this release,

  • cluster_status parameter has been added to JobCluster and SharedJobCluster. This should be set to a function or method. Whenever there is a change in status of a job (such as created, submitted to a node, finished executing etc.) or a change in status of a node (initialized for given cluster, closed), given function is called. See dispy page for details.
  • It includes httpd module that can be used to easily create a web interface to view cluster status. See View Cluster Status for information.
  • asyncoro module is no longer included in dispy package; it must be installed separately. If Python's package index (PyPI / pip) is used for installing dispy, asyncoro will be automatically installed from PyPI. If dispy is installed manually, then asyncoro must be installed at appropriate dist-packages.
  • dispy package includes basic documentation (in text form) and a fews examples. If installed with PyPI, these are stored under the directory where dispy module is stored (e.g., /usr/local/lib/python2.7/dist-packages/dispy).
Posted by Giridhar Pemmasani 2015-04-21

dispy 3.22 released

dispy version 3.22 has been released. With this release, dependencies can be distributed for each job (in addition to distributing dependencies for the whole computation). Job's dependencies (functions, classes, modules and files if computations are Python functions, and modules and files if computations are standalone programs) are sent with the job to the node and removed after the job is done. Job's dependencies should be given as a list to 'cluster.submit()' when creating a job. See Cluster methods for details.... read more

Posted by Giridhar Pemmasani 2015-03-24

dispy 3.21 released

dispy version 3.21 has been released. This version adds support for computations to transfer files to client.

Posted by Giridhar Pemmasani 2015-03-10

dispy 3.20 released

dispy version 3.20 has been released. This version fixes SSL setup so dispy, dispynode and dispyscheduler can be configured to use SSL certificates as documented.

Posted by Giridhar Pemmasani 2014-11-21

Version 3.19 released

dispy version 3.19 has been released. This release

  • Fixes issue with 'print' function for Python 2.7 version (in dispy 3.18, print function from __future__ has been imported, which made any user program using standard print statement to be invalid),
  • asyncoro has been upgraded to latest release 3.1,
  • Packaging/installing with pypi for Python 3.1+ has been fixed.
Posted by Giridhar Pemmasani 2014-11-04

Version 3.18 released

dispy version 3.18 has been released. In this version dispyscheduler.py (used with SharedJobCluster) has been fixed to close the socket used in transferring files.

Posted by Giridhar Pemmasani 2014-10-24

Version 3.17 released

dispy version 3.17 has been released. This release adds 'setup' parameter to JobCluster and SharedJobCluster. If given, this must be a Python function which is run on a node before running any jobs. The parameter 'cleanup' can also be a function, which is run at a node after the computation is done. See dispy and Examples for details.

Posted by Giridhar Pemmasani 2014-10-16

Version 3.16 released

dispy version 3.16 has been released. This release fixes following issues:

  • dispynode executes programs with subprocess.Popen without 'shell=True' to avoid security issues, as explained in subprocess reference. This apparently doesn't work under Windows when Python programs are executed, as Windows command shell is needed to invoke Python interpreter to execute the given program. This is fixed in this release by invoking the interpreter directly (without using 'shell=True' option).... read more
Posted by Giridhar Pemmasani 2014-10-07

Version 3.15 released

dispy version 3.15 has been released. This version fixes crash with dispyscheduler in Windows. Files are now copied to tempfile.gettempdir() instead of '/tmp'.

Posted by Giridhar Pemmasani 2014-06-27

Version 3.14 released

dispy version 3.14 has been released. This version fixes socket timeout error processing in Windows (in asyncoro module).

Posted by Giridhar Pemmasani 2014-06-15

Version 3.13 released

dispy version 3.13 has been released. This version fixes an issue with processing provisional results. asyncoro has also been updated to latest version.

Note that asyncoro project now supports distributed / parallel computing (among other features) where computations are distributed to nodes, as done by dispy. With asyncoro the computation tasks and client(s) can communicate using message passing, which is not possible with dispy (other than sending provisional results). However, asyncoro doesn't provide job scheduling. See 'discoro_client.py' in asyncoro's files for an example.

Posted by Giridhar Pemmasani 2014-06-10

Version 3.12 released

dispy version 3.12 has been released. The major changes since version 3.11 are:

  • With JobCluster, multiple network interfaces on the client computer are now supported:
    • If ip_addr parameter is None (default), all interfaces are used.
    • If ip_addr is a string (name or IP address), only that interface is used.
    • If ip_addr is a list of strings, then each interface corresponding to that list is used.
  • Similarly, ext_ip_addr can be either a string or list of strings.
  • SharedJobCluster has extra parameter scheduler_port. This should be the port used by dispyscheduler (if different from default value 51349).
Posted by Giridhar Pemmasani 2014-04-21

Version 3.11 released

dispy version 3.11 has been released. Summary of changes since version 3.10:

  • ssh tunneling (remote port forwarding through ssh) supported. It can be used to work with nodes in remote networks. See Examples for details.
  • Added 'polling_interval' option. This can be used if nodes can't connect to client (i.e., gateway/firewall prevents connections and there is no way to setup port forwarding, even with ssh tunneling). However, this is inefficient, so it must be used only when needed and if number of nodes is small.
  • Added setup script for installing dispy. The script is provided by Roger-Bermudez Chacon.
  • Updated asyncoro to latest release (1.5).
Posted by Giridhar Pemmasani 2014-03-17

Version 3.10 released

dispy version 3.10 has been released. In this release

  • Fixed 'callback' processing. In dispy-3.9 callback processing was broken (dispy would raise exception after calling callback).
  • Fixed 'dest_path' processing in dispynode. Until dispy-3.9 (not sure since when) if 'dest_path' option is used and the given directory already exists on the node, dispynode still tries to create it, causing exception.
  • Fixed a typo in dispyscheduler (used in case of SharedCluster), which would prevent sending 'ping' messages.
Posted by Giridhar Pemmasani 2013-10-05