dispy / News: Recent posts

Version 4.8.3 released

dispy version 4.8.3 has been released. In this version

  • Fixed parsing of ext_ip_addr option to dispynode.
  • Fixed recover_jobs function (which can be used for fault-recovery in case client crashes or loses network connection to nodes).
  • recover_file option to recover_jobs is now optional; if this option is not given, latest file with prefix _dispy_ is used (default falut-recovery files are of the form _dispy_*).
Posted by Giridhar Pemmasani 2017-08-01

Version 4.8.2 released

dispy version 4.8.2 has been released. In this version,

  • License has been changed to Apache 2.0 (from MIT).
  • If netifaces is not available, appropriate address is used for binding to UDP for broadcasting.
Posted by Giridhar Pemmasani 2017-06-27

Version 4.8.1 released

dispy version 4.8.1 has been released. In this version crash with dispyscheduler (due to conflicts of names of variables and methods with change from asyncoro to pycos) has been fixed.

Posted by Giridhar Pemmasani 2017-06-07

Version 4.8.0 released

dispy version 4.8.0 has been released. In this release asyncoro module has been replaced with pycos module.

Posted by Giridhar Pemmasani 2017-05-31

Version 4.7.7 released

dispy version 4.7.7 has been released. This version fixes crash of dispyscheduler when starting.

Posted by Giridhar Pemmasani 2017-05-24

dispy 4.7.6 released

dispy version 4.7.6 has been released. In this version httpd and dispynetrelay have been fixed to support IPv6.

Posted by Giridhar Pemmasani 2017-05-03

dispy 4.7.5 release

dispy version 4.7.5 has been released. In this version

  • Fixed IPv6 for Windows and OS X. IPv6 with Python 2.7 under Windows needs package win_inet_ptorn (it is not required for Python 3.6+). For IPv6 under OS X, package netifaces is required. Even if not required with other platforms, it is recommended to install netifaces to select suitable interface from multiple configured interfaces.
  • Fixed dispyscheduler's httpd functionality (to monitor computations scheduled with SharedJobScheduler)
Posted by Giridhar Pemmasani 2017-04-20

Version 4.7.4 released

dispy version 4.7.4 has been released. In this version

  • Fixed (default) host name resolution under IPv6.
  • Path and directory names in unicode are supported.
  • Added show_job_args parameter to httpd module to control whether job arguments should be shown in browser or not.
Posted by Giridhar Pemmasani 2017-04-06

Version 4.7.3 released

dispy version 4.7.3 has been released. In this version

  • Fixed fault recovery of jobs (with dispy.recover_jobs function) after client crashes / loses connection to nodes.
  • Fixed parsing and initializing configuration (with --config) saved in file by dispyscheduler.
Posted by Giridhar Pemmasani 2017-03-20

Version 4.7.2 released

dispy version 4.7.2 has been released. In this version

  • Fixed httpd module crash when vieweing jobs (in 'Node' page).
  • Added timeout optional parameter to wait and close methods of JobCluster and SharedJobCluster methods. timeout, if given, is maximum time in seconds to wait before all pending jobs are completed.
  • Added terminate optional parameter to close. If it is set to True any pending jobs are cancelled/terminated before cluster is closed.
  • Added save_config and config options to dispyscheduler to save and load configuration parameters in a file.
  • Added ping_interval option to dispynode to send periodic messages to discover nodes.
Posted by Giridhar Pemmasani 2017-03-15

Version 4.7.1 released

dispy version 4.7.1 has been released. In this version

  • SSL has been fixed. The fix is in asyncoro 4.4.0 version. If dispy is installed with pip, asyncoro will automatically be upgraded. Otherwise, please upgrade asyncoro to latest version.
  • Python function computations can now refer to 'dispy_node_name' and 'dispy_node_ip_addr'. These give name and IP address of node where computation is being executed.
  • dispynode will remove any files generated by computation and left behind when computation is closed. Earlier such files were left behind by dispynode, taking up disk space.
  • dispyscheduler has new option cleanup_nodes that if set (to True), will cause dispynode to cleanup (i.e., remove any files transferred and generated by computations) even if computations set cleanup=False. This can be used when nodes are being shared by multiple users and computations don't leave beind any files, which may take up disk space.
  • Added support for IPv6. It is strongly recommended that netifaces module is installed to use IPv6.
Posted by Giridhar Pemmasani 2017-01-18

Version 4.7.0 released

dispy version 4.7.0 has been released. In this version

  • Jobs (DispyJob instances) created by cluster.submit no longer have args and kwargs attributes. These attributes used to store parameters used in job creation (i.e., parameters to cluster.submit). When many jobs are created and not freed as soon as a job is finised, it is likely that the memory used to store such arguments may cause problems at the client / scheduler. Now these arguments are still stored in DispyJob instances, but are cleared by scheduler as soon as possible (e.g., when job is finished / terminated). If access to parameters used in job creation is necessary, then they can be maintained in client program using id attribute of job. In any case, when submitting large number of jobs, consider using bounded_submit.py to schedule only enough jobs to keep the processors busy and not all at once (which can cause memory issues).
  • When jobs are terminated (killed) at nodes, their status attribute is now set to DispyJob.Terminated. In the past few releases jobs may not always be terminated and even if they did, their status may indicate jobs finished and not terminated.
Posted by Giridhar Pemmasani 2016-12-20

Version 4.6.18 released

dispy version 4.6.18 has been released. In this version

  • Fixed dispy client and scheduler to use add_cluster etc. to use as coroutines instead of regular functions, as they use iterators shared with other coroutines. This prevents potential crashes when shared data structures are updated by other coroutines.
  • Fixed issue https://github.com/pgiri/dispy/issues/51 with transferring large files with OS X. The fix is actually in asyncoro version 4.3.3. If dispy is upgraded with pip, asyncoro will be upgraded automatically; otherwise, please upgrade asyncoro.
Posted by Giridhar Pemmasani 2016-11-30

Version 4.6.17 released

dispy version 4.6.17 has been released. In this release

  • NodeAllocate's allocate method is now called with additional parameter platform, which is value returned by platform.platform() on the node. This can be used to filter / allocate nodes depending on platform when cluster consists of nodes with different platforms.
  • Fixed dispynode to terminate when 'quit' or 'exit' commands are given.
Posted by Giridhar Pemmasani 2016-09-28

Version 4.6.16 released

dispy version 4.6.16 has been released. In this release

  • Added --client_shutdown option to dispynode. If this option is given, client program can call dispynode_shutdown() in cleanup function to shutdown dispynode.
  • --save_config option in dispynode now takes filename argument to save configuration in. In earlier releases --config option (which is used to load configuration) had to be specified to give the filename to save configuration.
  • dispynode shutdown improved - if dispynode is terminated (killed or interrupted with Ctrl+C), signal handler cleans up before quitting. In earlier releases (4.6.15 in particular), dispynode left behind some files that prevented next dispynode to start.
  • Fixed a race condition with dispyscheduler and client. In earlier releases if a client submitted jobs that took very little time, the scheduler may create new jobs with same ID as some job that finished at dispyscheduler, but not finished at client, causing client to ignore such jobs.
Posted by Giridhar Pemmasani 2016-08-16

Version 4.6.15 released

dispy version 4.6.15 has been released. In this version

  • Modules and files transferred to remote servers are saved with same relative paths as at client. With this, modules with multiple files, and submodules etc. can be sent with depends options. The paths are preserved only for files relative to client's current working directory; files with paths not under client's working directory don't preserve paths, as saving them at remote server with such paths may not be possible (due to permissions), or unwarrnated.
  • Saving and restoring configuration to initialize options to dispynode (to avoid having to give options every time / on every node) have been fixed.... read more
Posted by Giridhar Pemmasani 2016-07-19

Version 4.6.14 released

dispy version 4.6.14 has been released. In this version

  • Job arguments are stored in only DispyJob instance. In earlier versions another copy of arguments is stored in _DispyJob_ internal structure of dispy scheduler. When job arguments are large in space usage (which can happen when large arrays or lists are sent as job arguments), this required almost double the space, as these structures are kept in memory at least until jobs are finished. In this version arguments in _DispyJob_ internal structure refer to arguments in DispyJob structure (instead of saving a serialized copy of arguments). Note that in many examples submitted jobs are kept in a list and processed one after another and the list is never cleared. This is acceptable for simple cases, but when submittting large structures as arguments, it may be necessary to dispose of DispyJob instance, or at least clear job.args and job.kwargs attributes as soon as job is done (e.g., with callback feature) to avoid issues with memory usage.
  • Added bounded_submits.py example to illustrate how to submit enough jobs to keep processors busy, but not all at once, which can consume large amount of memory, especially if the job arguments are large structures, as explained above. The example can be esaily customized as appropriate for specific cluster.
  • Under some circumstances dispy scheduler may leave connections open, causing failures with "too many files open" issue. This has now been fixed. Thanks to Stylianos Kyriacou for pointing this issue.
  • Logging with 'error' or 'warning' levels is fixed; this was broken in 4.0 release. The fix actually is in asyncoro 4.1. If dispy is upgraded with pip, asyncoro will be upgraded as well; otherwise, please upgrade to asyncoro-4.1 (due to other changes, dispy-4.6.14 will require asyncoro-4.1).
Posted by Giridhar Pemmasani 2016-06-09

Version 4.6.13 released

dispy version 3.6.15 has been released. Following are changes since last release:

  • Default value for MaxFileSize, which is maximum size of file that can be transferred to dispynode (server) or dispyscheduler, has been changed from 10MB to 0, which means now there is no limit size of files to transfer with the default value. This can be changed with --max_file_size n option (to dispynode and dispyscheduler) to use n as maximum file size allowed.
  • By default dispynode removes any files transferred from client when computation is done. However, any files generated by computation were being left behind. Now, all files transferred / generated by computation are removed, unless cleanup is set to False (then it is up to clients to remove the files and directories).
  • Swap space information is added to DispyNodeAvailInfo so clients can use it to monitor node / application performance. This information is also shown in web clients when httpd is used.
Posted by Giridhar Pemmasani 2016-04-18

Version 4.6.12 released

dispy version 4.6.12 has been released. This release adds support for using psutil module to frequently gather and send node availability status information (availbe CPU as percent, memory in bytes and disk space in bytes) to cluster_status callback. This information is also shown in web browser (if httpd is used). The information is also sent to NodeAllocate's allocate method so clients can filter available nodes depending on computation's requirements.

Posted by Giridhar Pemmasani 2016-03-21

Version 4.6.11 released

dispy version 4.6.11 has been released. In this relase

  • dispy_job_depends keyword argument is supported for computations that are programs. dispy_job_depends can be used to send job-specific file(s). Until now this was suported only for computations that are Python functions.
  • Fixed dispynode to work with Python 3 under Windows. It seems multiprocessing.Process seems to wait for reading line(s) under Python 3 under Windows if main program reads standard input, so new jobs won't start until 'Enter' is pressed couple of times. For now the solution is to not support input commands with Python 3 under Windows; i.e., dispynode works as a daemon even if it is started from command line.
  • Fixed rescheduling of jobs when nodes are detected zombies or when nodes are restarted.
  • Fixed node_setup.py example to work under Windows.
Posted by Giridhar Pemmasani 2016-03-16

Version 4.6.10 released

dispy version 4.6.10 has been released. This release fixes initializing and closing nodes so a node is not initialized more than once (which then causes node not to respond to future computations due to a spurious computation still pending) and closing nodes is finished before client quits (otherwise the node may not be ready yet for new client if it starts quickly).

Posted by Giridhar Pemmasani 2016-03-07

dispy 4.6.9 release

dispy version 4.6.9 has been released. In this version

  • cooperative option has been added to dispyscheduler.py. If this option is given, or if client cluster is marked exclusive, the client(s) can update CPUs of node(s) (otherwise, clients won't be allowed to change CPUs, as this may prevent other clients to run their jobs). If node CPUs are changed by exclusive cluster and cooperative option is not used with dispyscheduler, the CPUs are reset to how they were before that cluster started. If CPUs are changed due to cooperative option, though, it is up to client clusters to cooperate and set/reset CPUs.
  • dispynode now removes modules loaded by computations from sys.modules when they are done, so that previous computation's modules are not used by new computations (because they were cached in sys.modules).
Posted by Giridhar Pemmasani 2016-03-02

dispy 4.6.8 release

dispy version 4.6.8 has been released. This version implements 3 scheduling algorithms for jobs in dispyscheduler (scheduler for SharedJobCluster):

  • fair_cluster_scheduler first picks a node with least load (i.e., jobs running divided by CPUs available), then among the clusters that can use that node chooses cluster that was least recently choosen from last time (i.e., has been pending longest time since last time a job was run for it). It then schedules earliest job created for that cluster on that node.
  • early_cluster_scheduler first picks a node with least load, then among the clusters that can use that node chooses cluster that was created earliest (i.e., client created SharedJobCluster earliest). It then schedules earliest job created for that cluster on that node.
  • The default is to choose earliest created job among all clusters that can use the node with least load.
Posted by Giridhar Pemmasani 2016-02-15

dispy 4.6.7 release

dispy version 4.6.7 has been released. This version fixes initialization of SharedJobCluster client (broken in 4.6.6 release).

Posted by Giridhar Pemmasani 2016-01-29

dispy 4.6.6 release

dispy version 4.6.6 has been released. Following is short summary of changes since last release:

  • In the last couple of releases, node discovery messages are sent from UDP server, by when TCP server may not have been ready to receive response from nodes, causing the client to not detect some nodes; it may have worked most of the time, but due to concurrency of coroutines, not always guaranteed. Now discovery messages are sent only after both UDP and TCP servers are ready to process responses from nodes.
  • In previous releases the computations were sent to nodes sequentially. Especially with SharedJibCluster, sending large computations to many nodes may cause unnecessary delays. Now nodes are setup concurrently.
  • If servers don't accept a computation (due to issues with computation, for example), appropriate error message is sent back to the client so the issue can be fixed; earlier error code -1 is sent which doesn't help to understand the reasons.
Posted by Giridhar Pemmasani 2016-01-27