dispy / News: Recent posts

dispy version 4.12.3 has been released. Major changes since previous release are:

• dispynode under Windows doesn't force daemon mode. Command line now can be used to see the
  status of computations, change CPUs, quit disypynode etc.

• It is possible now to use client and nodes with different versions of Python 3 even if pickle
  protocol versions are different. See pycos configuration variable PickleProtocolVersion; dispy
  uses pycos for communication so setting this variable appropriately will allow serialization /
  deserialization even if different Python versions use different default protocol versions (e.g.,
  Python 3.7 and 3.8 use different protocol versions). Note that this will not work between Python
  3 and Python 2.... read more

dispy version 4.12.2 has been released. In this version

• Fixed white space issues in httpd module in Python 3 so it works (same as httpd in Python 2).

dispy version 4.12.1 has been released. In this version

• Fixed dispynode in version 4.12.0 to run setup only client uses this feature; otherwise, dispynode crashes. With this programs that don't use setup, includingsample.py in examples, work now.

dispy version 4.12.0 has been released. In this version

• Added resetup_node method to JobCluster. This method can be used to run cleanup and setup function with different depends and arguments to prepare the node for a new run of jobs. This is useful to, e.g., replace in-memory data to run jobs with data.
• pulse_interval can now be set to as small as 0.1 (seconds).

dispy version 4.11.1 has been released. In this version

• Fixed dispyadmin.py to parse ports in configuration file (config.py) as integers.

dispy version 4.11.0 has been released. In this version

• Default ports used have been changed from 51347-51349 to 61590-61592. This may require appropriate changes to firewall setup.
• Instead of configuring different parts of dispy with different ports (e.g., dispynode with node_port, dispyscheduler with scheduler_port etc.), now only base dispy port (61590) needs to be given to all of them.
• Ports and other parameters can be changed in config.py module in dispy for convenience (useful in site-wide configuration).
• Added dispyadmin, an administrative interface to manage / control nodes. All the nodes, even the ones not used by any scheduler, can be controlled with this program. Only nodes started with --admin_secret option, nodes can be controlled with dispyadmin; otherwise, nodes are shown in the web interface but can't be managed.
• Renamed dispy.py command line program to dispy_cmd.py.
• Added depends and setup_args optional parameters to NodeAllocate. This feature can be used to send node and computation specific dependent files and to use node and computation spefiic arguments to run setup function. setup and cleanup functions can no longer be partial functions.
• Added relay optional parameter to dispy_provisional_result and dispy_send_file functions (that can be used in computation functions). If this parameter is False (default), results and files are sent from nodes to client directly, even if SharedJobCluster is used. If this parameter is True, they are sent via dispyscheduler, which is required if nodes can't communicate with client directly (e.g., if secret between dispyscheduler and nodes is different from that between dispyscheduler and client or SSL setup is different).

dispy version 4.10.6 has been released. In this version

• Added submit_job_id and submit_job_id_node methods to cluster. These methods take 'id' argument that is set to job's id attribute when job is created by scheduler so that when callback methods are called, the id can be used by those methods. If id is initialized by client after submit methods return job instance, the callback may be called before client can initialize (as dispy and callback run in other threads).
• Callback functions are now called with copies of jobs returned with submit methods (i.e., not the same instances). Thus callback functions should use id attribute of jobs to distinguish jobs instead of the job instance itself; if jobs need to be stored in a dictionary, for example, they should be managed with id attribute (which should be unique for each job created) and not job itself.
• If sending a job to a node fails (either node is disconnected from network or node is terminated), that node is removed from scheduler so if node is discovered later, it is initialized and reusable.
• Fixed job termination when suid feature is used (so one computation doesn't access another computation's files).

dispy version 4.10.5 has been released. In this version

• DispyNode instances now have tx and rx attributes that maintain amount data sent to / received from that node. print_status method and web interface now show this information.

dispy version 4.10.4 has been released. In this release

• Added support for Python 3.7.

dispy version 4.10.3 has been released. In this release

• Fixed job scheduler to handle jobs submitted with submit_node.
• Added discover_nodes method to JobCluster that clients can use to establish communication with nodes that may not be found when cluster initialized.

dispy version 4.10.2 has been released. In this version

• Added option force_cleanup to dispynode so all files transferred or created by computation are removed when computation is closed, even if computation disables cleanup.
• Fixed processing of provisional results so jobs are not dropped after first such result.
• Fixed close_node method.

dispy version 4.10.1 has been released. In this release

• Added support for suid and sgid features in Unix so a computation can not access files of other computations. See "Isolate Computation Files" for details and how to use this feature.
• id attribute of jobs is now set by dispy to successeive integers starting with 1.
• dispynode checks if any job process is dead at zombie_interval interval and if so, sends reply for it.
• If client / scheduler determines that a node is dead (or disconnected), jobs scheduled on it are not discarded now, so if a node comes back, client can accept replies for those jobs.
• set_node_cpus, deallocate_node etc. now accept DispyNode instances as arguments (in addition to host name or IP address, as done before).

dispy version 4.10.0 has been released. In this version

• Added support for *BSD (FreeBSD, NetBSD etc).
• Run setup and cleanup functions in a subprocess (instead of in main dispynode process). This prevents computations from interfering each other (i.e., there are no side effects from one computation loading modules etc.).
• Semantics of return value from setup function has changed. See setup function description in dispy JobCluster for details.
• Added --unsafe_setup option to dispynode to run setup and cleanup functions in main dispynode process, as used to be the case until 4.10.0 release.
• Fixed SharedJobCluster so accept remote address as scheduler_node parameter.
• Improved process termination when client cancels jobs and when clean option is used to terminate processes in prior run. If psutil module is available (which is strongly recommended), dispynode makes additional checks to make sure processes being terminated are from previous dispynode run (when clean option is used).

dispy version 4.9.1 has been released. In this version

• Fixed host name resolution when netifaces module is not available.
• Changed option listen_port for dispynetrelay to relay_port.

dispy version 4.9.0 has been released. In this release

• Added secret option and SSL support for dispynetrelay so it communicates only with schedulers that use same settings with those options.

dispy version 4.8.9 has been released. In this version

• Fixed processing ext_ip_addr option so even if netifaces module is available; in 4.8.8 this option didn't work if netifaces module is available.
• Added ipv4_udp_multicast and daemon options to dispynetrelay.

dispy version 4.8.8 has been released. In this version

• Added ipv4_udp_multicast option to control whether to use multicast or broadcast for UDP (to discover nodes) with IPv4.
• Fixed shutdown so it cleans up completely, including shutting down pycos. Now it is possible to shutdown dispy and start another dispy (not just another cluster) in the same program.
• Set multicast group address to a distinct from (in earlier releases this was a simple, generic form, that may interfere with other devices on the network).

dispy version 4.8.7 has been released. Changes since last release are:

• Fixed shutdown method to delete fault recovery file when SharedJobCluster used (and client doesn't crash).
• In some cases with IPv6 sockets, attribute IPV6_V6ONLY is not defined, causing crash during start; this is now fixed.
• Fixed parsing of ip_addrs and ext_ip_addrs fields saved in config file with dispynode.
• Fixed dispynode so changes to os.environ by one client don't interfere other clients.

dispy version 4.6.8 has been released. In this version

• Added deallocate_node and close_node methods to cluster.
• Added controls to web interface to allocate / deallocate / close node(s).
• Added "release" command to dispynode to release from current client if client is not active.

dispy version 4.8.5 has been released. In this version

• Fixed dispyscheduler crash when started in 'daemon' mode.
• When a node is rediscovered (i.e., after a node is lost and comes alive), use it again for any active clusters.
• When jobs are rescheduled or abandoned after a node is lost, call 'status' function (if provided) with appropriate job argument.
• Fixed issue with binding to UDP with Windows (issue #111])

dispy version 4.8.4 has been released. In this version

• Fixed client crash when SharedJobCluster is closed with Python 3.
• Improved warning messsages when client's UDP server binding to its port fails.

dispy version 4.8.3 has been released. In this version

• Fixed parsing of ext_ip_addr option to dispynode.
• Fixed recover_jobs function (which can be used for fault-recovery in case client crashes or loses network connection to nodes).
• recover_file option to recover_jobs is now optional; if this option is not given, latest file with prefix _dispy_ is used (default falut-recovery files are of the form _dispy_*).

dispy version 4.8.2 has been released. In this version,

• License has been changed to Apache 2.0 (from MIT).
• If netifaces is not available, appropriate address is used for binding to UDP for broadcasting.

dispy version 4.8.1 has been released. In this version crash with dispyscheduler (due to conflicts of names of variables and methods with change from asyncoro to pycos) has been fixed.

dispy version 4.8.0 has been released. In this release asyncoro module has been replaced with pycos module.