Activity for Giridhar Pemmasani
-
Giridhar Pemmasani posted a comment on ticket #21
I have emailed you an implementation of this feature.
-
I will look into this and find a generic way to add resource requirements. I will post patch to try once ready.
-
The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...
-
The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...
-
The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...
-
The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...
-
The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...
-
This is already possible with NodeAllocate feature. For example if you would like to only use nodes that have 32G RAM and 16 cores, you can filter nodes with: if __name__ == '__main__': ... class AllocateNode(dispy.NodeAllocate): def allocate(self, cluster, ip_addr, name, cpus, avail_info=None, platform='*'): if cpus < 16: return 0 # don't use this node if avail_info.memory < (32*1024*1024*1024): return 0 # don't use this node return (cpus - 2) # reserve 2 cores for other uses cluster = dispy.JobCluster(compute,...
-
Well, the last couple of years I have been working on improving and adding features to dispy to make it enterprise product (with support). One major feature is to integrate with multiple cloud platforms so applications can dynamically allocate and use cloud computing (in addition to local/remote compute nodes) to meet deadlines / peak demands. If you are using dispy in enterprise, would you or your team be interested in exploring? If so, email me at pgiri@yahoo.com or giri@uniaccel.com and I will...
-
Hmm, client using SharedJobCluster only connects to scheduler; it doesn't communicate with nodes. So there is no need to support more than one address for client. You may want to use -i option more than once to start dispyscheduler.py to use multiple addresses at scheduler.
-
Thanks for submitting bug and fix. As you know, 4.10.x is quite old release. While I can commit this fix in github, releasing will confuse many users that use new(er) releases. Since this issue is fixed in 4.11.0, I would prefer not to push this. If you don't mind, can you describe your use case? Thanks again!
-
pycos version 4.10.0 has been released. In this release: Chagend dispycos Client constructor to take scheduler location as optional parameter. Default value for this parameter causes constructor to start private/local scheduler automatically so no need for user programs to start one for such cases. Replaced MonitorException with MonitorStatus class with 3 attributes that can be used for different use cases. Errors when running node_setup or server_setup or tasks with dispycos are sent back to client...
-
pycos version 4.9.1 has been released. In this release Added 'dispycos_close_node' function that computations may call to close the node where that computation is executing. Added pico_client.py and pico_service.py examples that implement tiny (pico size) services. dispycos servers now can recover even if computation tasks execute 'exit' or 'os.exit' either directly or indirectly (e.g., a library function that may exit). Fixed dispycos to wait for pending tasks to finish if computation closes. Fixed...
-
Version 4.12.3 released
-
pycos version 4.9.0 has been released. Major changes from previous release are: Value of tasks now is the value with 'return(value)' (in Python 3.7+) or with 'raise StopIteration(value)' in versions of Python prior to 3.7. It is no longer last value used with 'yield value', as done in Pycos versions up to 4.8. Thus, now tasks must use 'return' or 'raise StopIteration' to finish with value; otherwise, it is treated as 'return None' or 'raise StopIteration(None)'. Renamed 'Computation' in dispycos...
-
Version 4.12.2 released
-
Version 1.63 released
-
Version 4.12.1 released
-
Version 4.12.0 released
-
pycos version 4.8.15 has been released. In this version Added support for abandoning zombie dispycos nodes. This can be controlled with abandon_zombie_nodes parameter with Compute and abandon_zombie method. Added tasks and close_node methods to Compute.
-
pycos version 4.8.14 has been released. In this version pycos now works with Python 3.8 under Windows. Fixed dispycos scheduler to set scheduler port as configured in config.py.
-
I have tested with Windows but not as much as with Linux, so likely there are issues with Windows. Can you try passing type to getaddrinfo, so the try block is: info = None for addr in socket.getaddrinfo(node, None, 0, socket.SOCK_STREAM): if not info or addr[0] == socket.AF_INET: info = addr assert info Thanks.
-
I have tested with Windows but not as much as with Linux, so likely there are issues with Windows. Can you try passing type to getaddrinfo, so the try block is: info = None for addr in socket.getaddrinfo(node, None, 0, socket.SOCK_STREAM): if not info or addr[0] == socket.AF_INET: info = addr assert info Thanks.
-
Version 4.11.1 released
-
pycos version 4.8.13 has been released. In this version Configuration parameters, such as MsgTimeout, MaxConnectionErrors etc. have been moved to config.py file in the package. This file can be modified for site-specific configuration instead of having to modify them in each program. If zombie (dispycos) nodes come back, connection is re-established so such nodes can be used for distributed computing. Fixed docker build script for Python 2.7. Version 4.8.12 was also updated a few minutes ago, but...
-
Version 4.11.0 released
-
Version 4.10.6 released
-
pycos version 4.8.11 has been released. In this version Added checks to make sure files installed for Python versions up to 3.6 are not used with Python 3.7 and vice versa. For Python 3.7, the only way to install pycos is through pip and only those files can be used (and not, for example, files downloaded from github).
-
Version 1.62 released
-
The changes are already in github. You can get zip file or git master and use it. Pelease let me know if this works. If it does, may be time to release new version.
-
Version 4.10.5 released
-
pycos version 4.8.10 has been released. In this version Changed node_allocations parameter to dispycos' Computation to nodes, which works similar to nodes of dispy project: Only nodes that match given allocations are used to run computation. Sub-classing DispycosNodeAllocate (elements of nodes) now works even if remote dispycos scheduler is used.
-
Version 4.10.4 released
-
pycos version 4.8.9 has been released. In this version Fixed installation with Python 3.7. Version 4.8.8 had appropriate install files but installation itself was broken.
-
pycos version 4.8.8 has been released. In this release Added support for Python 3.7 Fixed dispycos so clients automatically wait for computation to finish and terminate (broken in 4.8.7 release).
-
Version 4.10.3 released
-
pycos version 4.8.7 has been released. In this version Added 'dispycos_cloud_computing.py' example to illustrate using dispycos for cloud computing. Fixed a few issues / added improvements in dispycos. Thanks to Sergey Kozlov for his contributions in this release.
-
pycos version 4.8.6 has been released. In this release Fixed dispycosnode for Python3 so servers don't close connection to node process (issue in 4.8.5 release). Fixed docker file for Python 3 to build images for Python 3 (instead of Python 2).
-
Version 4.10.2 released
-
pycos version 4.8.5 has been released. In this release Fixed dispycos module for Python 2.7 to import 'stat' module.
-
Version 4.10.1 released
-
pycos version 4.8.4 has been released. In this release If Python interpreter has 'suid' and 'sgid' bits set, switch to that user and group when running computation. This protects files used by dispycosnode (e.g., files that store PID of processes) from side effects of computation.
-
Version 4.10.0 released
-
pycos version 4.8.3 has been released. In this version Improved process termination in dispycosnode when servers close and when clean option is used to terminate processes in prior run. If psutil module is available (which is strongly recommended), additional checks are made to check that processes being terminated (from previous run) are actually dispycosnode servers.
-
It depends on your setup and how you would like to use remote nodes. If you would like to use SSH port forwarding, sshportfwpy can be used. This should work even if client is in private network (behind a router that is on public network). However, it is not viable if there are many nodes or if you want to use it many times. Alternate would be to use the setup Cloud Computing. This requires changing router to forward port 51347 to your client computer. Let me know which setup is suitable and if you...
-
pycos version 4.8.2 has been released. In this release Added support for *BSD systems (FreeBSD, NetBSD etc).
-
pycos version 4.8.2 has been released. In this release Added support for *BSD (FreeBSD, NetBSD etc).
-
I have made changes but don't have a way to test. If you are interested in testing, can you email me so I can send it to you test?
-
pycos version 4.8.1 has been released. In this release Monitors for a task receive GeneratorExit (with argument "terminated") message when that task is terminated. Fixed service start / stop / end times in dispycosnode.
-
pycos version 4.8.0 has been released. In this release DispycosNodeAllocate can now be used to add nodes to dispycos scheduler (in case node are not detected automatically in local network, or if nodes are in remote network). Subclassed DispycosNodeAllocate objects can also be used with remote scheduler. Remote tasks and channel references are protected from ID reuse clashes (i.e., interpretter may reuse an ID if object goes away so remote references may not refer to object originally sent). dispycosnode...
-
Version 4.9.1 released
-
pycos version 4.7.7 has been released. In this version Fixed host name resolution when netifaces is not available and resolving name with IPv6 may fail. Changed peer method's broadcast parameter to relay.
-
pycos version 4.7.6 has been released. In this version Fixed host name resolution with Windows when netifaces module is not available. Fixed dispycosnode to allocate only CPUs reserved by the scheduler (instead of all availbale even if reserved number is smaller). Any nodes given with DispycosNodeAllocate will be automatically located by the scheduler (so the client doesn't need to use peer method do discover). Added port parameter to DispycosNodeAllocate so clients can indicate which port is used...
-
pycos version 4.7.5 has been released. In this release Fixed exception (crash) when server and client programs are run on same computer (EADDRINUSE from errno should be imported in netpycos instead of pycos).
-
Version 4.9.0 released
-
pycos version 4.7.4 has been released. The changes since last release are: Fixed potential crash during shutdown while SSL sockets are used. Fixed potential issue with binding to link-local IPv6 address with OS X.
-
Version 4.8.9 released
-
Version 4.8.8 released
-
pycos version 4.7.3 has been released. In this version Fixed crash whenAsyncFile/AsyncPipe is used (broken in version 4.7.1). Updated pipe_csum.py and pipe_grep.py example files to use pycos (these were left behind when convertnig from asyncoro). Fixed UDP broadcasting with OS X. Fixed dispycosnode with Python 2 under Windows.
-
pycos version 4.7.2 has been released. In this version Changed default value of ipv4_udp_multicast in py3 branch (for Python 3) to be False as documented (this was incorrectly True in version 4.7.1). Changed multicast group address for IPv6 to a distinct form (from a generic simple form) so pycos doesn't interfere with any other devices that use generic multicast address.
-
pycos version 4.7.1 has been released. Changes since last release are: Fixed networking (netpycos module) under Windows (broken in 4.7.0) Added optional parameter ipv4_udp_multicast to use multicasting (instead of default broadcasting) for UDP with IPv4.
-
pycos version 4.7.0 has been released. Changes since last release are pycos now uses 9705 port by default (instead of 51350). dispycosnode program uses ports 9706 onwards by default. If using firewall, please update your settings to match these ports. By default tcp_port used is 9705, not a random port as done in earlier releases. finish method of pycos (either from pycos module or netpycos module) now cleans up all setup so after this method is done, another pycos scheduler can be created if necessary;...
-
It is difficult for me to follow your compute to understand what the problem is. Moreover, dispy version you are using is quite old. Can you try latest release (4.8.7) and post the issue with it in detail? As explained in documentation, you can use 'pip' to install dispy instead of 'pyinstaller' (I am not familiar with it). If you have issues with latest release with your compute function, try to simplify it, e.g., start with 'sample.py' in 'examples' and work up to figure out what feature causes...
-
pycos version 4.6.6 has been released. The changes since last release are: Fixed IPv6 network configuration when netifaces module is available. Added 'suspend' and 'resume' methods for computation to suspend / resume nodes or servers. Web interface supports these features as well. Fixed shutdown methods so after one instance of Pycos is shutdown, another instance can be started in the same program (if necessary). Now only one release file is available from sourceforge. This file can be directly used...
-
Version 4.8.7 released
-
Version 4.8.7 released
-
Version 4.6.8 released
-
Version 4.8.5 released
-
Version 4.8.5 released
-
As sourceforge.net has been unreliable the past few weeks, the documentation has been mirrored at github at https://pgiri.github.io/pycos; once it is possible to update webpages at sourceforge, this note will be added to documentation pages.
-
As sourceforge.net has been unreliable the past few weeks, the documentation has been mirrored at github at https://pgiri.github.io/dispy; once it is possible to update webpages at sourceforge, this note will be added to documentation pages.
-
What version of dispy are you using? If not the latest version, upgrade to it; asyncoro has been replaced with pycos.
-
One possible issue is dispynode on AWS may not be able to communicate with your client (e.g., if client is behind a router). In that case your router should forward port 51347 to your client. You also need to set ext_ip_addr=<your router IP> to dispy.JobCluster.
-
One possible issue is dispynode on AWS may not be able to communicate with your client (e.g., if client is behind a router). In that case your router should forward port 51347 to your client.
-
Version 4.8.4 released
-
ndiswrapper works with only XP drivers. It seems driver you are trying is newer WDF which is not supported by ndiswrapper. If you can find an XP driver, it may work.
-
pycos version 4.6.5 has been released. In this version Added monitor method to RTI (Remote Task Invocation). This can be used to set or reset monitor task for future tasks created with an RTI instance (instead of having to set monitor method for each task created). Fixed a race condition with closing IOCP sockets when pycos is terminating.
-
Yes, you can set node cpus to 0 to not schedule anymore jobs to that node. However, currently running jobs on that node will continue to execute.
-
pycos version 4.6.4 has been released. In this version Fixed send_file method of Pycos to use appropriate socket family for the remote peer. This was broken since 4.6.0 release when support for multiple socket families was added to netpycos (but send_file was not updated to use modified structures). Fixed name property (method) of Task and Channel. Since 4.6.0 release the name returned missed first character.
-
pycos version 4.6.3 has been released. In this version, Added zombie_period option to Computation. This represents maximum time in seconds allowed for a node to have no communication before scheduler deems a node to be dead. When a node is deemed zombie, scheduler informs status_task of Computation with abandoned status for tasks, servers and that node. zombie_period and pulse_interval parameters set in Computation do not influence setting of zombie_period option for dispycosnode program - dispycosnode...
-
pycos version 4.6.2 has been released. In this release dispycosnode has been fixed so server processes and node task communicate.
-
Version 4.8.3 released
-
Version 4.8.2 released
-
pycos version 4.6.1 has been released. This release changes licence to Apache 2.0 (from MIT). In this release, UDP server task has been fixed to handle multiple network interfaces having same broadcast address (in the case of multihome networking).
-
Version 4.8.1 released
-
pycos version 4.6.0 has been released. In this version support for multihome networking, i.e., host connected to multiple networks, has been added. To use this feature, node parameter to netpycos.Pycos can be given as list of host names or IP addresses.
-
Version 4.8.0 released
-
Version 4.7.7 released
-
asyncoro version 4.5.6 has been released. In this release, Channel's deliver method implementation has been changed to get reply asynchronously from remote server. Otherwise, if n parameter is bigger than 1 and channel doesn't have enough scubscribers yet, delivering message may block until enough subscribers register with the channel (which would mean other messages to that remote server can't be delivered). Now during the time reply fromdeliver is pending, any other messages to that remote server...
-
dispy 4.7.6 released
-
asyncoro version 4.5.5 has been released. In this version httpd has been fixed to support IPv6 addresses.
-
asyncoro version 4.5.5 has been released. In this version httpd has been fixed to support IPv6 addresses.
-
dispy 4.7.5 release
-
asyncoro version 4.5.4 has been released. In this release Fixed IPv6 for Windows and OS X. IPv6 with Python 2.7 under Windows needs package win_inet_pton (it is not required for Python 3.6+). For IPv6 under OS X, package netifaces is required. Even if not required with other platforms, netifaces is also strongly recommended. Fixed a potential deadlock in discoro module (for distributed concurrent communicating processes).
1 >
