Giridhar Pemmasani

I have emailed you an implementation of this feature.

I will look into this and find a generic way to add resource requirements. I will post patch to try once ready.

The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...

The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...

The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...

The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...

The second type of resource check is also possible with setup function, which runs on each node before any jobs are submited. For example, this function can check GPUs / packages available etc. In fact, using setup to load all modules is better as it not only checks that packages are available on a node but avoids loading modules in computations (saves memory and time); e.g., def node_setup(data_file): global tensorflow, data # global variables are evailable to jobs import tensorflow with open(data_file,...

This is already possible with NodeAllocate feature. For example if you would like to only use nodes that have 32G RAM and 16 cores, you can filter nodes with: if __name__ == '__main__': ... class AllocateNode(dispy.NodeAllocate): def allocate(self, cluster, ip_addr, name, cpus, avail_info=None, platform='*'): if cpus < 16: return 0 # don't use this node if avail_info.memory < (32*1024*1024*1024): return 0 # don't use this node return (cpus - 2) # reserve 2 cores for other uses cluster = dispy.JobCluster(compute,...