Version 0.3

2011-12-04
2012-10-06
  • Giridhar Pemmasani

    Version 0.3 has been released. The changes since 0.2 have been fixes since
    first 0.2 release - I have been updating 0.2 release with fixes, so you may
    not see many change since latest 0.2 release. We now also have svn, so going
    forward it will be easy to see updates/fixes.

    I still don't know if dispy works with Windows; if you try it, please confirm
    if it works or not.

    It would help further development if you can post feedback (e.g., if you found
    it useful, how you use it, any shortcomings in features etc.).

     
  • MattyJ

    MattyJ - 2011-12-06

    Thanks for creating what looks like a great tool. I have tried to run it on
    Windows, with Python 2.6, but had some errors. My code is below:


    import dispy
    
    def compute(n):
        import math, time
        f_ans = math.factorial(n)
        return f_ans
    
    cluster = dispy.JobCluster(compute, nodes=['172.22.0.251','172.22.0.196'], cleanup=True)
    jobs = []
    
    i = 1
    inputs_list = []
    while i <= 20000:
        job = cluster.submit(i)
        job.id = i
        jobs.append(job)
        i = i + 1
    
    cluster.wait()
    cluster.stats()
    

    I had two nodes which were running and subsequently had errors, as follows:

    2011-12-06 13:20:13,904 Listening at 172.22.0.196:51348
    Traceback (most recent call last):
    File "dispynode.py", line 700, in <module>
    node._serve()
    File "dispynode.py", line 315, in _serve
    func_proc.start()
    File "C:\Python26\ArcGIS10.0\lib\multiprocessing\process.py", line 104, in
    start
    self._popen = Popen(self)
    File "C:\Python26\ArcGIS10.0\lib\multiprocessing\forking.py", line 239, in
    init
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
    File "C:\Python26\ArcGIS10.0\lib\multiprocessing\forking.py", line 162, in
    dump
    ForkingPickler(file, protocol).dump(obj)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 224, in dump
    self.save(obj)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, rv)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 419, in save_reduce
    save(state)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 681, in _batch_setitems
    save(v)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 562, in save_tuple
    save(element)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj,
    rv)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 396, in save_reduce
    save(cls)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 748, in save_global
    (obj, module, name))
    PicklingError: Can't pickle <type 'code'="">: it's not found as builtin.code
    2011-12-06 13:30:22,838 None
    2011-12-06 13:30:22,838 Server terminated (possibly due to an error);
    restarting

    Traceback (most recent call last):
    File "dispynode.py", line 700, in <module>
    node._serve()
    File "dispynode.py", line 315, in _serve
    func_proc.start()
    File "C:\Python26\ArcGIS10.0\lib\multiprocessing\process.py", line 104, in
    start
    self._popen = Popen(self)
    File "C:\Python26\ArcGIS10.0\lib\multiprocessing\forking.py", line 239, in
    init
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
    File "C:\Python26\ArcGIS10.0\lib\multiprocessing\forking.py", line 162, in
    dump
    ForkingPickler(file, protocol).dump(obj)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 224, in dump
    self.save(obj)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, rv)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 419, in save_reduce
    save(state)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 681, in _batch_setitems
    save(v)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 562, in save_tuple
    save(element)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj,
    rv)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 396, in save_reduce
    save(cls)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 748, in save_global
    (obj, module, name))
    PicklingError: Can't pickle <type 'code'="">: it's not found as builtin.code
    2011-12-06 13:30:22,868 None
    2011-12-06 13:30:22,869 Server terminated (possibly due to an error);
    restarting

    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Python26\ArcGIS10.0\lib\multiprocessing\forking.py", line 342, in
    main
    self = load(from_parent)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 1370, in load
    return Unpickler(file).load()
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 858, in load
    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    dispatch(self)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 880, in load_eof
    File "C:\Python26\ArcGIS10.0\lib\multiprocessing\forking.py", line 342, in
    main
    raise EOFError
    EOFError
    self = load(from_parent)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 1370, in load
    return Unpickler(file).load()
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 858, in load
    dispatch(self)
    File "C:\Python26\ArcGIS10.0\lib\pickle.py", line 880, in load_eof
    raise EOFError
    EOFError


    Any help is appreciated. Should I be using a version of Python greater than
    2.6?

    Thanks

     
  • MattyJ

    MattyJ - 2011-12-06

    The same errors occur when using Python 2.7.2 on Windows.

     
  • Giridhar Pemmasani

    Looks to be easy to fix. Will follow up with email directly.

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks