First, thanks for all the effort you have put into pycam. I am using it to process stl files from both freecad and Google sketchup. It has performed flawlessly.
Until today, when I decided to try my hand at distributed processing.
My laptop is running the GUI, it is a Intel(R) Core(TM)2 Duo CPU P8700 running at 2.53GHz.
My remote machine is a QUAD CPU AMD Opteron™ Processor 848 running at 2.19GHz. This machine is running CentOS, and I have done the following:
Installed ActivePython-2.7 in my home directory. Modify my path so that Active Python is being used.
On both machines I am running pycam from svn trunk.
On the remote machine I am starting pycam with the following shell script:
I am able to use the laptop, disconnected from the rmemote machine without problem. I can connect, using the GUI, to the remote server.
However, when I attempt to generate a toolpath while connected the following error is thrown:
An unexpected exception occoured: please send the text below to the developers of PyCAM. Thanks a lot!
Traceback (most recent call last):
File "/home/shaworth/pycam/src/pycam/Gui/Project.py", line 3491, in generate_toolpath
self.model, toolpath_settings, callback=draw_callback)
File "/home/shaworth/pycam/src/pycam/Toolpath/Generator.py", line 60, in generate_toolpath_from_settings
backend, callback)
File "/home/shaworth/pycam/src/pycam/Toolpath/Generator.py", line 369, in generate_toolpath
maxz, dz, callback)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 249, in GenerateToolPath
draw_callback, progress_counter, num_of_triangles)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 258, in GenerateToolPathSlice
z, progress_counter=progress_counter)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 302, in get_potential_contour_lines
for result, ignore_triangle_id_list in results_iter:
File "/home/shaworth/pycam/src/pycam/Utils/threading.py", line 418, in run_in_parallel_remote
remote_cache.add(data_uuid, arg)
File "<string>", line 2, in add
File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
Traceback (most recent call last):
File "/home/shaworth/ActivePython-2.7/lib/python2.7/multiprocessing/managers.py", line 214, in serve_client
request = recv()
UnpicklingError: NEWOBJ class argument isn't a type object
I hope I have provided enough detail. If you need additional assistance or information contact me at shannon dot haworth@gmail.com (<- replace the dot and remove the spaces).
Thank you,
Shannon
An unexpected exception occoured: please send the text below to the developers of PyCAM. Thanks a lot!
Traceback (most recent call last):
File "/home/shaworth/pycam/src/pycam/Gui/Project.py", line 3491, in generate_toolpath
self.model, toolpath_settings, callback=draw_callback)
File "/home/shaworth/pycam/src/pycam/Toolpath/Generator.py", line 60, in generate_toolpath_from_settings
backend, callback)
File "/home/shaworth/pycam/src/pycam/Toolpath/Generator.py", line 369, in generate_toolpath
maxz, dz, callback)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 249, in GenerateToolPath
draw_callback, progress_counter, num_of_triangles)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 258, in GenerateToolPathSlice
z, progress_counter=progress_counter)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 302, in get_potential_contour_lines
for result, ignore_triangle_id_list in results_iter:
File "/home/shaworth/pycam/src/pycam/Utils/threading.py", line 418, in run_in_parallel_remote
remote_cache.add(data_uuid, arg)
File "<string>", line 2, in add
File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
Traceback (most recent call last):
File "/home/shaworth/ActivePython-2.7/lib/python2.7/multiprocessing/managers.py", line 214, in serve_client
request = recv()
UnpicklingError: NEWOBJ class argument isn't a type object
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
here in my setup I am running various versions of Python (v2.5, 2.6, 2.7) for distributed processing. This works flawlessly here.
Thus I would guess, that maybe ActivePython could somehow introduce a problem in your setup? Could you try to use Python instead? (I never used ActivePython - so I can't tell, if there is a difference)
Another approach could be to use the same version of ActivePython on both sides (currently you use 2.6 and 2.7, right?).
You did not mention it, but I guess, your laptop runs linux, or?
Are you using the same version of PyCAM on both sides?
Apart from that I don't have a clue, what could cause your problem. Theoretically pickling/unpickling should work without problems with different flavours and versions of Python …
I am curiously waiting for reports of your progress!
cheers,
Lars
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2011-01-22
Lars,
Thank you for your assistance.
The server which I would like to distribute work to is running CentOS. CentOS ships with python 2.4.3. Which obviously isn't going to work. This is the official "supported" version. Upgrading to an unsupported release is reported to wreck havoc. :(
Which is where ActivePython comes into play. It is a 100% stand-alone python install that can coexist with the "real" python.
Now, the server which I use (rather infrequently) to compile software has 32G of ram and 4 CPUs. It also has a rather large SCSI array, with lots and lots of empty, unpartitioned space.
Thanks to your post it becomes clear that the code works. I think I'll carve out a Ubuntu partition on the hard drive. It feels like the quickest way to get things running. Life is too short to spend on some problems. When I want to do CAM work I will just boot the Ubuntu partition.
Which has got me thinking… My one complaint with PyCAM is the slow performance while calculating tool-paths, contour paths especially. Bringing four additional CPUs to bear will improve things a lot. But imagine bringing, say, 64 CPUs to bear. Isn't cloud computing is all about on-demand processing? I wonder if it is affordable. Some of my jobs currently take 2-3 hours. Reducing that to 5-10 min would be very valuable to me.
Hmmmm. I'm going to have to look into this.
Shannon
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
yes, the current contour algorithm is quite slow. I guess there is still a lot of space for improvements, but this will take some time …
Regarding your server and the dual boot mode: I am quite sure that the server can run something like KVM, or? In case this is not yet supported by your version of Cent OS, then maybe running a different virtualization host system would be an option? Then the Cent OS system could run along in parallel with the PyCAM server …
Regarding the PyCAM processing cloud: that was also a thought of mine, when I implemented it. Obviusly this is an amazingly attractive idea :)
But sadly I did not find a good way for creating a safe Python sandbox. This would be necessary, since the current implementation completely trusts in the other nodes not injecting malicious code. Thus I guess, there are three options:
1) the nodes of the cloud run in a strict OS based sandbox (quite hard to do this right, I guess)
2) we find a restrictive Python sandbox implementation
3) we change the current communication/distribution implementation from the generic python "multiprocessing" mode to a custom one (this is a lot of complicated work)
Do you have any other ideas?
cheers,
Lars
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
just another note:
if KVM or any other virtualization is too heavy or complicated for your needs, then you could just create a small chroot environment (e.g. running debian or something similar). On debian based systems (debian/ubuntu/…) this can be easily done with the "debootstrap" tool. If you want to follow this road, then the first three paragraphs of the following howto could be interesting for you: http://www.escience.cam.ac.uk/~mcal00/xen-centos5.2-etch-howto.html
Lars
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
First, thanks for all the effort you have put into pycam. I am using it to process stl files from both freecad and Google sketchup. It has performed flawlessly.
Until today, when I decided to try my hand at distributed processing.
My laptop is running the GUI, it is a Intel(R) Core(TM)2 Duo CPU P8700 running at 2.53GHz.
My remote machine is a QUAD CPU AMD Opteron™ Processor 848 running at 2.19GHz. This machine is running CentOS, and I have done the following:
Installed ActivePython-2.7 in my home directory. Modify my path so that Active Python is being used.
On both machines I am running pycam from svn trunk.
On the remote machine I am starting pycam with the following shell script:
#!/bin/sh
_PYTHON=~/ActivePython-2.6/bin/python
$_PYTHON pycam -enable-server -start-server-only -server-auth-key=secret_key
I am able to use the laptop, disconnected from the rmemote machine without problem. I can connect, using the GUI, to the remote server.
However, when I attempt to generate a toolpath while connected the following error is thrown:
An unexpected exception occoured: please send the text below to the developers of PyCAM. Thanks a lot!
Traceback (most recent call last):
File "/home/shaworth/pycam/src/pycam/Gui/Project.py", line 3491, in generate_toolpath
self.model, toolpath_settings, callback=draw_callback)
File "/home/shaworth/pycam/src/pycam/Toolpath/Generator.py", line 60, in generate_toolpath_from_settings
backend, callback)
File "/home/shaworth/pycam/src/pycam/Toolpath/Generator.py", line 369, in generate_toolpath
maxz, dz, callback)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 249, in GenerateToolPath
draw_callback, progress_counter, num_of_triangles)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 258, in GenerateToolPathSlice
z, progress_counter=progress_counter)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 302, in get_potential_contour_lines
for result, ignore_triangle_id_list in results_iter:
File "/home/shaworth/pycam/src/pycam/Utils/threading.py", line 418, in run_in_parallel_remote
remote_cache.add(data_uuid, arg)
File "<string>", line 2, in add
File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
Traceback (most recent call last):
File "/home/shaworth/ActivePython-2.7/lib/python2.7/multiprocessing/managers.py", line 214, in serve_client
request = recv()
UnpicklingError: NEWOBJ class argument isn't a type object
I hope I have provided enough detail. If you need additional assistance or information contact me at shannon dot haworth@gmail.com (<- replace the dot and remove the spaces).
Thank you,
Shannon
An unexpected exception occoured: please send the text below to the developers of PyCAM. Thanks a lot!
Traceback (most recent call last):
File "/home/shaworth/pycam/src/pycam/Gui/Project.py", line 3491, in generate_toolpath
self.model, toolpath_settings, callback=draw_callback)
File "/home/shaworth/pycam/src/pycam/Toolpath/Generator.py", line 60, in generate_toolpath_from_settings
backend, callback)
File "/home/shaworth/pycam/src/pycam/Toolpath/Generator.py", line 369, in generate_toolpath
maxz, dz, callback)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 249, in GenerateToolPath
draw_callback, progress_counter, num_of_triangles)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 258, in GenerateToolPathSlice
z, progress_counter=progress_counter)
File "/home/shaworth/pycam/src/pycam/PathGenerators/ContourFollow.py", line 302, in get_potential_contour_lines
for result, ignore_triangle_id_list in results_iter:
File "/home/shaworth/pycam/src/pycam/Utils/threading.py", line 418, in run_in_parallel_remote
remote_cache.add(data_uuid, arg)
File "<string>", line 2, in add
File "/usr/lib/python2.6/multiprocessing/managers.py", line 740, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
Traceback (most recent call last):
File "/home/shaworth/ActivePython-2.7/lib/python2.7/multiprocessing/managers.py", line 214, in serve_client
request = recv()
UnpicklingError: NEWOBJ class argument isn't a type object
Hi Shannon,
here in my setup I am running various versions of Python (v2.5, 2.6, 2.7) for distributed processing. This works flawlessly here.
Thus I would guess, that maybe ActivePython could somehow introduce a problem in your setup? Could you try to use Python instead? (I never used ActivePython - so I can't tell, if there is a difference)
Another approach could be to use the same version of ActivePython on both sides (currently you use 2.6 and 2.7, right?).
You did not mention it, but I guess, your laptop runs linux, or?
Are you using the same version of PyCAM on both sides?
Apart from that I don't have a clue, what could cause your problem. Theoretically pickling/unpickling should work without problems with different flavours and versions of Python …
I am curiously waiting for reports of your progress!
cheers,
Lars
Lars,
Thank you for your assistance.
The server which I would like to distribute work to is running CentOS. CentOS ships with python 2.4.3. Which obviously isn't going to work. This is the official "supported" version. Upgrading to an unsupported release is reported to wreck havoc. :(
Which is where ActivePython comes into play. It is a 100% stand-alone python install that can coexist with the "real" python.
Now, the server which I use (rather infrequently) to compile software has 32G of ram and 4 CPUs. It also has a rather large SCSI array, with lots and lots of empty, unpartitioned space.
Thanks to your post it becomes clear that the code works. I think I'll carve out a Ubuntu partition on the hard drive. It feels like the quickest way to get things running. Life is too short to spend on some problems. When I want to do CAM work I will just boot the Ubuntu partition.
Which has got me thinking… My one complaint with PyCAM is the slow performance while calculating tool-paths, contour paths especially. Bringing four additional CPUs to bear will improve things a lot. But imagine bringing, say, 64 CPUs to bear. Isn't cloud computing is all about on-demand processing? I wonder if it is affordable. Some of my jobs currently take 2-3 hours. Reducing that to 5-10 min would be very valuable to me.
Hmmmm. I'm going to have to look into this.
Shannon
Hi Shannon,
yes, the current contour algorithm is quite slow. I guess there is still a lot of space for improvements, but this will take some time …
Regarding your server and the dual boot mode: I am quite sure that the server can run something like KVM, or? In case this is not yet supported by your version of Cent OS, then maybe running a different virtualization host system would be an option? Then the Cent OS system could run along in parallel with the PyCAM server …
Regarding the PyCAM processing cloud: that was also a thought of mine, when I implemented it. Obviusly this is an amazingly attractive idea :)
But sadly I did not find a good way for creating a safe Python sandbox. This would be necessary, since the current implementation completely trusts in the other nodes not injecting malicious code. Thus I guess, there are three options:
1) the nodes of the cloud run in a strict OS based sandbox (quite hard to do this right, I guess)
2) we find a restrictive Python sandbox implementation
3) we change the current communication/distribution implementation from the generic python "multiprocessing" mode to a custom one (this is a lot of complicated work)
Do you have any other ideas?
cheers,
Lars
just another note:
if KVM or any other virtualization is too heavy or complicated for your needs, then you could just create a small chroot environment (e.g. running debian or something similar). On debian based systems (debian/ubuntu/…) this can be easily done with the "debootstrap" tool. If you want to follow this road, then the first three paragraphs of the following howto could be interesting for you:
http://www.escience.cam.ac.uk/~mcal00/xen-centos5.2-etch-howto.html
Lars