Re: [Pyobjc-dev] parallel processing

[Daniel A. <an...@cc...>]

On Mar 28, 2009, at 4:25p, Ronald Oussoren wrote:
> As you noticed NSThreads are bound by the GIL, just as regular  
> Python threads (as long as you try to run Python code obviously). I  
> haven't tried using the multiprocessing library yet, I'd consider  
> hanging processes a bug in either PyObjC or multiprocessing until  
> proven otherwise.    I know there are issues when using the fork  
> system call without exec in a process using Apple frameworks, that's  
> a limitation in those frameworks and not caused by Python/PyObjC.

My guess is that it's a PyObjc bug; in multiprocessing I added a bunch  
of comments, and here's the hanging point:

     def run(self):
         '''
         Method to be run in sub-process; can be overridden in sub-class
         '''
         self.debug('in run')   # <--- THIS PRINTS
         self.debug('about to run target %s' % self._target)  # <---  
THIS DOESN'T
         if self._target:
             self._target(*self._args, **self._kwargs)

So apparently trying to access the _target function (which is just the  
PyObjc selector; when it runs properly it prints thing such as "about  
to run target <selector findGesture: of <EGL: mainEGL>>") is causing  
the freeze.

> One option you could try is to spawn of a background program that  
> does the actual calculation and have that use multiprocessing to  
> make use of multiple processes.

That's what I've been working on all afternoon. Too bad, because my  
prior solution was semi-elegant... but at least this way I'm learning  
all about posix_ipc and mmaps. I can read in 19MB of pickled data in . 
12s that way, so it shouldn't impact my life too badly.

> I have a note for the archives: don't use pyprocessing, it's basicly  
> dead at the moment (at least according to Jesse Noller, who wrote  
> multiprocessing).

I thought R Oudkerk wrote py/multiprocessing? But yes, it appears that  
multiprocessing is included in 2.6 and 3.0, and is available as a  
backport to 2.4/5 (http://code.google.com/p/python-multiprocessing/).

> W.r.t. hanging processes: can you reproduce that in a standalone  
> program?  I'm at Pycon at the moment and might be able to get access  
> to Jesse Noller to help research the issue.

Unlikely that I could before you leave; this is happening in my  
massive thesis project, and trying to pull out the parts that break  
would be a pretty heavy undertaking, I'm afraid.

Thanks,


dan