pyobjc-dev

[Pyobjc-dev] Cocoa, httplib and threads?

[Greg E. <gre...@ca...>]

I'm trying to use httplib to send an http request in
a background thread. It works fine on its own, but when
I do it inside a Cocoa application, the thread hangs
as soon as it tries to send the request.

It doesn't seem to be a GIL problem, because I can do
other things in the thread leading up to that point.
It's just sending the request that seems to block.

Is there anything I should be aware of when using
the threading module in conjunction with pyobjc?

-- 
Greg

Re: [Pyobjc-dev] Cocoa, httplib and threads?

[Diez B. R. <de...@we...>]

On Jul 15, 2011, at 3:43 AM, Greg Ewing wrote:

> I'm trying to use httplib to send an http request in
> a background thread. It works fine on its own, but when
> I do it inside a Cocoa application, the thread hangs
> as soon as it tries to send the request.
> 
> It doesn't seem to be a GIL problem, because I can do
> other things in the thread leading up to that point.
> It's just sending the request that seems to block.
> 
> Is there anything I should be aware of when using
> the threading module in conjunction with pyobjc?

I've used threads + Cocoa without a hitch, so without more information I can only assume it's something you do. 

Diez

Re: [Pyobjc-dev] Cocoa, httplib and threads?

[Ronald O. <ron...@ma...>]

On 15 Jul, 2011, at 3:43, Greg Ewing wrote:

> I'm trying to use httplib to send an http request in
> a background thread. It works fine on its own, but when
> I do it inside a Cocoa application, the thread hangs
> as soon as it tries to send the request.
> 
> It doesn't seem to be a GIL problem, because I can do
> other things in the thread leading up to that point.
> It's just sending the request that seems to block.
> 
> Is there anything I should be aware of when using
> the threading module in conjunction with pyobjc?

This should work.  One thing you could try is redefining urllib. proxy_bypass_macosx_sysconf and urllib.getproxies_macosx_sysconf

def getproxies_macosx_sysconf():
   return {}

def proxy_bypass_macosx_sysconf(host):
   return True

urllib. getproxies_macosx_sysconf = getproxies_macosx_sysconf
urllib.proxy_bypass_macosx_sysconf = proxy_bypass_macosx_sysconf

This ensures that urllib doesn't use the OSX specific code for getting the HTTP proxy configuration.   That code should work just fine in a Cocoa program, but it does call into Apple's frameworks and hence might cause problems in some way.

Ronald
> -- 
> Greg
> 
> ------------------------------------------------------------------------------
> AppSumo Presents a FREE Video for the SourceForge Community by Eric 
> Ries, the creator of the Lean Startup Methodology on "Lean Startup 
> Secrets Revealed." This video shows you how to validate your ideas, 
> optimize your ideas and identify your business strategy.
> http://p.sf.net/sfu/appsumosfdev2dev
> _______________________________________________
> Pyobjc-dev mailing list
> Pyo...@li...
> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev

Re: [Pyobjc-dev] Cocoa, httplib and threads?

[Jonathan S. <sa...@gm...>]

You might also consider adding a run loop to each thread you fork.  I'm not
sure how things are handled by pyobjc (eg if it manages run loops in threads
for you, I would tend to doubt it), but many networking API in Cocoa require
a run loop or Bad Things Happen (TM), where "Bad Things" are usually silent
failures or stuck threads.

http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/Multithreading/RunLoopManagement/RunLoopManagement.html

Speaking of Cocoa.  I don't know if you're doing anything that really
requires threaded programming or python networking.  If not, consider using
the asynchronous Cocoa networking APIs.  They'll keep you out of threaded
programming land.  Look at NSHttp* and NSURL* classes.  YMMV, etc.

Cheers,
Jonathan Saggau
<http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/Multithreading/RunLoopManagement/RunLoopManagement.html>

On Fri, Jul 15, 2011 at 4:42 AM, Ronald Oussoren <ron...@ma...>wrote:

>
> On 15 Jul, 2011, at 3:43, Greg Ewing wrote:
>
> > I'm trying to use httplib to send an http request in
> > a background thread. It works fine on its own, but when
> > I do it inside a Cocoa application, the thread hangs
> > as soon as it tries to send the request.
> >
> > It doesn't seem to be a GIL problem, because I can do
> > other things in the thread leading up to that point.
> > It's just sending the request that seems to block.
> >
> > Is there anything I should be aware of when using
> > the threading module in conjunction with pyobjc?
>
> This should work.  One thing you could try is redefining urllib.
> proxy_bypass_macosx_sysconf and urllib.getproxies_macosx_sysconf
>
> def getproxies_macosx_sysconf():
>   return {}
>
> def proxy_bypass_macosx_sysconf(host):
>   return True
>
> urllib. getproxies_macosx_sysconf = getproxies_macosx_sysconf
> urllib.proxy_bypass_macosx_sysconf = proxy_bypass_macosx_sysconf
>
> This ensures that urllib doesn't use the OSX specific code for getting the
> HTTP proxy configuration.   That code should work just fine in a Cocoa
> program, but it does call into Apple's frameworks and hence might cause
> problems in some way.
>
> Ronald
> > --
> > Greg
> >
> >
> ------------------------------------------------------------------------------
> > AppSumo Presents a FREE Video for the SourceForge Community by Eric
> > Ries, the creator of the Lean Startup Methodology on "Lean Startup
> > Secrets Revealed." This video shows you how to validate your ideas,
> > optimize your ideas and identify your business strategy.
> > http://p.sf.net/sfu/appsumosfdev2dev
> > _______________________________________________
> > Pyobjc-dev mailing list
> > Pyo...@li...
> > https://lists.sourceforge.net/lists/listinfo/pyobjc-dev
>
>
>
> ------------------------------------------------------------------------------
> AppSumo Presents a FREE Video for the SourceForge Community by Eric
> Ries, the creator of the Lean Startup Methodology on "Lean Startup
> Secrets Revealed." This video shows you how to validate your ideas,
> optimize your ideas and identify your business strategy.
> http://p.sf.net/sfu/appsumosfdev2dev
> _______________________________________________
> Pyobjc-dev mailing list
> Pyo...@li...
> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev
>
>


-- 
Jonathan Saggau
jonathansaggau.com

This amusement engaged me so much that [friends] were obliged to force me
from it; and thus it is with every inclination I give into, it continues to
augment, till at length it becomes so powerful, that I lose sight of
everything except the favorite amusement. - Rousseau

Re: [Pyobjc-dev] Cocoa, httplib and threads?

[Greg E. <gre...@ca...>]

This is getting extremely weird. It turns out that the
thread is hanging on the following line in httplib.py:

       host_enc = self.host.encode("ascii")

where the string being encoded in my test case is
'127.0.0.1'.

I can't for the life of me imagine what the ascii codec
could be doing to cause a thread to block.

Can anyone with more knowledge of the codec system
shed any light?

Some more data points:

* It doesn't happen when Cocoa is used in a simple way
   (just create an NSApplication and call its run() method).

* It also doesn't happen if I just create a PyGUI Application
   object and call its run() method.

So it's something about my big complicated PyGUI app... :-(

> Speaking of Cocoa. I don't know if you're doing anything that really requires
> threaded programming or python networking. If not, consider using the
> asynchronous Cocoa networking APIs.

This needs to be cross-platform, so I can't rely on anything
Cocoa-specific.

-- 
Greg

Re: [Pyobjc-dev] Cocoa, httplib and threads?

[Ronald O. <ron...@ma...>]

On 16 Jul, 2011, at 4:46, Greg Ewing wrote:

> This is getting extremely weird. It turns out that the
> thread is hanging on the following line in httplib.py:
> 
>       host_enc = self.host.encode("ascii")
> 
> where the string being encoded in my test case is
> '127.0.0.1'.
> 
> I can't for the life of me imagine what the ascii codec
> could be doing to cause a thread to block.
> 
> Can anyone with more knowledge of the codec system
> shed any light?

Not really. Have you tried getting a C stacktrace using gdb?  The only obvious reason for blocking is the GIL: if for some reason the main thread doesn't release the GIL at some point you'll get a hang in the secondary threads. That's pretty unlikely though given that the secondary thread does some work before it hangs.

Debugging would be easier if you had a smallish sample program (a dependency on PyGUI would be fine)

Ronald

[Pyobjc-dev] Cocoa, httplib and threads? - Solved

[Greg E. <gre...@ca...>]

I found out what was causing this. Turns out it was blocked
importing encodings.ascii, trying to acquire the import lock.

The reason it couldn't acquire the import lock is that the
main module of my application was a stub that imported the
real main module, which in turn started the event loop
directly from the module code.

So the whole app was running inside an import statement,
holding on to the import lock. Once I fixed that, everything
was fine.

-- 
Greg

Re: [Pyobjc-dev] Cocoa, httplib and threads? - Solved

[Diez B. R. <de...@we...>]

On Jul 22, 2011, at 9:13 AM, Greg Ewing wrote:

> I found out what was causing this. Turns out it was blocked
> importing encodings.ascii, trying to acquire the import lock.
> 
> The reason it couldn't acquire the import lock is that the
> main module of my application was a stub that imported the
> real main module, which in turn started the event loop
> directly from the module code.
> 
> So the whole app was running inside an import statement,
> holding on to the import lock. Once I fixed that, everything
> was fine.

Nice catch... 

Diez

Re: [Pyobjc-dev] Cocoa, httplib and threads? - Solved

[Ronald O. <ron...@ma...>]

On 22 Jul, 2011, at 9:13, Greg Ewing wrote:

> I found out what was causing this. Turns out it was blocked
> importing encodings.ascii, trying to acquire the import lock.
> 
> The reason it couldn't acquire the import lock is that the
> main module of my application was a stub that imported the
> real main module, which in turn started the event loop
> directly from the module code.
> 
> So the whole app was running inside an import statement,
> holding on to the import lock. Once I fixed that, everything
> was fine.

Ouch. That can be a hard one to debug.

Thank you for reporting back on the cause of your problem,

Ronald
> 
> -- 
> Greg
> 
> 
> ------------------------------------------------------------------------------
> 10 Tips for Better Web Security
> Learn 10 ways to better secure your business today. Topics covered include:
> Web security, SSL, hacker attacks & Denial of Service (DoS), private keys,
> security Microsoft Exchange, secure Instant Messaging, and much more.
> http://www.accelacomm.com/jaw/sfnl/114/51426210/
> _______________________________________________
> Pyobjc-dev mailing list
> Pyo...@li...
> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev