From: Ben P. <be...@pa...> - 2006-01-18 19:42:09
|
This is not a strict Webware question, more of a Python and Linux question, but I'm hoping someone may have some good insight into a fix. I have a web app that relies on data from a 3rd party, fetched via XML over HTTPS. I'm using urllib2 to connect to the remote server. Unfortunately, the 3rd party is unstable, and when they have a problem, sometimes the appserver crashes with this error: Traceback (most recent call last): File "WebKit/ThreadedAppServer.py", line 630, in run File "WebKit/ThreadedAppServer.py", line 156, in mainloop File "/usr/local/lib/python2.4/socket.py", line 169, in accept error: (24, 'Too many open files') I think this is a symptom of too many open connections to the 3rd party, which makes sense since these server crashes (only happened twice so far) are the same time as a reported "outage" by the 3rd party. I already know about increasing the number of file descriptors for Linux, which I did after the first occurance of this, but as I expected that only delayed the problem. I need a way to recover from this somehow without having the server crash. Thanks for any advice! Ben |
From: Jose G. <jo...@cy...> - 2006-01-18 20:16:36
|
Ben Parker wrote: > This is not a strict Webware question, more of a Python and Linux > question, but I'm hoping someone may have some good insight into a fix. > > I have a web app that relies on data from a 3rd party, fetched via XML > over HTTPS. I'm using urllib2 to connect to the remote server. Are you keeping the old data between connections? or do you fetch the data every time? More importantly how often are actually connecting to this 3rd party server to get data? i have a similar application but I only update my cache of the data on daily. > > Unfortunately, the 3rd party is unstable, and when they have a problem, > sometimes the appserver crashes with this error: > > Traceback (most recent call last): > File "WebKit/ThreadedAppServer.py", line 630, in run > File "WebKit/ThreadedAppServer.py", line 156, in mainloop > File "/usr/local/lib/python2.4/socket.py", line 169, in accept > error: (24, 'Too many open files') > If you're keeping the old data then you can simply wrap your 3rd party call into a try -except statement and use the old data if your update fails. > I think this is a symptom of too many open connections to the 3rd party, > which makes sense since these server crashes (only happened twice so > far) are the same time as a reported "outage" by the 3rd party. > > I already know about increasing the number of file descriptors for > Linux, which I did after the first occurance of this, but as I > expected that only delayed the problem. I need a way to recover from > this somehow without having the server crash. > > Thanks for any advice! > Ben > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Webware-discuss mailing list > Web...@li... > https://lists.sourceforge.net/lists/listinfo/webware-discuss > > Jose |
From: Ben P. <be...@pa...> - 2006-01-18 20:41:49
|
Jose Galvez wrote on 01/18/2006 12:16 PM: >Ben Parker wrote: > > >>This is not a strict Webware question, more of a Python and Linux >>question, but I'm hoping someone may have some good insight into a fix. >> >>I have a web app that relies on data from a 3rd party, fetched via XML >>over HTTPS. I'm using urllib2 to connect to the remote server. >> >> >Are you keeping the old data between connections? or do you fetch the >data every time? More importantly how often are actually connecting to >this 3rd party server to get data? i have a similar application but I >only update my cache of the data on daily. > > I'm doing lots of caching wherever possible, but most of the data must be fetched in real-time. It is for hotel room availability, which changes constantly. >>Unfortunately, the 3rd party is unstable, and when they have a problem, >>sometimes the appserver crashes with this error: >> >>Traceback (most recent call last): >> File "WebKit/ThreadedAppServer.py", line 630, in run >> File "WebKit/ThreadedAppServer.py", line 156, in mainloop >> File "/usr/local/lib/python2.4/socket.py", line 169, in accept >>error: (24, 'Too many open files') >> >> >> >If you're keeping the old data then you can simply wrap your 3rd party >call into a try -except statement and use the old data if your update fails. > > I have that already. I think the problem is that the sockets are not timing out or throwing any error, so the except block never triggers. Otherwise I'm already handling a few differnt kinds of socket errors. I've been adding except clauses whenever I discover a new type of error: These are trapped for urllib2.urlopen calls: socket.sslerror IOError AttributeError (thrown from somewhere in urllib2.urlopen when it gets a malformed response) httplib.BadStatusLine These are trapped during the read of the url object returned by urllib2.urlopen: socket.error AssertionError (again thrown somewhere inside urllib2 Regards - Ben |
From: Ben P. <be...@pa...> - 2006-01-27 19:52:26
|
Ben Parker wrote on 01/18/2006 12:45 PM: >>> Unfortunately, the 3rd party is unstable, and when they have a problem, >>> sometimes the appserver crashes with this error: >>> >>> Traceback (most recent call last): >>> File "WebKit/ThreadedAppServer.py", line 630, in run >>> File "WebKit/ThreadedAppServer.py", line 156, in mainloop >>> File "/usr/local/lib/python2.4/socket.py", line 169, in accept >>> error: (24, 'Too many open files') >>> >>> >> >> If you're keeping the old data then you can simply wrap your 3rd party >> call into a try -except statement and use the old data if your update >> fails. >> >> > I have that already. I think the problem is that the sockets are not > timing out or throwing any error, so the except block never triggers. > I'm going to partially answer my own question here: Doh! There is no timeout in the Python 2.4 socket module by default. For some reason I had thought it was 2 mins. I am now calling socket.setdefaulttimeout in the __init__ of my context to set a default timeout of 2.5 mins, which causes these connections to time out instead of consuming all available files. However, I need to control the timeout value on a per-socket basis. I don't want to use socket.setdefaulttimeout because of the variety of requests this application needs to make. There are some that can tolerate relatively short timeouts (20 secs or so), but there are critical transactions which could take a couple minutes to return from the remote server. A quick check of the socket module reveals that since Python 2.3 one can set a timeout on a particular socket object. So now I need to find a way of doing that through urllib2 or httplib or work around it by rolling my own httplib. I'll post some more info if I get anything going. In the meantime, if anyone has an application which is setting per-socket timeouts for https transactions I would be very interested to hear how you are doing it. :) - Ben |