Re: Wedged threads (was RE: [Webware-discuss] help evaluating WebWar e)

[Tracy S. R. <tr...@re...>]

I'm using 0.8 and experiencing similar problems.  I know some of my 
problem comes from an Out of Memory error.  I'm running Webware on a 
system that has a hard limit on the amount of RAM each application 
uses.  If our instance of Webware goes over that limit AND we then have 
an error, the gc (garbage collection) module causes a segmentation 
fault (I think because it keeps trying to recursively throw an Out of 
Memory exception and clean up old stuff at the same time).

Also, I see what David sees below, too.  Every so often, webware stops 
responding to requests, though I can still access the monitoring port.  
(If Monitor.py is running, it doesn't notice a problem because it still 
gets an okay response from there.)  Right now, I have an external 
script that tries to access a web page and if it happens to take longer 
than 5 seconds, it assumes that the problem is there and restarts the 
server (also having to use the kill -9 hardkill).

Right now, I assume (but you know what happens when you assume) that 
there is something wrong with *my* code and have been investigating it 
piece by piece to try to find the problem (without any luck so far).  A 
tell-tale symptom of our system is that it slowly takes up more and 
more memory in direct correlation to site traffic.  So, I have looked 
for problems in any data I put into the session objects and any module 
and class variables that I use for caching data.  I have gone so far as 
to not use caching at all and to put only very minimal python 
structured data in the sessions.  (The session files in 
WebKit/Sessions/ are never more than 4K a piece).

My next area to tackle are the Cheetah templates I'm using.  My /tmp 
directory is mysteriously being filled up with 
__CheetahTemp_<yyyymmddt>.pyo files but I'm not using caching within my 
template files at all.  But, this problem I will take up with the 
Cheetah list...

--Tracy

My system:
	FreeBSD 4.4
	Apache 1.3.27
	Python 2.2.2
	Webware and mod_webkit 0.8
	Cheetah 0.9.14
	MySQL-python 0.9.2

On Monday, March 10, 2003, at 01:37 PM, Hancock, David (DHANCOCK) wrote:

> We're now encountering a "wedge" every couple of days or so on our test
> servers (RedHat Linux 6.2, Webware 0.7, Apache 1.3.x with mod_webkit) 
> and
> it's starting to worry us.  So if there's progress on the 
> wedge-elimination
> front, we'd love to hear about it.  And if we can help, let me know.
> Caveat: We're developing WITH Webware, but don't feel competent yet to 
> make
> substantive changes to the core code.
>
> The symptoms are that NO webkit threads seem to respond to browser 
> requests.
> Sometimes there's a core dump, which gets labeled with the pid, I'm 
> assuming
> (such as core.34172, where 34171, 34170, etc. are Webkit processes.  
> Loading
> the core in the debugger tells me that ThreadedAppServer encountered a
> segmentation fault (signal 11).  There's nothing labeled in the stack 
> trace,
> just memory addresses, so I can't tell what function was executing.  
> netstat
> reports that there's a server listening (we're using 8086 for WebKit), 
> and
> in fact, we can do 'telnet localhost 8086' and get connected.  (Of 
> course,
> we can't do more at that point because it's looking for request data
> marshalled into some structure we don't know how to recreate from the
> command line!)  But at least there's a listening socket.
>
> When this occurs, stopping and starting the ThreadedAppServer clears 
> the
> problem, but the processes need to be 'kill -9'ed before they'll die.
>
> I'd be grateful for any ideas on how to figure out where this problem 
> is
> coming from, how to fix it, etc.
>
> I wish I had better information from the core dump (which doesn't 
> always
> happen), so if we need to instrument Webware or even Python to get 
> better
> stack traces, we could try that.  My past experiences with turning on
> voluminous debug information, though, is that it slows things down 
> enough
> that timing-related bugs don't show up anymore.
>
> We MAY be able to install 0.8 on the test system, but my handlers 
> would like
> some statement from me BESIDES "I hope the upgrade will fix this 
> problem."
> They keep quoting the title of a recent business book: _Hope Is Not a
> Strategy_.
>
> Cheers!
> --
> David Hancock | dha...@ar... | 410-266-4384
>
>
> -----Original Message-----
> From: Geoffrey Talvola [mailto:gta...@na...]
> Sent: Thursday, March 06, 2003 9:41 AM
> To: 'Hancock, David (DHANCOCK)'; web...@li...
> Subject: RE: [Webware-discuss] help evaluating WebWare
>
>
> Hancock, David (DHANCOCK) [mailto:DHA...@ar...] wrote:
>> We have had some occurrences of threads seeming to wedge
>> (mentioned just
>> recently on the list), and we're trying to figure out why.
>
> I'd like to improve WebKit so that you can enable a "DebugThreads" 
> flag.  It
> would then detect when a thread is wedged (i.e. hasn't finished its
> transaction within some configurable timeout) and send a warning email.
> Perhaps it would also add an additional thread to the thread pool to 
> make up
> for the wedged thread.  Finally, it would be nice if the wedged thread
> didn't prevent the appserver from shutting down cleanly.
>
> Any other ideas?  I'll probably try to implement this within the next 
> couple
> days.
>
> - Geoff
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Webware-discuss mailing list
> Web...@li...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>
>
--
Tracy S. Ruggles
Chief Technology Officer, Axiomfire
tr...@ax... -- 512/461.6199