We have seen some similar behaviour from disseminators during performance testing.However our test scenario was slightly different than yours.We used Apache's benchmarking tool "ab" to send concurrent requests to some of our disseminators.And what we found was, whenever it failed to respond to all the requests, the disseminator froze i.e. all the objects that were linked to that disseminator failed to respond even though they were not part of url sent to benchmarking tool.
After this happens the only way to bring everything back to normal state is to re-start tomcat.
None of us in the team is Java expert,so we could not really dig too deep into the codes of any of the servlets . However we did suspect some form of resource leaking going on which was preventing any new request to be processed.
We haven't performed any more tests recently as we felt we on our own can't go very far.But I am sure we can recreate the error, if you are interested in comparing the logs to see if we are experiencing same problem.
Irish Virtual Research Library and Archive.
From: firstname.lastname@example.org > To: email@example.com > Date: Mon, 3 Dec 2007 17:19:13 -0500 > CC: firstname.lastname@example.org > Subject: [Fedora-commons-users] Fedora crashing > > Hi all, > > We've been experiencing an issue with Fedora 2.2 where it crashes under > moderate load. Our systems administrator, who has been investigating the > issue, asked me to post the following to the list on his behalf. Any > information/feedback will be appreciated. > > -Nikolai > > ----- Original Message ----- > From: "Matthew McVey" <email@example.com> > To: "Anoop Kumar" <Anoop.Kumar@tufts.edu>; "Nikolai J. Schwertner" > <Nikolai.Schwertner@tufts.edu> > Sent: Monday, December 03, 2007 4:38 PM > Subject: my fedora bug writeup (with attachments) > > > > Here is the problem with Fedora we have noticed. We are running Fedora > > 2.2 on top of the bundled Tomcat 5.0.28 and using MySQL for the Fedora > > database. We believe Fedora is leaking threads when requests to this > > disemminator are abandoned or resubmitted prior to request completion, > > and that it eventually fills up maxThreads and stops processing requests > > altogether (though the Tomcat process never dies). > > > > We have a disseminator that chunks XML text. Calls to this disseminator > > can take anywhere from a second or so to as much as 20 or 30 seconds > > depending on the complexity of the text being chunked. > > > > We fire off the disseminator directly in a web browser as follows: > > http://[our_fedora_server]:8080/fedora/get/tufts:UA069.001.DO.MS008/bdef:TuftsText/getChunk?chunkID=MS008.001 > > > > This request process normally takes less than a second or so to complete. > > > > Now if we hit the reload button in the web browser say 15 times is a > > row...we have noticed the following. > > > > 1. Fedora hang up on attempting to disseminate this object, seemingly > > forever (until some timeout perhaps) > > > > 2. any requests to this disseminator, even for different objects, will > > fail. > > > > Tailing the Fedora server log while running this test, we see the > > following errors logged. (see attached "fedora-log-error.txt") > > > > The following is logged in Fedora's catalina.out file for each refresh > > request. (see attached "catalina-error.txt") > > > > Under these conditions and before maxthreads is full, Fedora will still > > run other types of disseminations. Restarting all backend services > > involved in the dissemination does not seem to "flush" Fedora of the > > hung connections for this particular disseminator. > > > > If we restart Fedora prior to maxthreads being full, we see a message as > > follows in catalina.out when shutting down Fedora. (see attached > > "fedora-unload-error.txt"). When Fedora comes back up, it will once > > again complete the Fedora dissemination...until we hammer it again. And > > it does not take much hammering, merely 10 or 15 rapid browser reload > > requests interrupting each previous request (like and end-user hitting > > reload when they get impatient, just without much time in between reload > > clicks). > > > > We believe that what may be happening is that, over time, we accumulate > > such requests in the real world and eventually all of Fedora's threads > > are consumed. We have not tried pushing it to failure in a test > > environment yet, but feel confident we could do so using the test > > described. > > > > Has anyone experienced anything similar? Might there be a bug in the > > AccessServlet or in Tomcat 5.0.28 (either its servlet engine or its HTTP > > connector) that cannot gracefully hang-up threads when repeat requests > > for the same resource come in mid-stream? > > > > -- > > Matt > > > > > > > > > > -- > > Matthew McVey > > Learning Technologies Developer > > University Information Technology (UIT) > > Tufts University > > 16 Dearborn Road > > Somerville, MA 02144 > > > > Phone: 617.627.4395 > > Fax: 617.627.3667 > > http://uit.tufts.edu/at/ > > > > > > > > -- > > Matthew McVey > > Learning Technologies Developer > > University Information Technology (UIT) > > Tufts University > > 16 Dearborn Road > > Somerville, MA 02144 > > > > Phone: 617.627.4395 > > Fax: 617.627.3667 > > http://uit.tufts.edu/at/
Get the next generation of Windows Live services now! Click here!