Menu
▾
▴
Re: [VuFind-General] Vufind crashing
|
From: Andrew N. <as...@gm...> - 2009-10-02 14:52:47
|
I'm no java performance expert - but the fact that you have 16GB ram on your server and your heap space settings set to 1024M shows that you are under utilizing your hardware. I'd crank up your xmx and xms settings so to take advantage of your RAM. Andrew On 10/2/09, Michael McFarlane <m.d...@ls...> wrote: > Hi Mark, > > Thanks for these. This is exactly what I need to help troubleshoot the > issue. To be honest we haven't done much tuning as there was no need to. > There is now!!! > > Answers to questions below: > > > Mark Triggs wrote: >> Hi Mike, >> >> Others might have some suggestions, but I think some more information on >> how your system is configured might be useful here. If I was going to >> guess I'd suppose that your problems are either occurring in Apache >> (hitting your MaxClients threshold), Java/Jetty (either running out of >> memory or hitting your MaxThreads threshold), or in your database, but >> it's hard to be sure without more information. >> >> Since I don't have any good ideas I thought I'd send a whole bunch of >> questions :o): >> >> * What command line arguments (JAVA_OPTS) are you running your java >> process with? Are there any OutOfMemoryErrors in the logs? Adding >> a few extra options like: >> >> -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails >> -XX:+PrintTenuringDistribution >> > No OutOfmemoryErrors in logs. > > The options we use JAVA_OPTIONS="-server -Xms1024m -Xmx1024m > -XX:+UseParallelGC -XX:NewRatio=5" > > I tried to use the one on the Vufind website (JAVA_OPTIONS="-server > -Xmx3800 -Xms3800 -XX:+UseParallelGC -XX:+AggressiveOpts > -XX:NewRatio=5") but java failed on test server so haven't tested on > production server. > > It would be good to get an ideal for the server we use. > >> might yield some information that helps troubleshoot memory issues >> in the JVM. >> >> * What's your MaxClients setting in your jetty.xml? >> > Don't have maxClients. I hope I'm not missing anything! > Using default: <Set name="maxThreads">200</Set> >> * Do the Apache logs mention anything about reaching MaxClients? >> What's your setting in httpd.conf for that value? >> > Nothing in logs about reaching MaxClients. > > ServerLimit 256 > MaxClients 256 >> * Does your Voyager's Oracle alert log say anything about reaching >> maximum cursors or being unable to allocate temporary table space? >> We've seen both at various times. >> > There is no indication of these in our logs, but will check this when > crashes again. Although, our webvoyage has been up and running during > this time. > >> * While the problems are occurring, does hitting a static page on >> Apache work? Something like >> https://catalogue.lse.ac.uk/idonotexist.txt should come back quickly >> > Will test this next time it happens. Last time I was at home and just > needed to get vufind started! >> * While the problems are occurring can you search against Solr >> directly? http://your.solr.host:8086/solr/select?q=fish should >> still return results if Solr is working >> > Again will check next time. >> * Are your Solr statistics generally healthy? Does the "Cache" >> section of http://your.solr.host:8086/solr/admin/stats.jsp show a >> reasonable hit rate against your caches? Particularly the >> filterCache... >> > Umm. I get: > > ** > lookups : 0 > hits : 0 > hitratio : 0.00 > inserts : 0 > evictions : 0 > size : 0 > warmupTime : 0 > cumulative_lookups : 0 > cumulative_hits : 0 > cumulative_hitratio : 0.00 > cumulative_inserts : 0 > cumulative_evictions : 0 > > > >> * Does the system show anything unusual in `dmesg' output? Do >> top/vmstat show anything unusual while the problems are happening? >> > Nothing odd in dmesg. > Top did not show anything obvious. >> Sorry for all the questions, but hopefully they'll give people a >> starting point to make suggestions. >> > Mark, I really appreciate you taking time to reply. It's given me > something to look at and will definitely help. >> Cheers, >> >> Mark >> >> >> "M.D...@ls..." <M.D...@ls...> writes: >> >> >>> Hi All, >>> >>> I hope somebody would be able to help on this. We went live with our >>> vufind instance (https://catalogue.lse.ac.uk) September 21st and all was >>> well until school term started and hits rocketed. Since then vufind has >>> crashed 3 times (do a search and you get the PHP error - a vufind >>> restart fixes it, but we are not aware of this until somebody notifies >>> us (Nagios does not report issues - probably need to add a custom check >>> of vufind, unless somebody has already done so!) >>> >>> Checking the logs does not throw much light on the system, there is no >>> obvious pattern. In the web logs there are some 304s and 400s, but there >>> are also some 200s and the same for the jetty logs - 400 errors, but 200 >>> successes as well. >>> >>> We are running on a pretty powerful machine (PowerEdge 1950, 2xQuad Core >>> Xeon 2.5GHz, 16GB RAM) and performance is pretty good normally. We have >>> had reports of a slowing down in response times, but no hard evidence. >>> >>> I'll need to look at the tuning options fully now (no need before), but >>> is there anybody in the community have any quick fixes I can put in >>> place (whether it's java, PHP, apache etc..) to ensure stable running? >>> >>> We are still running WebVoyage and do point users to that if they find >>> an error, but we really want vufind to be the catalogue of choice. >>> >>> We are running r1194 on RHEL5. >>> >>> Thanks Mike >>> >> >> > > > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.lse.ac.uk/collections/secretariat/legal/disclaimer.htm > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry® Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9-12, 2009. Register now! > http://p.sf.net/sfu/devconf > _______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general > -- Sent from my mobile device |