Just to add to what Benjamin has said. If you haven't read Demian's post on Java tuning, it is well worth a look: http://blog.library.villanova.edu/libtech/2011/03/31/java-tuning-made-easier/. The post covers how to output garbage collection logs which can then be analysed using gcviewer. This allows you to visualise what is happening. I also installed visualjvm (http://visualvm.java.net/) on the live server which allows realtime monitoring of garbage collection via its GC plugin.

We are on Linux but recently had similar garbage collection issues where full garbage collections started happening about every 2-3 hours and taking over 20-30seconds each until ultimately Solr would hang after about 17 - 24 hours since the last restart because all it was doing was GC. I added  -XX:+UseParallelOldGC  to  our JAVA_OPTIONS to use parallel collector on the "old" segment of the allocated jvm heap. Our full JAVA_OPTIONS including log output is:

JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+AggressiveOpts -XX:NewRatio=5 -Xloggc:/usr/local/vufind/solr/jetty/logs/gc-`/bin/date +%F-%H-%M`.log"

Obviously you should leave enough RAM free for the OS (i.e. if you only have 8GB on the system, don't give it all to Solr!). Since making this change, I haven't had to restart Solr in months. Looking at the logs, it still does a full GC about every 36 hours but it only takes about 5-6 seconds and collects much more of the "old" heap. 

I think our issue may have been exacerbated by over-zealous spiders, so at the same time I added a robots.txt file to limit the scope of their crawling (http://catalogue.nli.ie/robots.txt) to only include the /Record pages output by our sitemap. Looking at Google webmaster tools, the average pages crawled per day dropped from 250K to 15K while 95%+ of the sitemap pages are still being indexed.

Longer term you might want to look at:

Hope this helps. I'm copying this to vufind-tech as you might get more suggestions. There have been previous posts about this issue too, so be sure to search the Vufind mailing lists.


On 27 March 2013 17:17, Mosior, Benjamin <BEMosior@ship.edu> wrote:

I haven't looked much at the windows side of things (so everyone else, please feel free to step in and correct me), but it looks like you need to find the right place to set the JAVA_OPTIONS environment variable.

It looks like you have two options:

Set JAVA_OPTIONS environment variable at the system level
Set JAVA_OPTIONS in the run_vufind.bat script (probably not best practice, the above method is likely preferable)
  • Look for the first line that has: set JAVA_OPTIONS=-server...
In either case, you can then follow the performance documentation here: http://vufind.org/wiki/performance

Hope that helps.

Benjamin Mosior

From: Tomlin, Camille [TomlinC@freelibrary.org]
Sent: Wednesday, March 27, 2013 10:50
To: vufind-general@lists.sourceforge.net
Subject: [VuFind-General] Need Java Tuning Help on Windows box

We are running VuFind 1.1 on a Windows machine with Apache 2.2.17 and 8GB of RAM. We are having a difficult time tuning Java. Could someone explicitly explain which files (and their locations) need to be updated to increase memory to Java in this environment.

Own the Future-Intel&reg; Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest.
Compete for recognition, cash, and the chance to get your game
on Steam. $5K grand prize plus 10 genre and skill prizes.
Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d
VuFind-General mailing list