Looking back through this thread I thought I'd also throw in some other unrelated thoughts, since we are currently looking at issues related to GC in detail.

Something to consider is information like this: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

I mentioned that specifically in relation to the suggestion to use the system cache because it is a very similar idea. IF you have the RAM on your machine to hold the entire index in memory you might want to think about actually LOWERING your JVM RAM allocation (because it helps with GC times) and enabling MMapDirectory to get the kernel to cache the index in RAM for you. This way you only need to allocate enough RAM for the JVM to to hold data structures relating to things like caches, as opposed to your indexes (which will be rebuilt in memory every time a commit comes through, thus triggering GC). Less stuff inside the JVM equals less GC. If you look at the JVM with a tool like top in this instance it will appear to have a huge virtual memory allocation, but your JVM heap is much lower if connected to jconsole or something similar.

I should stress though, that we are still experimenting in this area, so no magic bullets yet. The more promising evolution to the above will be in Solr 4 to start using soft commits and stop scrapping our in-memory data structures so often to further reduce garbage collection.

Ta,
Greg

On 9 October 2012 22:21, Greg Pendlebury <greg.pendlebury@gmail.com> wrote:
I'm far from an expert on this stuff, but I believe we are using these tools to load balance in software (ie. not a dedicated hardware load balancer):
If you want further info I'd have to go ask one of our sysadmins, although I know Mark also loiters around the list and I assume he is intimately familiar with the setup.

Ta,
Greg


On 9 October 2012 20:09, Osullivan L. <L.Osullivan@swansea.ac.uk> wrote:
Hi Greg,

Thanks for this - can you point us in the direction of any documentation regarding load balancing which you think is particularly useful?

Cheers,

Luke

From: Greg Pendlebury [greg.pendlebury@gmail.com]
Sent: 08 October 2012 22:16
To: Osullivan L.
Cc: Tim Fletcher; vufind-general@lists.sourceforge.net

Subject: Re: [VuFind-General] Solr seems slow

You could potentially consider running multiple replicated indexes behind a load balancer and having them restart themselves when GC problems are detected. The load balancer will avoid service disruptions when only one of the indexes is offline. NLA uses this mechanism extensively, and spreading the load out across two machines will also help to reduce the severity of your problems as well.

Ta,
Greg

On 8 October 2012 20:21, Osullivan L. <L.Osullivan@swansea.ac.uk> wrote:
Hi Tim,

We're having similar problems at Swansea - not related to speed but rather to garbage collection. When your Java garbage gets full, vufind will be slower as the system is frantically trying to empty garbage / close down process to free up memory.

We use the GC monitoring process described in the VuFind Wiki to restart VuFind when the GC gets full. Unfortunately, this can happen at any time (depending on volume of searches) so we have established a 4 hour restart routine. We restart VuFind at 8am, 12pm and 4pm throughout the day. It's not ideal but it at least avoids restarts during the hour when there may be induction sessions in process.

We came to a decision on the precise restart times by monitoring the frequency of garbage collection. At present, during peak times, this averages 5 hours and 17 minutes. We chose a four hour rule to be on the safe side.

Cheers,

Luke





From: Tim Fletcher [T.Fletcher@bbk.ac.uk]
Sent: 08 October 2012 11:03
To: vufind-general@lists.sourceforge.net

Subject: Re: [VuFind-General] Solr seems slow

Hi,

 

We also had a slow down on Friday as use of the system increased and more or less ground to a halt. It came back after a quick restart so I assume that a regular restart will also help?

 

We haven’t adjusted any settings so will be doing that – can I confirm that the place to make the initial JAVA_OPTIONS changes is in vufind.sh?

 

Thanks,

 

Tim

----------

Tim Fletcher

Birkbeck Library

 

From: Demian Katz [mailto:demian.katz@villanova.edu]
Sent: 05 October 2012 18:13
To: kevin smith
Cc: vufind-general@lists.sourceforge.net
Subject: Re: [VuFind-General] Solr seems slow

 

I'm not aware of a specific recent Java problem that would cause this, but I haven't been following the Solr lists very closely lately, so it's entirely possible your theory is correct -- it sounds plausible even though I have no specific evidence that it is correct.

- Demian


From: kevin smith [ashkev@gmail.com]
Sent: Friday, October 05, 2012 1:08 PM
To: Demian Katz
Cc: Tuan Nguyen; vufind-general@lists.sourceforge.net
Subject: Re: [VuFind-General] Solr seems slow

Thanks,

 

I just did all of the package updates for Ubuntu and after this, there is a noticeable improvement in performance.  I suspect that there was a problem in the last update to openjdk-6-jre and associated packages, and the new update fixed it.  Does this sound plausible?

 

 

On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <demian.katz@villanova.edu> wrote:

Thanks for the tip -- I hadn't heard that one before.  I've added it to the wiki (with proper credit, of course):

http://vufind.org/wiki/performance#exploiting_the_system_cache

- Demian
________________________________________
From: Tuan Nguyen [tuan@yorku.ca]
Sent: Thursday, October 04, 2012 11:09 PM
To: kevin smith
Cc: vufind-general@lists.sourceforge.net
Subject: Re: [VuFind-General] Solr seems slow


If on linux, then you can also try to load your entire index into the system cache. This speeds up searches a lot for us.

cat /usr/local/vufind/solr/biblio/index/* > /dev/null




On 2012-10-04, at 3:03 PM, kevin smith wrote:

> We have switched all search links on our webpage from HIP to VuFind.  We are getting much more traffic, and I have noticed some real performance drops.  The first problem was my Apache settings, but I am no longer getting the  Server seems to be busy Apache errors.
>
> I have increased the memory allocated to Java.
>
> JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5"
>
> It still seems like Solr is running somewhat slowly...   Any ideas about speeding things up?
>
> Our catalog is at catalog.wakegov.com
>
> --
> Kevin Smith
> Digital Library Manager
> Wake County Public Libraries
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________
> VuFind-General mailing list
> VuFind-General@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/vufind-general


------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
VuFind-General mailing list
VuFind-General@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/vufind-general



 

--
Kevin Smith
Digital Library Manager
Wake County Public Libraries


------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
VuFind-General mailing list
VuFind-General@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/vufind-general