From: kevin s. <as...@gm...> - 2012-10-04 19:03:21
|
We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. I have increased the memory allocated to Java. JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? Our catalog is at catalog.wakegov.com -- Kevin Smith Digital Library Manager Wake County Public Libraries |
From: Demian K. <dem...@vi...> - 2012-10-04 19:21:59
|
You might want to read my blog post about Java tuning if you haven't already - if nothing else, it includes some ideas on studying Java garbage collection behavior, which can sometimes offer clues: http://blog.library.villanova.edu/libtech/2011/03/31/java-tuning-made-easier/ Also, some other ideas: 1.) Sometimes you can give Java too much memory. If you end up starving the OS so that it can't manage other resources well, having a lot of memory available to Java is actually a bad thing. 2.) If you turn on debug mode in config.ini, you can see all of the Solr queries as they execute. If you copy and paste one of the Solr request URLs into a new browser window, change localhost to the appropriate server name, and add "&debugQuery=true" to the end, you'll see details on what Solr is actually doing. This might offer clues about where bottlenecks lie. - Demian From: kevin smith [mailto:as...@gm...] Sent: Thursday, October 04, 2012 3:03 PM To: vuf...@li... Subject: [VuFind-General] Solr seems slow We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. I have increased the memory allocated to Java. JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? Our catalog is at catalog.wakegov.com<http://catalog.wakegov.com> -- Kevin Smith Digital Library Manager Wake County Public Libraries |
From: Tuan N. <tu...@yo...> - 2012-10-05 03:09:53
|
If on linux, then you can also try to load your entire index into the system cache. This speeds up searches a lot for us. cat /usr/local/vufind/solr/biblio/index/* > /dev/null On 2012-10-04, at 3:03 PM, kevin smith wrote: > We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. > > I have increased the memory allocated to Java. > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" > > It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? > > Our catalog is at catalog.wakegov.com > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general |
From: Demian K. <dem...@vi...> - 2012-10-05 10:51:48
|
Thanks for the tip -- I hadn't heard that one before. I've added it to the wiki (with proper credit, of course): http://vufind.org/wiki/performance#exploiting_the_system_cache - Demian ________________________________________ From: Tuan Nguyen [tu...@yo...] Sent: Thursday, October 04, 2012 11:09 PM To: kevin smith Cc: vuf...@li... Subject: Re: [VuFind-General] Solr seems slow If on linux, then you can also try to load your entire index into the system cache. This speeds up searches a lot for us. cat /usr/local/vufind/solr/biblio/index/* > /dev/null On 2012-10-04, at 3:03 PM, kevin smith wrote: > We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. > > I have increased the memory allocated to Java. > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" > > It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? > > Our catalog is at catalog.wakegov.com > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ VuFind-General mailing list VuF...@li... https://lists.sourceforge.net/lists/listinfo/vufind-general |
From: Demian K. <dem...@vi...> - 2012-10-05 17:13:24
|
I'm not aware of a specific recent Java problem that would cause this, but I haven't been following the Solr lists very closely lately, so it's entirely possible your theory is correct -- it sounds plausible even though I have no specific evidence that it is correct. - Demian ________________________________ From: kevin smith [as...@gm...] Sent: Friday, October 05, 2012 1:08 PM To: Demian Katz Cc: Tuan Nguyen; vuf...@li... Subject: Re: [VuFind-General] Solr seems slow Thanks, I just did all of the package updates for Ubuntu and after this, there is a noticeable improvement in performance. I suspect that there was a problem in the last update to openjdk-6-jre and associated packages, and the new update fixed it. Does this sound plausible? On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...<mailto:dem...@vi...>> wrote: Thanks for the tip -- I hadn't heard that one before. I've added it to the wiki (with proper credit, of course): http://vufind.org/wiki/performance#exploiting_the_system_cache - Demian ________________________________________ From: Tuan Nguyen [tu...@yo...<mailto:tu...@yo...>] Sent: Thursday, October 04, 2012 11:09 PM To: kevin smith Cc: vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow If on linux, then you can also try to load your entire index into the system cache. This speeds up searches a lot for us. cat /usr/local/vufind/solr/biblio/index/* > /dev/null On 2012-10-04, at 3:03 PM, kevin smith wrote: > We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. > > I have increased the memory allocated to Java. > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" > > It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? > > Our catalog is at catalog.wakegov.com<http://catalog.wakegov.com> > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > VuFind-General mailing list > VuF...@li...<mailto:VuF...@li...> > https://lists.sourceforge.net/lists/listinfo/vufind-general ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ VuFind-General mailing list VuF...@li...<mailto:VuF...@li...> https://lists.sourceforge.net/lists/listinfo/vufind-general -- Kevin Smith Digital Library Manager Wake County Public Libraries |
From: Tim F. <T.F...@bb...> - 2012-10-08 10:03:30
|
Hi, We also had a slow down on Friday as use of the system increased and more or less ground to a halt. It came back after a quick restart so I assume that a regular restart will also help? We haven't adjusted any settings so will be doing that - can I confirm that the place to make the initial JAVA_OPTIONS changes is in vufind.sh? Thanks, Tim ---------- Tim Fletcher Birkbeck Library From: Demian Katz [mailto:dem...@vi...] Sent: 05 October 2012 18:13 To: kevin smith Cc: vuf...@li... Subject: Re: [VuFind-General] Solr seems slow I'm not aware of a specific recent Java problem that would cause this, but I haven't been following the Solr lists very closely lately, so it's entirely possible your theory is correct -- it sounds plausible even though I have no specific evidence that it is correct. - Demian ________________________________ From: kevin smith [as...@gm...] Sent: Friday, October 05, 2012 1:08 PM To: Demian Katz Cc: Tuan Nguyen; vuf...@li... Subject: Re: [VuFind-General] Solr seems slow Thanks, I just did all of the package updates for Ubuntu and after this, there is a noticeable improvement in performance. I suspect that there was a problem in the last update to openjdk-6-jre and associated packages, and the new update fixed it. Does this sound plausible? On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...> wrote: Thanks for the tip -- I hadn't heard that one before. I've added it to the wiki (with proper credit, of course): http://vufind.org/wiki/performance#exploiting_the_system_cache - Demian ________________________________________ From: Tuan Nguyen [tu...@yo...] Sent: Thursday, October 04, 2012 11:09 PM To: kevin smith Cc: vuf...@li... Subject: Re: [VuFind-General] Solr seems slow If on linux, then you can also try to load your entire index into the system cache. This speeds up searches a lot for us. cat /usr/local/vufind/solr/biblio/index/* > /dev/null On 2012-10-04, at 3:03 PM, kevin smith wrote: > We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. > > I have increased the memory allocated to Java. > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" > > It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? > > Our catalog is at catalog.wakegov.com > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > ------------------------------------------------------------------------ ------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev____________________________________ ___________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general ------------------------------------------------------------------------ ------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ VuFind-General mailing list VuF...@li... https://lists.sourceforge.net/lists/listinfo/vufind-general -- Kevin Smith Digital Library Manager Wake County Public Libraries |
From: Demian K. <dem...@vi...> - 2012-10-08 12:41:36
|
Yes, if you edit the JAVA_OPTIONS in vufind.sh, that should get you the effect you desire. And as Luke says, restarting periodically is helpful to cut down on garbage collection problems. (At Villanova, we only restart once a day - in the middle of the night - and things work pretty reliably all day; obviously this all depends on how much memory you have available and how heavy your load is). - Demian From: Tim Fletcher [mailto:T.F...@bb...] Sent: Monday, October 08, 2012 6:03 AM To: vuf...@li... Subject: Re: [VuFind-General] Solr seems slow Hi, We also had a slow down on Friday as use of the system increased and more or less ground to a halt. It came back after a quick restart so I assume that a regular restart will also help? We haven't adjusted any settings so will be doing that - can I confirm that the place to make the initial JAVA_OPTIONS changes is in vufind.sh? Thanks, Tim ---------- Tim Fletcher Birkbeck Library From: Demian Katz [mailto:dem...@vi...] Sent: 05 October 2012 18:13 To: kevin smith Cc: vuf...@li... Subject: Re: [VuFind-General] Solr seems slow I'm not aware of a specific recent Java problem that would cause this, but I haven't been following the Solr lists very closely lately, so it's entirely possible your theory is correct -- it sounds plausible even though I have no specific evidence that it is correct. - Demian ________________________________ From: kevin smith [as...@gm...] Sent: Friday, October 05, 2012 1:08 PM To: Demian Katz Cc: Tuan Nguyen; vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow Thanks, I just did all of the package updates for Ubuntu and after this, there is a noticeable improvement in performance. I suspect that there was a problem in the last update to openjdk-6-jre and associated packages, and the new update fixed it. Does this sound plausible? On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...<mailto:dem...@vi...>> wrote: Thanks for the tip -- I hadn't heard that one before. I've added it to the wiki (with proper credit, of course): http://vufind.org/wiki/performance#exploiting_the_system_cache - Demian ________________________________________ From: Tuan Nguyen [tu...@yo...<mailto:tu...@yo...>] Sent: Thursday, October 04, 2012 11:09 PM To: kevin smith Cc: vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow If on linux, then you can also try to load your entire index into the system cache. This speeds up searches a lot for us. cat /usr/local/vufind/solr/biblio/index/* > /dev/null On 2012-10-04, at 3:03 PM, kevin smith wrote: > We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. > > I have increased the memory allocated to Java. > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" > > It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? > > Our catalog is at catalog.wakegov.com<http://catalog.wakegov.com> > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > VuFind-General mailing list > VuF...@li...<mailto:VuF...@li...> > https://lists.sourceforge.net/lists/listinfo/vufind-general ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ VuFind-General mailing list VuF...@li...<mailto:VuF...@li...> https://lists.sourceforge.net/lists/listinfo/vufind-general -- Kevin Smith Digital Library Manager Wake County Public Libraries |
From: kevin s. <as...@gm...> - 2012-10-05 17:08:43
|
Thanks, I just did all of the package updates for Ubuntu and after this, there is a noticeable improvement in performance. I suspect that there was a problem in the last update to openjdk-6-jre and associated packages, and the new update fixed it. Does this sound plausible? On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...>wrote: > Thanks for the tip -- I hadn't heard that one before. I've added it to > the wiki (with proper credit, of course): > > http://vufind.org/wiki/performance#exploiting_the_system_cache > > - Demian > ________________________________________ > From: Tuan Nguyen [tu...@yo...] > Sent: Thursday, October 04, 2012 11:09 PM > To: kevin smith > Cc: vuf...@li... > Subject: Re: [VuFind-General] Solr seems slow > > If on linux, then you can also try to load your entire index into the > system cache. This speeds up searches a lot for us. > > cat /usr/local/vufind/solr/biblio/index/* > /dev/null > > > > > On 2012-10-04, at 3:03 PM, kevin smith wrote: > > > We have switched all search links on our webpage from HIP to VuFind. We > are getting much more traffic, and I have noticed some real performance > drops. The first problem was my Apache settings, but I am no longer > getting the Server seems to be busy Apache errors. > > > > I have increased the memory allocated to Java. > > > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC > -XX:NewRatio=5" > > > > It still seems like Solr is running somewhat slowly... Any ideas about > speeding things up? > > > > Our catalog is at catalog.wakegov.com > > > > -- > > Kevin Smith > > Digital Library Manager > > Wake County Public Libraries > > > ------------------------------------------------------------------------------ > > Don't let slow site performance ruin your business. Deploy New Relic APM > > Deploy New Relic app performance management and know exactly > > what is happening inside your Ruby, Python, PHP, Java, and .NET app > > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > > > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > > VuFind-General mailing list > > VuF...@li... > > https://lists.sourceforge.net/lists/listinfo/vufind-general > > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general > -- Kevin Smith Digital Library Manager Wake County Public Libraries |
From: Osullivan L. <L.O...@sw...> - 2012-10-08 10:22:12
|
Hi Tim, We're having similar problems at Swansea - not related to speed but rather to garbage collection. When your Java garbage gets full, vufind will be slower as the system is frantically trying to empty garbage / close down process to free up memory. We use the GC monitoring process described in the VuFind Wiki to restart VuFind when the GC gets full. Unfortunately, this can happen at any time (depending on volume of searches) so we have established a 4 hour restart routine. We restart VuFind at 8am, 12pm and 4pm throughout the day. It's not ideal but it at least avoids restarts during the hour when there may be induction sessions in process. We came to a decision on the precise restart times by monitoring the frequency of garbage collection. At present, during peak times, this averages 5 hours and 17 minutes. We chose a four hour rule to be on the safe side. Cheers, Luke ________________________________ From: Tim Fletcher [T.F...@bb...] Sent: 08 October 2012 11:03 To: vuf...@li... Subject: Re: [VuFind-General] Solr seems slow Hi, We also had a slow down on Friday as use of the system increased and more or less ground to a halt. It came back after a quick restart so I assume that a regular restart will also help? We haven’t adjusted any settings so will be doing that – can I confirm that the place to make the initial JAVA_OPTIONS changes is in vufind.sh? Thanks, Tim ---------- Tim Fletcher Birkbeck Library From: Demian Katz [mailto:dem...@vi...] Sent: 05 October 2012 18:13 To: kevin smith Cc: vuf...@li... Subject: Re: [VuFind-General] Solr seems slow I'm not aware of a specific recent Java problem that would cause this, but I haven't been following the Solr lists very closely lately, so it's entirely possible your theory is correct -- it sounds plausible even though I have no specific evidence that it is correct. - Demian ________________________________ From: kevin smith [as...@gm...] Sent: Friday, October 05, 2012 1:08 PM To: Demian Katz Cc: Tuan Nguyen; vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow Thanks, I just did all of the package updates for Ubuntu and after this, there is a noticeable improvement in performance. I suspect that there was a problem in the last update to openjdk-6-jre and associated packages, and the new update fixed it. Does this sound plausible? On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...<mailto:dem...@vi...>> wrote: Thanks for the tip -- I hadn't heard that one before. I've added it to the wiki (with proper credit, of course): http://vufind.org/wiki/performance#exploiting_the_system_cache - Demian ________________________________________ From: Tuan Nguyen [tu...@yo...<mailto:tu...@yo...>] Sent: Thursday, October 04, 2012 11:09 PM To: kevin smith Cc: vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow If on linux, then you can also try to load your entire index into the system cache. This speeds up searches a lot for us. cat /usr/local/vufind/solr/biblio/index/* > /dev/null On 2012-10-04, at 3:03 PM, kevin smith wrote: > We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. > > I have increased the memory allocated to Java. > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" > > It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? > > Our catalog is at catalog.wakegov.com<http://catalog.wakegov.com> > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > VuFind-General mailing list > VuF...@li...<mailto:VuF...@li...> > https://lists.sourceforge.net/lists/listinfo/vufind-general ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ VuFind-General mailing list VuF...@li...<mailto:VuF...@li...> https://lists.sourceforge.net/lists/listinfo/vufind-general -- Kevin Smith Digital Library Manager Wake County Public Libraries |
From: Greg P. <gre...@gm...> - 2012-10-08 21:16:24
|
You could potentially consider running multiple replicated indexes behind a load balancer and having them restart themselves when GC problems are detected. The load balancer will avoid service disruptions when only one of the indexes is offline. NLA uses this mechanism extensively, and spreading the load out across two machines will also help to reduce the severity of your problems as well. Ta, Greg On 8 October 2012 20:21, Osullivan L. <L.O...@sw...> wrote: > Hi Tim, > > We're having similar problems at Swansea - not related to speed but rather > to garbage collection. When your Java garbage gets full, vufind will be > slower as the system is frantically trying to empty garbage / close down > process to free up memory. > > We use the GC monitoring process described in the VuFind Wiki to restart > VuFind when the GC gets full. Unfortunately, this can happen at any time > (depending on volume of searches) so we have established a 4 hour restart > routine. We restart VuFind at 8am, 12pm and 4pm throughout the day. It's > not ideal but it at least avoids restarts during the hour when there may be > induction sessions in process. > > We came to a decision on the precise restart times by monitoring the > frequency of garbage collection. At present, during peak times, this > averages 5 hours and 17 minutes. We chose a four hour rule to be on the > safe side. > > Cheers, > > Luke > > > > > ------------------------------ > *From:* Tim Fletcher [T.F...@bb...] > *Sent:* 08 October 2012 11:03 > *To:* vuf...@li... > > *Subject:* Re: [VuFind-General] Solr seems slow > > Hi, > > > > We also had a slow down on Friday as use of the system increased and more > or less ground to a halt. It came back after a quick restart so I assume > that a regular restart will also help? > > > > We haven’t adjusted any settings so will be doing that – can I confirm > that the place to make the initial JAVA_OPTIONS changes is in vufind.sh? > > > > Thanks, > > > > Tim > > ---------- > > Tim Fletcher > > Birkbeck Library > > > > *From:* Demian Katz [mailto:dem...@vi...] > *Sent:* 05 October 2012 18:13 > *To:* kevin smith > *Cc:* vuf...@li... > *Subject:* Re: [VuFind-General] Solr seems slow > > > > I'm not aware of a specific recent Java problem that would cause this, but > I haven't been following the Solr lists very closely lately, so it's > entirely possible your theory is correct -- it sounds plausible even though > I have no specific evidence that it is correct. > > - Demian > ------------------------------ > > *From:* kevin smith [as...@gm...] > *Sent:* Friday, October 05, 2012 1:08 PM > *To:* Demian Katz > *Cc:* Tuan Nguyen; vuf...@li... > *Subject:* Re: [VuFind-General] Solr seems slow > > Thanks, > > > > I just did all of the package updates for Ubuntu and after this, there is > a noticeable improvement in performance. I suspect that there was a > problem in the last update to openjdk-6-jre and associated packages, and > the new update fixed it. Does this sound plausible? > > > > > > On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...> > wrote: > > Thanks for the tip -- I hadn't heard that one before. I've added it to > the wiki (with proper credit, of course): > > http://vufind.org/wiki/performance#exploiting_the_system_cache > > - Demian > ________________________________________ > From: Tuan Nguyen [tu...@yo...] > Sent: Thursday, October 04, 2012 11:09 PM > To: kevin smith > Cc: vuf...@li... > Subject: Re: [VuFind-General] Solr seems slow > > > If on linux, then you can also try to load your entire index into the > system cache. This speeds up searches a lot for us. > > cat /usr/local/vufind/solr/biblio/index/* > /dev/null > > > > > On 2012-10-04, at 3:03 PM, kevin smith wrote: > > > We have switched all search links on our webpage from HIP to VuFind. We > are getting much more traffic, and I have noticed some real performance > drops. The first problem was my Apache settings, but I am no longer > getting the Server seems to be busy Apache errors. > > > > I have increased the memory allocated to Java. > > > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC > -XX:NewRatio=5" > > > > It still seems like Solr is running somewhat slowly... Any ideas about > speeding things up? > > > > Our catalog is at catalog.wakegov.com > > > > -- > > Kevin Smith > > Digital Library Manager > > Wake County Public Libraries > > > ------------------------------------------------------------------------------ > > Don't let slow site performance ruin your business. Deploy New Relic APM > > Deploy New Relic app performance management and know exactly > > what is happening inside your Ruby, Python, PHP, Java, and .NET app > > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > > > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > > VuFind-General mailing list > > VuF...@li... > > https://lists.sourceforge.net/lists/listinfo/vufind-general > > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general > > > > > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general > > |
From: Osullivan L. <L.O...@sw...> - 2012-10-09 09:10:14
|
Hi Greg, Thanks for this - can you point us in the direction of any documentation regarding load balancing which you think is particularly useful? Cheers, Luke ________________________________ From: Greg Pendlebury [gre...@gm...] Sent: 08 October 2012 22:16 To: Osullivan L. Cc: Tim Fletcher; vuf...@li... Subject: Re: [VuFind-General] Solr seems slow You could potentially consider running multiple replicated indexes behind a load balancer and having them restart themselves when GC problems are detected. The load balancer will avoid service disruptions when only one of the indexes is offline. NLA uses this mechanism extensively, and spreading the load out across two machines will also help to reduce the severity of your problems as well. Ta, Greg On 8 October 2012 20:21, Osullivan L. <L.O...@sw...<mailto:L.O...@sw...>> wrote: Hi Tim, We're having similar problems at Swansea - not related to speed but rather to garbage collection. When your Java garbage gets full, vufind will be slower as the system is frantically trying to empty garbage / close down process to free up memory. We use the GC monitoring process described in the VuFind Wiki to restart VuFind when the GC gets full. Unfortunately, this can happen at any time (depending on volume of searches) so we have established a 4 hour restart routine. We restart VuFind at 8am, 12pm and 4pm throughout the day. It's not ideal but it at least avoids restarts during the hour when there may be induction sessions in process. We came to a decision on the precise restart times by monitoring the frequency of garbage collection. At present, during peak times, this averages 5 hours and 17 minutes. We chose a four hour rule to be on the safe side. Cheers, Luke ________________________________ From: Tim Fletcher [T.F...@bb...<mailto:T.F...@bb...>] Sent: 08 October 2012 11:03 To: vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow Hi, We also had a slow down on Friday as use of the system increased and more or less ground to a halt. It came back after a quick restart so I assume that a regular restart will also help? We haven’t adjusted any settings so will be doing that – can I confirm that the place to make the initial JAVA_OPTIONS changes is in vufind.sh? Thanks, Tim ---------- Tim Fletcher Birkbeck Library From: Demian Katz [mailto:dem...@vi...<mailto:dem...@vi...>] Sent: 05 October 2012 18:13 To: kevin smith Cc: vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow I'm not aware of a specific recent Java problem that would cause this, but I haven't been following the Solr lists very closely lately, so it's entirely possible your theory is correct -- it sounds plausible even though I have no specific evidence that it is correct. - Demian ________________________________ From: kevin smith [as...@gm...<mailto:as...@gm...>] Sent: Friday, October 05, 2012 1:08 PM To: Demian Katz Cc: Tuan Nguyen; vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow Thanks, I just did all of the package updates for Ubuntu and after this, there is a noticeable improvement in performance. I suspect that there was a problem in the last update to openjdk-6-jre and associated packages, and the new update fixed it. Does this sound plausible? On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...<mailto:dem...@vi...>> wrote: Thanks for the tip -- I hadn't heard that one before. I've added it to the wiki (with proper credit, of course): http://vufind.org/wiki/performance#exploiting_the_system_cache - Demian ________________________________________ From: Tuan Nguyen [tu...@yo...<mailto:tu...@yo...>] Sent: Thursday, October 04, 2012 11:09 PM To: kevin smith Cc: vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow If on linux, then you can also try to load your entire index into the system cache. This speeds up searches a lot for us. cat /usr/local/vufind/solr/biblio/index/* > /dev/null On 2012-10-04, at 3:03 PM, kevin smith wrote: > We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. > > I have increased the memory allocated to Java. > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" > > It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? > > Our catalog is at catalog.wakegov.com<http://catalog.wakegov.com> > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > VuFind-General mailing list > VuF...@li...<mailto:VuF...@li...> > https://lists.sourceforge.net/lists/listinfo/vufind-general ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ VuFind-General mailing list VuF...@li...<mailto:VuF...@li...> https://lists.sourceforge.net/lists/listinfo/vufind-general -- Kevin Smith Digital Library Manager Wake County Public Libraries ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ VuFind-General mailing list VuF...@li...<mailto:VuF...@li...> https://lists.sourceforge.net/lists/listinfo/vufind-general |
From: Till K. <kin...@gm...> - 2012-10-09 11:43:24
|
It is very easy to use HTTP load balancing with Solr: Because it is a completely stateless HTTP application, you don't need advanced mechanisms for session management or so (eg. routing of HTTP requests belonging to the same "user session" always to the the same backend server, or advanced mechanisms on the backend application to maintain the same session data on all instances). For each single request let the load balancer select one backend server based on some simple rule. So any simple HTTP load balancer with a fairly simple configuration will do. We are using mod_proxy_balancer of Apache with three Solr servers in the back on one of our web servers (that runs Apache anyway, so it is just an additional module on one Apache, that does several other jobs as well): http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html We use the "bybusyness" method as load balancer scheduler algorithm, because we see quite different response times from the Solr servers based on request complexity. But I guess, you don't need to bother too much about the appropriate algorithm with Solr... We haven't tried dedicated load balancing software, because this Apache solution does the job for us (keep it simple), so I can't comment on them... Till Am 09.10.2012 11:09, schrieb Osullivan L.: > Hi Greg, > > Thanks for this - can you point us in the direction of any documentation > regarding load balancing which you think is particularly useful? > > Cheers, > > Luke > ------------------------------------------------------------------------ > *From:* Greg Pendlebury [gre...@gm...] > *Sent:* 08 October 2012 22:16 > *To:* Osullivan L. > *Cc:* Tim Fletcher; vuf...@li... > *Subject:* Re: [VuFind-General] Solr seems slow > > You could potentially consider running multiple replicated indexes > behind a load balancer and having them restart themselves when GC > problems are detected. The load balancer will avoid service disruptions > when only one of the indexes is offline. NLA uses this mechanism > extensively, and spreading the load out across two machines will also > help to reduce the severity of your problems as well. > > Ta, > Greg > > On 8 October 2012 20:21, Osullivan L. <L.O...@sw... > <mailto:L.O...@sw...>> wrote: > > Hi Tim, > > We're having similar problems at Swansea - not related to speed but > rather to garbage collection. When your Java garbage gets full, > vufind will be slower as the system is frantically trying to empty > garbage / close down process to free up memory. > > We use the GC monitoring process described in the VuFind Wiki to > restart VuFind when the GC gets full. Unfortunately, this can happen > at any time (depending on volume of searches) so we have established > a 4 hour restart routine. We restart VuFind at 8am, 12pm and 4pm > throughout the day. It's not ideal but it at least avoids restarts > during the hour when there may be induction sessions in process. > > We came to a decision on the precise restart times by monitoring the > frequency of garbage collection. At present, during peak times, this > averages 5 hours and 17 minutes. We chose a four hour rule to be on > the safe side. > > Cheers, > > Luke > > > > > ------------------------------------------------------------------------ > *From:* Tim Fletcher [T.F...@bb... > <mailto:T.F...@bb...>] > *Sent:* 08 October 2012 11:03 > *To:* vuf...@li... > <mailto:vuf...@li...> > > *Subject:* Re: [VuFind-General] Solr seems slow > > Hi, > > > > We also had a slow down on Friday as use of the system increased and > more or less ground to a halt. It came back after a quick restart so > I assume that a regular restart will also help? > > > > We haven’t adjusted any settings so will be doing that – can I > confirm that the place to make the initial JAVA_OPTIONS changes is > in vufind.sh? > > > > Thanks, > > > > Tim > > ---------- > > Tim Fletcher > > Birkbeck Library > > > > *From:*Demian Katz [mailto:dem...@vi... > <mailto:dem...@vi...>] > *Sent:* 05 October 2012 18:13 > *To:* kevin smith > *Cc:* vuf...@li... > <mailto:vuf...@li...> > *Subject:* Re: [VuFind-General] Solr seems slow > > > > I'm not aware of a specific recent Java problem that would cause > this, but I haven't been following the Solr lists very closely > lately, so it's entirely possible your theory is correct -- it > sounds plausible even though I have no specific evidence that it is > correct. > > - Demian > > ------------------------------------------------------------------------ > > *From:*kevin smith [as...@gm... <mailto:as...@gm...>] > *Sent:* Friday, October 05, 2012 1:08 PM > *To:* Demian Katz > *Cc:* Tuan Nguyen; vuf...@li... > <mailto:vuf...@li...> > *Subject:* Re: [VuFind-General] Solr seems slow > > Thanks, > > > > I just did all of the package updates for Ubuntu and after this, > there is a noticeable improvement in performance. I suspect that > there was a problem in the last update to openjdk-6-jre and > associated packages, and the new update fixed it. Does this sound > plausible? > > > > > > On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz > <dem...@vi... <mailto:dem...@vi...>> wrote: > > Thanks for the tip -- I hadn't heard that one before. I've added it > to the wiki (with proper credit, of course): > > http://vufind.org/wiki/performance#exploiting_the_system_cache > > - Demian > ________________________________________ > From: Tuan Nguyen [tu...@yo... <mailto:tu...@yo...>] > Sent: Thursday, October 04, 2012 11:09 PM > To: kevin smith > Cc: vuf...@li... > <mailto:vuf...@li...> > Subject: Re: [VuFind-General] Solr seems slow > > > If on linux, then you can also try to load your entire index into > the system cache. This speeds up searches a lot for us. > > cat /usr/local/vufind/solr/biblio/index/* > /dev/null > > > > > On 2012-10-04, at 3:03 PM, kevin smith wrote: > > > We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. > > > > I have increased the memory allocated to Java. > > > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" > > > > It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? > > > > Our catalog is at catalog.wakegov.com <http://catalog.wakegov.com> > > > > -- > > Kevin Smith > > Digital Library Manager > > Wake County Public Libraries > > ------------------------------------------------------------------------------ > > Don't let slow site performance ruin your business. Deploy New Relic APM > > Deploy New Relic app performance management and know exactly > > what is happening inside your Ruby, Python, PHP, Java, and .NET app > > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > > VuFind-General mailing list > > VuF...@li... > <mailto:VuF...@li...> > > https://lists.sourceforge.net/lists/listinfo/vufind-general > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ > VuFind-General mailing list > VuF...@li... > <mailto:VuF...@li...> > https://lists.sourceforge.net/lists/listinfo/vufind-general > > > > > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ > VuFind-General mailing list > VuF...@li... > <mailto:VuF...@li...> > https://lists.sourceforge.net/lists/listinfo/vufind-general > > > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > > > > _______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general > -- http://twitter.com/tillk |
From: Greg P. <gre...@gm...> - 2012-10-09 11:22:09
|
I'm far from an expert on this stuff, but I believe we are using these tools to load balance in software (ie. not a dedicated hardware load balancer): - http://haproxy.1wt.eu/ (the main one for load balancing) - http://wiki.nginx.org/Main If you want further info I'd have to go ask one of our sysadmins, although I know Mark also loiters around the list and I assume he is intimately familiar with the setup. Ta, Greg On 9 October 2012 20:09, Osullivan L. <L.O...@sw...> wrote: > Hi Greg, > > Thanks for this - can you point us in the direction of any documentation > regarding load balancing which you think is particularly useful? > > Cheers, > > Luke > ------------------------------ > *From:* Greg Pendlebury [gre...@gm...] > *Sent:* 08 October 2012 22:16 > *To:* Osullivan L. > *Cc:* Tim Fletcher; vuf...@li... > > *Subject:* Re: [VuFind-General] Solr seems slow > > You could potentially consider running multiple replicated indexes > behind a load balancer and having them restart themselves when GC problems > are detected. The load balancer will avoid service disruptions when only > one of the indexes is offline. NLA uses this mechanism extensively, and > spreading the load out across two machines will also help to reduce the > severity of your problems as well. > > Ta, > Greg > > On 8 October 2012 20:21, Osullivan L. <L.O...@sw...> wrote: > >> Hi Tim, >> >> We're having similar problems at Swansea - not related to speed but >> rather to garbage collection. When your Java garbage gets full, vufind will >> be slower as the system is frantically trying to empty garbage / close down >> process to free up memory. >> >> We use the GC monitoring process described in the VuFind Wiki to restart >> VuFind when the GC gets full. Unfortunately, this can happen at any time >> (depending on volume of searches) so we have established a 4 hour restart >> routine. We restart VuFind at 8am, 12pm and 4pm throughout the day. It's >> not ideal but it at least avoids restarts during the hour when there may be >> induction sessions in process. >> >> We came to a decision on the precise restart times by monitoring the >> frequency of garbage collection. At present, during peak times, this >> averages 5 hours and 17 minutes. We chose a four hour rule to be on the >> safe side. >> >> Cheers, >> >> Luke >> >> >> >> >> ------------------------------ >> *From:* Tim Fletcher [T.F...@bb...] >> *Sent:* 08 October 2012 11:03 >> *To:* vuf...@li... >> >> *Subject:* Re: [VuFind-General] Solr seems slow >> >> Hi, >> >> >> >> We also had a slow down on Friday as use of the system increased and more >> or less ground to a halt. It came back after a quick restart so I assume >> that a regular restart will also help? >> >> >> >> We haven’t adjusted any settings so will be doing that – can I confirm >> that the place to make the initial JAVA_OPTIONS changes is in vufind.sh? >> >> >> >> Thanks, >> >> >> >> Tim >> >> ---------- >> >> Tim Fletcher >> >> Birkbeck Library >> >> >> >> *From:* Demian Katz [mailto:dem...@vi...] >> *Sent:* 05 October 2012 18:13 >> *To:* kevin smith >> *Cc:* vuf...@li... >> *Subject:* Re: [VuFind-General] Solr seems slow >> >> >> >> I'm not aware of a specific recent Java problem that would cause this, >> but I haven't been following the Solr lists very closely lately, so it's >> entirely possible your theory is correct -- it sounds plausible even though >> I have no specific evidence that it is correct. >> >> - Demian >> ------------------------------ >> >> *From:* kevin smith [as...@gm...] >> *Sent:* Friday, October 05, 2012 1:08 PM >> *To:* Demian Katz >> *Cc:* Tuan Nguyen; vuf...@li... >> *Subject:* Re: [VuFind-General] Solr seems slow >> >> Thanks, >> >> >> >> I just did all of the package updates for Ubuntu and after this, there is >> a noticeable improvement in performance. I suspect that there was a >> problem in the last update to openjdk-6-jre and associated packages, and >> the new update fixed it. Does this sound plausible? >> >> >> >> >> >> On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...> >> wrote: >> >> Thanks for the tip -- I hadn't heard that one before. I've added it to >> the wiki (with proper credit, of course): >> >> http://vufind.org/wiki/performance#exploiting_the_system_cache >> >> - Demian >> ________________________________________ >> From: Tuan Nguyen [tu...@yo...] >> Sent: Thursday, October 04, 2012 11:09 PM >> To: kevin smith >> Cc: vuf...@li... >> Subject: Re: [VuFind-General] Solr seems slow >> >> >> If on linux, then you can also try to load your entire index into the >> system cache. This speeds up searches a lot for us. >> >> cat /usr/local/vufind/solr/biblio/index/* > /dev/null >> >> >> >> >> On 2012-10-04, at 3:03 PM, kevin smith wrote: >> >> > We have switched all search links on our webpage from HIP to VuFind. >> We are getting much more traffic, and I have noticed some real performance >> drops. The first problem was my Apache settings, but I am no longer >> getting the Server seems to be busy Apache errors. >> > >> > I have increased the memory allocated to Java. >> > >> > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC >> -XX:NewRatio=5" >> > >> > It still seems like Solr is running somewhat slowly... Any ideas >> about speeding things up? >> > >> > Our catalog is at catalog.wakegov.com >> > >> > -- >> > Kevin Smith >> > Digital Library Manager >> > Wake County Public Libraries >> > >> ------------------------------------------------------------------------------ >> > Don't let slow site performance ruin your business. Deploy New Relic APM >> > Deploy New Relic app performance management and know exactly >> > what is happening inside your Ruby, Python, PHP, Java, and .NET app >> > Try New Relic at no cost today and get our sweet Data Nerd shirt too! >> > >> http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ >> > VuFind-General mailing list >> > VuF...@li... >> > https://lists.sourceforge.net/lists/listinfo/vufind-general >> >> >> >> ------------------------------------------------------------------------------ >> Don't let slow site performance ruin your business. Deploy New Relic APM >> Deploy New Relic app performance management and know exactly >> what is happening inside your Ruby, Python, PHP, Java, and .NET app >> Try New Relic at no cost today and get our sweet Data Nerd shirt too! >> http://p.sf.net/sfu/newrelic-dev2dev >> _______________________________________________ >> VuFind-General mailing list >> VuF...@li... >> https://lists.sourceforge.net/lists/listinfo/vufind-general >> >> >> >> >> >> -- >> Kevin Smith >> Digital Library Manager >> Wake County Public Libraries >> >> >> ------------------------------------------------------------------------------ >> Don't let slow site performance ruin your business. Deploy New Relic APM >> Deploy New Relic app performance management and know exactly >> what is happening inside your Ruby, Python, PHP, Java, and .NET app >> Try New Relic at no cost today and get our sweet Data Nerd shirt too! >> http://p.sf.net/sfu/newrelic-dev2dev >> _______________________________________________ >> VuFind-General mailing list >> VuF...@li... >> https://lists.sourceforge.net/lists/listinfo/vufind-general >> >> > |
From: Greg P. <gre...@gm...> - 2012-10-09 11:33:31
|
Looking back through this thread I thought I'd also throw in some other unrelated thoughts, since we are currently looking at issues related to GC in detail. Something to consider is information like this: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html I mentioned that specifically in relation to the suggestion to use the system cache because it is a very similar idea. IF you have the RAM on your machine to hold the entire index in memory you might want to think about actually LOWERING your JVM RAM allocation (because it helps with GC times) and enabling MMapDirectory to get the kernel to cache the index in RAM for you. This way you only need to allocate enough RAM for the JVM to to hold data structures relating to things like caches, as opposed to your indexes (which will be rebuilt in memory every time a commit comes through, thus triggering GC). Less stuff inside the JVM equals less GC. If you look at the JVM with a tool like top in this instance it will appear to have a huge virtual memory allocation, but your JVM heap is much lower if connected to jconsole or something similar. I should stress though, that we are still experimenting in this area, so no magic bullets yet. The more promising evolution to the above will be in Solr 4 to start using soft commits and stop scrapping our in-memory data structures so often to further reduce garbage collection. Ta, Greg On 9 October 2012 22:21, Greg Pendlebury <gre...@gm...> wrote: > I'm far from an expert on this stuff, but I believe we are using these > tools to load balance in software (ie. not a dedicated hardware load > balancer): > > - http://haproxy.1wt.eu/ (the main one for load balancing) > - http://wiki.nginx.org/Main > > If you want further info I'd have to go ask one of our sysadmins, although > I know Mark also loiters around the list and I assume he is intimately > familiar with the setup. > > Ta, > Greg > > > On 9 October 2012 20:09, Osullivan L. <L.O...@sw...> wrote: > >> Hi Greg, >> >> Thanks for this - can you point us in the direction of any documentation >> regarding load balancing which you think is particularly useful? >> >> Cheers, >> >> Luke >> ------------------------------ >> *From:* Greg Pendlebury [gre...@gm...] >> *Sent:* 08 October 2012 22:16 >> *To:* Osullivan L. >> *Cc:* Tim Fletcher; vuf...@li... >> >> *Subject:* Re: [VuFind-General] Solr seems slow >> >> You could potentially consider running multiple replicated indexes >> behind a load balancer and having them restart themselves when GC problems >> are detected. The load balancer will avoid service disruptions when only >> one of the indexes is offline. NLA uses this mechanism extensively, and >> spreading the load out across two machines will also help to reduce the >> severity of your problems as well. >> >> Ta, >> Greg >> >> On 8 October 2012 20:21, Osullivan L. <L.O...@sw...> wrote: >> >>> Hi Tim, >>> >>> We're having similar problems at Swansea - not related to speed but >>> rather to garbage collection. When your Java garbage gets full, vufind will >>> be slower as the system is frantically trying to empty garbage / close down >>> process to free up memory. >>> >>> We use the GC monitoring process described in the VuFind Wiki to restart >>> VuFind when the GC gets full. Unfortunately, this can happen at any time >>> (depending on volume of searches) so we have established a 4 hour restart >>> routine. We restart VuFind at 8am, 12pm and 4pm throughout the day. It's >>> not ideal but it at least avoids restarts during the hour when there may be >>> induction sessions in process. >>> >>> We came to a decision on the precise restart times by monitoring the >>> frequency of garbage collection. At present, during peak times, this >>> averages 5 hours and 17 minutes. We chose a four hour rule to be on the >>> safe side. >>> >>> Cheers, >>> >>> Luke >>> >>> >>> >>> >>> ------------------------------ >>> *From:* Tim Fletcher [T.F...@bb...] >>> *Sent:* 08 October 2012 11:03 >>> *To:* vuf...@li... >>> >>> *Subject:* Re: [VuFind-General] Solr seems slow >>> >>> Hi, >>> >>> >>> >>> We also had a slow down on Friday as use of the system increased and >>> more or less ground to a halt. It came back after a quick restart so I >>> assume that a regular restart will also help? >>> >>> >>> >>> We haven’t adjusted any settings so will be doing that – can I confirm >>> that the place to make the initial JAVA_OPTIONS changes is in vufind.sh? >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Tim >>> >>> ---------- >>> >>> Tim Fletcher >>> >>> Birkbeck Library >>> >>> >>> >>> *From:* Demian Katz [mailto:dem...@vi...] >>> *Sent:* 05 October 2012 18:13 >>> *To:* kevin smith >>> *Cc:* vuf...@li... >>> *Subject:* Re: [VuFind-General] Solr seems slow >>> >>> >>> >>> I'm not aware of a specific recent Java problem that would cause this, >>> but I haven't been following the Solr lists very closely lately, so it's >>> entirely possible your theory is correct -- it sounds plausible even though >>> I have no specific evidence that it is correct. >>> >>> - Demian >>> ------------------------------ >>> >>> *From:* kevin smith [as...@gm...] >>> *Sent:* Friday, October 05, 2012 1:08 PM >>> *To:* Demian Katz >>> *Cc:* Tuan Nguyen; vuf...@li... >>> *Subject:* Re: [VuFind-General] Solr seems slow >>> >>> Thanks, >>> >>> >>> >>> I just did all of the package updates for Ubuntu and after this, there >>> is a noticeable improvement in performance. I suspect that there was a >>> problem in the last update to openjdk-6-jre and associated packages, and >>> the new update fixed it. Does this sound plausible? >>> >>> >>> >>> >>> >>> On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...> >>> wrote: >>> >>> Thanks for the tip -- I hadn't heard that one before. I've added it to >>> the wiki (with proper credit, of course): >>> >>> http://vufind.org/wiki/performance#exploiting_the_system_cache >>> >>> - Demian >>> ________________________________________ >>> From: Tuan Nguyen [tu...@yo...] >>> Sent: Thursday, October 04, 2012 11:09 PM >>> To: kevin smith >>> Cc: vuf...@li... >>> Subject: Re: [VuFind-General] Solr seems slow >>> >>> >>> If on linux, then you can also try to load your entire index into the >>> system cache. This speeds up searches a lot for us. >>> >>> cat /usr/local/vufind/solr/biblio/index/* > /dev/null >>> >>> >>> >>> >>> On 2012-10-04, at 3:03 PM, kevin smith wrote: >>> >>> > We have switched all search links on our webpage from HIP to VuFind. >>> We are getting much more traffic, and I have noticed some real performance >>> drops. The first problem was my Apache settings, but I am no longer >>> getting the Server seems to be busy Apache errors. >>> > >>> > I have increased the memory allocated to Java. >>> > >>> > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC >>> -XX:NewRatio=5" >>> > >>> > It still seems like Solr is running somewhat slowly... Any ideas >>> about speeding things up? >>> > >>> > Our catalog is at catalog.wakegov.com >>> > >>> > -- >>> > Kevin Smith >>> > Digital Library Manager >>> > Wake County Public Libraries >>> > >>> ------------------------------------------------------------------------------ >>> > Don't let slow site performance ruin your business. Deploy New Relic >>> APM >>> > Deploy New Relic app performance management and know exactly >>> > what is happening inside your Ruby, Python, PHP, Java, and .NET app >>> > Try New Relic at no cost today and get our sweet Data Nerd shirt too! >>> > >>> http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ >>> > VuFind-General mailing list >>> > VuF...@li... >>> > https://lists.sourceforge.net/lists/listinfo/vufind-general >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Don't let slow site performance ruin your business. Deploy New Relic APM >>> Deploy New Relic app performance management and know exactly >>> what is happening inside your Ruby, Python, PHP, Java, and .NET app >>> Try New Relic at no cost today and get our sweet Data Nerd shirt too! >>> http://p.sf.net/sfu/newrelic-dev2dev >>> _______________________________________________ >>> VuFind-General mailing list >>> VuF...@li... >>> https://lists.sourceforge.net/lists/listinfo/vufind-general >>> >>> >>> >>> >>> >>> -- >>> Kevin Smith >>> Digital Library Manager >>> Wake County Public Libraries >>> >>> >>> ------------------------------------------------------------------------------ >>> Don't let slow site performance ruin your business. Deploy New Relic APM >>> Deploy New Relic app performance management and know exactly >>> what is happening inside your Ruby, Python, PHP, Java, and .NET app >>> Try New Relic at no cost today and get our sweet Data Nerd shirt too! >>> http://p.sf.net/sfu/newrelic-dev2dev >>> _______________________________________________ >>> VuFind-General mailing list >>> VuF...@li... >>> https://lists.sourceforge.net/lists/listinfo/vufind-general >>> >>> >> > |
From: Demian K. <dem...@vi...> - 2012-10-09 12:51:26
|
Thanks, Greg - I've linked this in the Wiki for future reference. From: Greg Pendlebury [mailto:gre...@gm...] Sent: Tuesday, October 09, 2012 7:33 AM To: Osullivan L. Cc: Tim Fletcher; vuf...@li... Subject: Re: [VuFind-General] Solr seems slow Looking back through this thread I thought I'd also throw in some other unrelated thoughts, since we are currently looking at issues related to GC in detail. Something to consider is information like this: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html I mentioned that specifically in relation to the suggestion to use the system cache because it is a very similar idea. IF you have the RAM on your machine to hold the entire index in memory you might want to think about actually LOWERING your JVM RAM allocation (because it helps with GC times) and enabling MMapDirectory to get the kernel to cache the index in RAM for you. This way you only need to allocate enough RAM for the JVM to to hold data structures relating to things like caches, as opposed to your indexes (which will be rebuilt in memory every time a commit comes through, thus triggering GC). Less stuff inside the JVM equals less GC. If you look at the JVM with a tool like top in this instance it will appear to have a huge virtual memory allocation, but your JVM heap is much lower if connected to jconsole or something similar. I should stress though, that we are still experimenting in this area, so no magic bullets yet. The more promising evolution to the above will be in Solr 4 to start using soft commits and stop scrapping our in-memory data structures so often to further reduce garbage collection. Ta, Greg On 9 October 2012 22:21, Greg Pendlebury <gre...@gm...<mailto:gre...@gm...>> wrote: I'm far from an expert on this stuff, but I believe we are using these tools to load balance in software (ie. not a dedicated hardware load balancer): * http://haproxy.1wt.eu/ (the main one for load balancing) * http://wiki.nginx.org/Main If you want further info I'd have to go ask one of our sysadmins, although I know Mark also loiters around the list and I assume he is intimately familiar with the setup. Ta, Greg On 9 October 2012 20:09, Osullivan L. <L.O...@sw...<mailto:L.O...@sw...>> wrote: Hi Greg, Thanks for this - can you point us in the direction of any documentation regarding load balancing which you think is particularly useful? Cheers, Luke ________________________________ From: Greg Pendlebury [gre...@gm...<mailto:gre...@gm...>] Sent: 08 October 2012 22:16 To: Osullivan L. Cc: Tim Fletcher; vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow You could potentially consider running multiple replicated indexes behind a load balancer and having them restart themselves when GC problems are detected. The load balancer will avoid service disruptions when only one of the indexes is offline. NLA uses this mechanism extensively, and spreading the load out across two machines will also help to reduce the severity of your problems as well. Ta, Greg On 8 October 2012 20:21, Osullivan L. <L.O...@sw...<mailto:L.O...@sw...>> wrote: Hi Tim, We're having similar problems at Swansea - not related to speed but rather to garbage collection. When your Java garbage gets full, vufind will be slower as the system is frantically trying to empty garbage / close down process to free up memory. We use the GC monitoring process described in the VuFind Wiki to restart VuFind when the GC gets full. Unfortunately, this can happen at any time (depending on volume of searches) so we have established a 4 hour restart routine. We restart VuFind at 8am, 12pm and 4pm throughout the day. It's not ideal but it at least avoids restarts during the hour when there may be induction sessions in process. We came to a decision on the precise restart times by monitoring the frequency of garbage collection. At present, during peak times, this averages 5 hours and 17 minutes. We chose a four hour rule to be on the safe side. Cheers, Luke ________________________________ From: Tim Fletcher [T.F...@bb...<mailto:T.F...@bb...>] Sent: 08 October 2012 11:03 To: vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow Hi, We also had a slow down on Friday as use of the system increased and more or less ground to a halt. It came back after a quick restart so I assume that a regular restart will also help? We haven't adjusted any settings so will be doing that - can I confirm that the place to make the initial JAVA_OPTIONS changes is in vufind.sh? Thanks, Tim ---------- Tim Fletcher Birkbeck Library From: Demian Katz [mailto:dem...@vi...<mailto:dem...@vi...>] Sent: 05 October 2012 18:13 To: kevin smith Cc: vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow I'm not aware of a specific recent Java problem that would cause this, but I haven't been following the Solr lists very closely lately, so it's entirely possible your theory is correct -- it sounds plausible even though I have no specific evidence that it is correct. - Demian ________________________________ From: kevin smith [as...@gm...<mailto:as...@gm...>] Sent: Friday, October 05, 2012 1:08 PM To: Demian Katz Cc: Tuan Nguyen; vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow Thanks, I just did all of the package updates for Ubuntu and after this, there is a noticeable improvement in performance. I suspect that there was a problem in the last update to openjdk-6-jre and associated packages, and the new update fixed it. Does this sound plausible? On Fri, Oct 5, 2012 at 6:51 AM, Demian Katz <dem...@vi...<mailto:dem...@vi...>> wrote: Thanks for the tip -- I hadn't heard that one before. I've added it to the wiki (with proper credit, of course): http://vufind.org/wiki/performance#exploiting_the_system_cache - Demian ________________________________________ From: Tuan Nguyen [tu...@yo...<mailto:tu...@yo...>] Sent: Thursday, October 04, 2012 11:09 PM To: kevin smith Cc: vuf...@li...<mailto:vuf...@li...> Subject: Re: [VuFind-General] Solr seems slow If on linux, then you can also try to load your entire index into the system cache. This speeds up searches a lot for us. cat /usr/local/vufind/solr/biblio/index/* > /dev/null On 2012-10-04, at 3:03 PM, kevin smith wrote: > We have switched all search links on our webpage from HIP to VuFind. We are getting much more traffic, and I have noticed some real performance drops. The first problem was my Apache settings, but I am no longer getting the Server seems to be busy Apache errors. > > I have increased the memory allocated to Java. > > JAVA_OPTIONS="-server -d64 -Xms8192m -Xmx8192m -XX:+UseParallelGC -XX:NewRatio=5" > > It still seems like Solr is running somewhat slowly... Any ideas about speeding things up? > > Our catalog is at catalog.wakegov.com<http://catalog.wakegov.com> > > -- > Kevin Smith > Digital Library Manager > Wake County Public Libraries > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev_______________________________________________ > VuFind-General mailing list > VuF...@li...<mailto:VuF...@li...> > https://lists.sourceforge.net/lists/listinfo/vufind-general ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ VuFind-General mailing list VuF...@li...<mailto:VuF...@li...> https://lists.sourceforge.net/lists/listinfo/vufind-general -- Kevin Smith Digital Library Manager Wake County Public Libraries ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ VuFind-General mailing list VuF...@li...<mailto:VuF...@li...> https://lists.sourceforge.net/lists/listinfo/vufind-general |
From: Till K. <kin...@gm...> - 2012-10-09 13:24:15
|
Am 09.10.2012 13:33, schrieb Greg Pendlebury: > Looking back through this thread I thought I'd also throw in some other > unrelated thoughts, since we are currently looking at issues related to > GC in detail. Are you experimenting with different garbage collectors? We got very different results with the different collectors available in (then) SUN Java virtual machines. Which is to be expected because of the different nature of the collection strategies. We finally found that UseConcMarkSweepGC for concurrent collection works best for us. We didn't encounter any major "application halts" (due to garbage cleanup) or Out of Memory exceptions any more. But since we did this two or three years ago with Solr 1.4 and now just continue with the same Java settings for Solr 3.6 (never change a working system), it would be interesting to hear your results... Till |
From: Greg P. <gre...@gm...> - 2012-10-09 20:25:00
|
>> We finally found that UseConcMarkSweepGC Yes, the team here went through the same experimentation just before my time here started and 'UseConcMarkSweepGC' was also the one settled on. Unfortuanately, whilst things improved, the problems have not gone away. Ta, Greg On 10 October 2012 00:24, Till Kinstler <kin...@gm...> wrote: > Am 09.10.2012 13:33, schrieb Greg Pendlebury: > > > Looking back through this thread I thought I'd also throw in some other > > unrelated thoughts, since we are currently looking at issues related to > > GC in detail. > > Are you experimenting with different garbage collectors? We got very > different results with the different collectors available in (then) SUN > Java virtual machines. Which is to be expected because of the different > nature of the collection strategies. We finally found that > UseConcMarkSweepGC for concurrent collection works best for us. We > didn't encounter any major "application halts" (due to garbage cleanup) > or Out of Memory exceptions any more. > But since we did this two or three years ago with Solr 1.4 and now just > continue with the same Java settings for Solr 3.6 (never change a > working system), it would be interesting to hear your results... > > Till > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general > |
From: kevin s. <as...@gm...> - 2012-10-10 19:23:19
|
I just changed a setting in Jetty, and it looks like things have sped up quite a bit. <Set name="headerBufferSize">65536</Set> (This will increase the header limit from the default of 4KB to 64KB.) On Tue, Oct 9, 2012 at 4:24 PM, Greg Pendlebury <gre...@gm...>wrote: > >> We finally found that UseConcMarkSweepGC > > Yes, the team here went through the same experimentation just before my > time here started and 'UseConcMarkSweepGC' was also the one settled on. > Unfortuanately, whilst things improved, the problems have not gone away. > > Ta, > Greg > > > On 10 October 2012 00:24, Till Kinstler <kin...@gm...> wrote: > >> Am 09.10.2012 13:33, schrieb Greg Pendlebury: >> >> > Looking back through this thread I thought I'd also throw in some other >> > unrelated thoughts, since we are currently looking at issues related to >> > GC in detail. >> >> Are you experimenting with different garbage collectors? We got very >> different results with the different collectors available in (then) SUN >> Java virtual machines. Which is to be expected because of the different >> nature of the collection strategies. We finally found that >> UseConcMarkSweepGC for concurrent collection works best for us. We >> didn't encounter any major "application halts" (due to garbage cleanup) >> or Out of Memory exceptions any more. >> But since we did this two or three years ago with Solr 1.4 and now just >> continue with the same Java settings for Solr 3.6 (never change a >> working system), it would be interesting to hear your results... >> >> Till >> >> >> ------------------------------------------------------------------------------ >> Don't let slow site performance ruin your business. Deploy New Relic APM >> Deploy New Relic app performance management and know exactly >> what is happening inside your Ruby, Python, PHP, Java, and .NET app >> Try New Relic at no cost today and get our sweet Data Nerd shirt too! >> http://p.sf.net/sfu/newrelic-dev2dev >> _______________________________________________ >> VuFind-General mailing list >> VuF...@li... >> https://lists.sourceforge.net/lists/listinfo/vufind-general >> > > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general > > -- Kevin Smith Digital Library Manager Wake County Public Libraries |