Re: [VuFind-Tech] Indexing Speed

[Greg P. <Gre...@us...>]

Oops, forgot to mention. The more I read on JVM tuning it's become apparent that 'optimum' settings can vary wildly. Your JVM has certain defaults based on your OS.

eg. I found out that Solaris defaults to turning on the server flag and parallel garbage collection. Both of these were settings I expected to make a difference... but they were already on by default.

This means of course that settings that made no (apparent) difference for me could have a dramatic impact for others.


Greg Pendlebury
Electronic Services Officer (Systems Team)
Division of Academic Information Services
University of Southern Queensland
Phone: +61 7 4631 1501
Fax: +61 7 4631 1841



________________________________
From: Greg Pendlebury [mailto:Gre...@us...]
Sent: Thursday, 10 September 2009 8:38 AM
To: vuf...@li...
Subject: [VuFind-Tech] Indexing Speed

I've attached a document I've been puttering over for a while on optimizing our index process through JVM and solrconfig settings. We have a small enough collection that I'm able to fully re-index it twice a day, which certainly helps.

I'm no guru on either matter (java or solr), so I'm happy to be corrected on anything contained inside. If the numbers help someone I'm happy.

I am also keenly aware that some of the differences (eg. 3%) don't represent a statistically significant difference, but the overall decrease from the beginning to the end of our testing is good enough for us. We shaved 10% off the indexing times on our test system whilst at the same time pushing segment merges to retain a single segment on the disk (should improve search times), which should have increased times, so the real gain is better then 10%.

When I find some quiet time I'm looking forward to playing with the work several on the list have done with JMeter and doing something similar for the front-end searching JVM.


Greg Pendlebury
Electronic Services Officer (Systems Team)
Division of Academic Information Services
University of Southern Queensland
Phone: +61 7 4631 1501
Fax: +61 7 4631 1841


________________________________
This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government (CRICOS Institution Code No's. QLD 00244B / NSW 02225M)

This email (including any attached files) is confidential and is for the
intended recipient(s) only.  If you received this email by mistake,
please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily
reflect those of the University of Southern Queensland.  Although all
reasonable precautions were taken to ensure that this email contained no
viruses at the time it was sent we accept no liability for any losses
arising from its receipt.

The University of Southern Queensland is a registered provider of
education with the Australian Government (CRICOS Institution Code No's.
QLD 00244B / NSW 02225M)