Re: [VuFind-Tech] excessive commits / autowarming

[Barnett, J. <jef...@ya...>]

Solaris 10 on a five year old v880 (Sparc) 6 cpu, 16GB ram.
JDK 1.5.0_7

While indexing never use more that 20% of one cpu and 6GB memory, 0 swapping.

We used to get about 400K/hr, but never had more than 2 million loaded at a time.  Time became a problem when we went above that number and/or when we upgraded beyond rev 680 (current rev is 759).

I doubt the hardware or OS is the culprit.  The same configuration runs our production Voyager system for 600 staff and 20000+ faculty and students.  We also run a staff-only Tomcat server on a smaller machine.  Somewhere we just have a bad choice/default of parameters.

-----Original Message-----
From: Wayne Graham [mailto:ws...@wm...]
Sent: Monday, July 07, 2008 12:20 PM
To: Barnett, Jeffrey
Cc: vuf...@li...
Subject: Re: excessive commits / autowarming

Are you running on Slowlaris, er, Solaris? Which JDK are you using? What
kind of processor?

A million records really shouldn't take that long. I'm doing 2 million
on my  desktop in about 90 minutes.

Wayne

Barnett, Jeffrey wrote:
> Thanks again.  I'm going to give all of these a try on my next batch of a million records (current batch still loading after 19 hours) and report back on the results.
>
> -----Original Message-----
> From: Wayne Graham [mailto:ws...@wm...]
> Sent: Monday, July 07, 2008 11:55 AM
> To: Barnett, Jeffrey
> Cc: vuf...@li...
> Subject: Re: excessive commits / autowarming
>
> I'm still going through the new stuff in Solr, but there are some
> interesting new elements.
>
> ramBufferedSizeMB (can be set with maxBufferedDocs) If both are set, the
> first one reached triggers a flush. The default is 32MB. You may also
> want to look at the httpCaching section if you are using caching, also
> if you're behind a load balancer, check out the healthcheck section
> (uncomment the server-eneabled section).
>
> Wayne
>
> Barnett, Jeffrey wrote:
>
>> Great, thanks.
>>
>> Are there any other solrconfig tweaks you migh recommend for building large indices?  I've scanned and learned from the solr wiki you pointed to earlier (thanks for that too), but I'm thinking there might be specific things about the vufind environment that might make special case logic apply.  For one thing, I'm doing all of this offline, with no simultaneous query activity.
>>
>> -----Original Message-----
>> From: Wayne Graham [mailto:ws...@wm...]
>> Sent: Monday, July 07, 2008 9:58 AM
>> To: Barnett, Jeffrey
>> Cc: vuf...@li...
>> Subject: Re: excessive commits / autowarming
>>
>> In solrconfig.xml:
>>
>> ...
>> <maxTime>20000</maxTime>
>> ...
>>
>> Barnett, Jeffrey wrote:
>>
>>
>>> I'm glad to hear this is tuneable.  If the change isn't too complex (I assume a solrconfig parameter), could you post it separately (or send me a note) so that it can be customized on a site by site (or even run by run) basis?
>>>
>>> -----Original Message-----
>>> From: Wayne Graham [mailto:ws...@wm...]
>>> Sent: Monday, July 07, 2008 9:31 AM
>>> To: Barnett, Jeffrey
>>> Cc: vuf...@li...
>>> Subject: Re: excessive commits / autowarming
>>>
>>> This is a "feature" of Solr 1.3. In 1.2, you only set a maxDocs for
>>> autocommits in the update handler. This behavior changed a bit in 1.3 to
>>> also include a maxTime element. Essentially, if one of the two criteria
>>> are met, a commit occurs (which you're seeing).  I just committed a
>>> change that will extend this to 20 seconds or 10,000 records.
>>>
>>> Wayne
>>>
>>> Barnett, Jeffrey wrote:
>>>
>>>
>>>
>>>> On closer examination of a longer log snippet, the better question might be Why is MarcImporter opening closing and autowarming Searchers all over the place?
>>>> ....
>>>>
>>>>
>>>>
>> --
>> /**
>>   * Wayne Graham
>>   * Earl Gregg Swem Library
>>   * PO Box 8794
>>   * Williamsburg, VA 23188
>>   * 757.221.3112
>>   * http://swem.wm.edu/blogs/waynegraham/
>>   */
>>
>>
>>
>>
>
> --
> /**
>   * Wayne Graham
>   * Earl Gregg Swem Library
>   * PO Box 8794
>   * Williamsburg, VA 23188
>   * 757.221.3112
>   * http://swem.wm.edu/blogs/waynegraham/
>   */
>
>
>

--
/**
  * Wayne Graham
  * Earl Gregg Swem Library
  * PO Box 8794
  * Williamsburg, VA 23188
  * 757.221.3112
  * http://swem.wm.edu/blogs/waynegraham/
  */