Re: [VuFind-General] hardware requirements for 7mio records

[Ere M. <ere...@he...>]

Hi,

I'd suggest starting by reading 
<https://wiki.apache.org/solr/SolrPerformanceProblems> if you haven't 
already. While it doesn't give you definitive answers, you'll get an 
overall picture of how the memory is managed and what's useful to check 
out.

The first thing I'd check is whether you actually need 5G memory for 
Solr. Maybe you do, since we run 31G with ~50 million records. In any 
case the memory available for the OS to use for caching, i.e. unused by 
any application, is crucial for good performance. In an optimal 
situation your whole index should fit into the free memory so that the 
OS can cache it. Sometimes you can get decent performance also with 75% 
or 50% or so, but the best bet would be to have enough for the index and 
then a bit. In your case that would be something like 24G (16G + 5G + 
whatever the OS and any other stuff needs). If you can't add enough 
memory, faster disk might help too.

To sum it up: Keep the Java heap as small as possible to leave as much 
memory as possible for the file system cache. But not too small since 
that'd lead to excessive garbage collection and/or out of memory situations.

--Ere

thomas schwaerzler kirjoitti 5.10.2017 klo 15.09:
> hi,
> 
> i just wanted to ask if there is someone also having about 7million
> records to find out what hardware requirements there are.
> 
> we have a vufind instance with about 7million records. the index takes
> about 16G disk space
> 
> java has 5G memory: -Xms4608m -Xmx5120m
> 
> total memory of the machine is 8G
> 
> except solr there is not much running on the machine. its a single slave
> instance connected to a master index.
> 
> 
> currently i have some real slow queries taking 20 seconds or more when
> searching for long strings in all fields.
> 
> after reading some comments on solr and search speed i guess that more
> memory could be a solution.
> 
> can anyone out there with a similar amount of data an faster search
> results show their hardware configuration?
> 
> 
> thanks
> tom.
> 

-- 
Ere Maijala
Kansalliskirjasto / The National Library of Finland