Re: [VuFind-General] Several million records?

[Till K. <kin...@gm...>]

Am 21.07.2010 13:32, schrieb re...@gm...:
> Hello.
>
> I'd like to know, whether VuFind works with several million records (30-40 million).

We currently have an index holding around 22 million records.
There are installations of Solr (the search engine that VuFind uses), 
that have far more records (for example some well known aggregated 
search index products in the library business with some 100 million 
records)...
Solr scales very well, you can set up distributed installations in 
different ways to fit different needs etc... I wouldn't expect problems, 
though of course you should think about hardware and architecture of 
your solution depending on use cases...

> In particular, does faceted search works then yet?

Yes, it does.

> Do you have experience with such a big index or maybe do you know someone, who is it using?

As I said: We do with 22 million records...
Not an issue so far. On pretty standard "round the corner" hardware we 
are currently serving around 10 requests per second with peaks around 
30-40 requests/second (on the Solr search backend side, it's less on the 
VuFind user frontend, our customized VuFind installation causes a lot of 
search requests eg. because of autocomplete functionality). It simply 
works...

So, just do it, it will work with 30 or 40 million records. If you 
encounter bottlenecks, look at the Solr side. It's really easy to scale 
Solr for large indexes or high loads or both (at least compared to other 
technology)...

Till

-- 
http://twitter.com/tillk