[Archive-access-discuss] Best practices for indexing a growing 2+ billion document collection

[Kristinn S. <kri...@la...>]

Dear all,

We are planning on updating our Wayback installation and I would like to poll your collective wisdom on the best approach for managing the Wayback index.

Currently, our collection is about 2.2 billion items. It is also growing at a rate of approximately 350-400 million records per year.

The obvious approach would be to use a sorted CDX file (or files) as the index. I'm, however, concerned about its performance at this scale. Additionally, updating a CDX based index can be troublesome. Especially as we would like to update it continuously as new material is ingested.

Any relevant experience and advice you could share on this topic would be greatly appreciated.


Best regards,
Mr. Kristinn Sigurðsson
Head of IT
National and University Library of Iceland




-------------------------------------------------------------------------
Landsbókasafn Íslands - Háskólabókasafn | Arngrímsgötu 3 - 107 Reykjavík
Sími/Tel: +354 5255600 | www.landsbokasafn.is
-------------------------------------------------------------------------
fyrirvari/disclaimer - http://fyrirvari.landsbokasafn.is