[Archive-access-discuss] NutchWax Index Merging

[Pope, J. <Jac...@bl...>]

Hiya all,

 

I've created a lot of nutchwax indices deployed the segments and index
for each to the search directory, and got nutchwax/wayback to search
these successfully. 

 

However, when I try to add more than 40 I hit the 'too many open files'
problem I mentioned before. Several people have suggested upping the
'ulimit' to 32678, but I've already got it set to 1024, so upping it to
32768 would theoretically allow me to create 30 x 40 indices, still an
order of magnitude smaller than I need.

 

Next step I've tried is index merging.

 

I've run the IndexMerger over some of my indices successfully, but when
I replace the indexes directory (which contains the individual indices)
with the new index, nutchwax stops working. It tells me that it's found
some hits for my search term, but it doesn't list them, and wayback
claims the index is unavailable. What else do I need to do to deploy a
merged index?

 

Cheers,

 

Jack

 

Jackson Pope

Technical Lead

Web Archiving Team

The British Library

+44 (0)1937 54 6942

**************************************************************************
 
Experience the British Library online at www.bl.uk
 
The British Library's new interactive Annual Report and Accounts 2006/07 : www.bl.uk/mylibrary
 
Help the British Library conserve the world's knowledge. Adopt a Book. www.bl.uk/adoptabook
 
The Library's St Pancras site is WiFi - enabled
 
*************************************************************************
 
The information contained in this e-mail is confidential and may be legally privileged. It is intended for the addressee(s) only. If you are not the intended recipient, please delete this e-mail and notify the pos...@bl... : The contents of this e-mail must not be disclosed or copied without the sender's consent. 
 
The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the British Library. The British Library does not take any responsibility for the views of the author. 
 
*************************************************************************