|
From: Pope, J. <Jac...@bl...> - 2008-02-08 11:38:04
|
Hiya all, I've created a lot of nutchwax indices deployed the segments and index for each to the search directory, and got nutchwax/wayback to search these successfully. However, when I try to add more than 40 I hit the 'too many open files' problem I mentioned before. Several people have suggested upping the 'ulimit' to 32678, but I've already got it set to 1024, so upping it to 32768 would theoretically allow me to create 30 x 40 indices, still an order of magnitude smaller than I need. Next step I've tried is index merging. I've run the IndexMerger over some of my indices successfully, but when I replace the indexes directory (which contains the individual indices) with the new index, nutchwax stops working. It tells me that it's found some hits for my search term, but it doesn't list them, and wayback claims the index is unavailable. What else do I need to do to deploy a merged index? Cheers, Jack Jackson Pope Technical Lead Web Archiving Team The British Library +44 (0)1937 54 6942 ************************************************************************** Experience the British Library online at www.bl.uk The British Library's new interactive Annual Report and Accounts 2006/07 : www.bl.uk/mylibrary Help the British Library conserve the world's knowledge. Adopt a Book. www.bl.uk/adoptabook The Library's St Pancras site is WiFi - enabled ************************************************************************* The information contained in this e-mail is confidential and may be legally privileged. It is intended for the addressee(s) only. If you are not the intended recipient, please delete this e-mail and notify the pos...@bl... : The contents of this e-mail must not be disclosed or copied without the sender's consent. The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the British Library. The British Library does not take any responsibility for the views of the author. ************************************************************************* |