Re: [Archive-access-discuss] Indexing and searching

[Michael S. <st...@ar...>]

Natalia Torres wrote:
> Hello
>
> I tried to add the new job moving the indexes directory before starting 
> index process and it works fine. Thanks!!
>
> So, every time I want to index a new job I need to move indexes 
> directory? If I move this directory the nuch wax search still working?
>   

I presume you are using the 'all' command each time?  It will complain 
if there are already indices in place from a previous run.

The 'all' command is a convenience.  It assumes you want to do a 
single-pass indexing of a set of ARCs.   Running the 'all' command to 
bring in a new set of ARCs will run through all steps and index all the 
new additions as well as reindex all ARCs added previously.

Sounds like you want to do incremental updates to your index.  
Experiment by calling the jobs that comprise the 'all' command 
individually.  For example, run the import passing it a directory that 
contains a file that points to just the new ARCs you want to ingest.   
Then do 'update' and 'invert'.  Next run indexing just of the segments 
that were added by the ingest step.  Save aside the indexes made 
previously first.  Run your deduplication.   Finally merge the new 
indices and the old.

I'm working currently on tools and documentation to better support 
incremental updates to indices.  They'll form core of next release 
(Coming soon -- month or so).


> This proces takes many hours ...
>
>   
Yes.  It can.  Depends on number of ARCs you have.  Sounds like too that 
you are running in the standalone mode.  You might consider starting a 
small hadoop cluster.  That should improve your throughput.

Yours,
St.Ack

> Natalia
>
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Archive-access-discuss mailing list
> Arc...@li...
> https://lists.sourceforge.net/lists/listinfo/archive-access-discuss
>