Re: [Archive-access-discuss] Env Setup Help for Large Set of ARCs

[Alex Wu <aw...@sd...>]

Hi Brad,

Another update. Thank you for the input.

We modified org.archive.wayback.cdx.RemoteCDXIndex.java to search  
multiple remote indexes. One wayback instance (the frontend) has been  
configured to use RemoteCDX in the web.xml and the other wayback  
instances are using LocalBDB configuration.


[code block]
private String sdscSearchUrlBases[] = {"http://machine:9000/wayback/ 
xmlquery", "http://machine:9002/wayback/xmlquery", ...}

public SearchResults query(WaybackRequest wbRequest) {
...
       SearchResults searchResults = new SearchResults();
                 for(int i=0;i<sdscSearchUrlBases.length;i++) {
                         try {
				doc = queryOneUrl(sdscSearchUrlBases[i], wbRequest);
       ...
       }
...
}
[end code block]







On Oct 18, 2006, at 2:58 PM, Brad Tofel wrote:

> Sorry for the delayed response -- lots of balls in the air right now..
>
> I just did a large check in of the software the supports sorted  
> flat files, but will not have time to update the docs for another  
> week or so. There are some comments in the code and in the new  
> web.xml, (which has changed pretty significantly) that might be  
> enough to make sense of how to use the new functionality.
>
>
> Currently, there is no support for querying multiple remote  
> indexes, but it seems like this should be relatively  
> straightforward, in the good-guy case, using the new software:  
> you'd just need to make a RemoteSearchResultSource out of the  
> RemoteResourceIndex, and modify the SearchResultSourceFactory to  
> build a composite from several of them...
>
> I say "in the good-guy case" because the failure modes might get  
> complicated in terms of timeouts, failed connections, etc. However,  
> if your hardware is stable, then the "easy solution" I outlined  
> might be good enough.
>
> I'll drop you another line when the documentation has been updated.
>
> Brad
>
> Bing Zhu wrote:
>> Dear Mr. Brad,
>>
>> This is Bing Zhu from University of California: San Diego.
>>
>> We really appreciate your time to put the answers for our questions.
>>
>> Is it possible for a Wayback machine to query multiple index  
>> sources (e.g. index info
>> in multiple Wayback machines) when using RemoteCDXIndex ? If yes,  
>> would you
>> let us how to do so? Many thanks.
>>
>> Sincerely,
>> Bing
>>
>>
>>
>>

Alex Wu
858-534-5074