[Archive-access-discuss] Update: Env Setup Help for Large Set of ARCs

[Alex Wu <aw...@sd...>]

Hi Brad,

Thank you for your input. Wanted to give an update on our experience  
with the wayback application within Tomcat.

We tried one setup, where on one machine, we ran 12 instances of the  
wayback application, each in it's own Tomcat container, and gave  
about 2,700 ARC files for each instance. Each Tomcat was allocated  
1GB memory. This was done over the weekend, and over 30,000 ARCs were  
processed.

Another setup was tried on the same machine, where 3 tomcat instances  
were run, each with 6 wayback applications. Each wayback application  
handles 2,700 ARC files. Each tomcat was allocated 1 to 3 GB memory.  
Within the instance of Tomcat with 3GB allocated, the result in about  
48 hours was just over 3,000 ARCs processed. The other two tomcat  
instances are mostly idle, having indexed/merged their respective set  
of ARCs almost completely over the weekend.

We are experimenting with different setups that involve many  
variables, such as the varying size of ARC files, non-wayback load on  
the machine, etc., so it's difficult to give a more accurate  
performance comparison without controlling the variables more.

We modified slightly org/archive/wayback/cdx/indexer/ 
IndexPipeline.class so that the indexing, queuing, and merging are  
running in separate threads, and sleeping at different intervals. And  
with this, 3 indexing threads are running.

Lastly, I was not able to view the CVS at http:// 
crawltools.archive.org:8080/cruisecontrol/buildresults/HEAD-archive- 
access. "Firefox can't establish a connection to the server at  
crawltools.archive.org:8080."

Thank you again,
Alex


> Hi,
>
> We have a project with about 48000 ARC files, and would like inputs  
> on the best way to implement the wayback machine 0.6.0
>
> Our setup is Tomcat 5.5.17, JDK 1.5, 1GB memory for JVM. We have  
> only 6000 ARCs indexed at this point over a 1 week period. We would  
> like to increase this rate significantly.
>
>
> Some questions we have are:
>
> 1. Suggested environment setup for this number of ARC files and  
> greater.
>
>
Your current setup should be fine for this, but when the distributed  
index option is available, it would be advisable to move to this  
configuration.


> 2. Parallel indexing option for the current version or additional  
> tools that will allow for this.
>
>

The pipeline-client command line tool has a new option to generate a  
flat-file version of the index data on STDOUT. This process could be  
executed in parallel across multiple nodes, and their outputs sorted,  
and merged together to form a single flat-file. This flat-file can be  
used today with the BDBJE option, by manually placing the file into  
the "toBeMerged" directory on the host holding the index. We've seen  
acceptable performance inserting large sorted files in this manner.

With the new flat-file binary searching ResourceIndex code, this  
sorted flat-file could be used as-is, bypassing the BDBJE altogether.  
I'll let you know when it's checked in.

> 3. The index is tied to the machine name. How to avoid this.
>
>

Not sure what you mean. Do you mean there is data internal to the  
BDBJE that is aware of the host where it was created and cannot be  
used on other hosts? Can you elaborate?


> 4. Is it possible to have multiple wayback installations, each with  
> its own JVM, use the same arc files and/or index.
>
>
Yes. We have a couple of installations that include front end UIs for  
Proxy, Timeline, and Archival URL replay modes on top of the same  
index, where each installation uses a RemoteCDXIndex. I'll add some  
documentation to the User Manual outlining this configuration in the  
next day or two.



> 5. The user manual at http://archive-access.sourceforge.net/ 
> projects/wayback/user_manual.html mentions a non- 
> LocalBDBResourceIndex resource implementation that communicates  
> with a remote wayback installation. The user manual does not cover  
> the preparation of the index data. What are the steps for this  
> setup, including index data preparation.
>
>
As mentioned in #4, I'll outline this configuration in the User  
Manual, but the basics: set up one webapp with a  
LocalBDBResourceIndex, making sure it has a QueryUI with the  
QueryXMLUI jsps set up. This will allow HTTP-XML queries of the  
index. Then you set up one or more webapps, using whatever replay  
modes you prefer, using the RemoteCDXIndex ResourceIndex  
implementation to connect to the HTTP-XML exported ResourceIndex.


> 6. Is there a limitation to the number of ARCs wayback will handle.
>
>
With the 0.8.0 features, we expect the WM to be able to scale to  
arbitrarily large numbers of ARC files. Generating indexes for larger  
installations will be handled offline, and will be a manual process  
until the 1.0.0 release.

Thanks for the feedback and questions. We're very interested in your  
experiences and making this software as easy to use as possible.