|
From: Alex Wu <aw...@sd...> - 2006-10-10 19:11:16
|
Hi Brad, Thank you for your input. Wanted to give an update on our experience with the wayback application within Tomcat. We tried one setup, where on one machine, we ran 12 instances of the wayback application, each in it's own Tomcat container, and gave about 2,700 ARC files for each instance. Each Tomcat was allocated 1GB memory. This was done over the weekend, and over 30,000 ARCs were processed. Another setup was tried on the same machine, where 3 tomcat instances were run, each with 6 wayback applications. Each wayback application handles 2,700 ARC files. Each tomcat was allocated 1 to 3 GB memory. Within the instance of Tomcat with 3GB allocated, the result in about 48 hours was just over 3,000 ARCs processed. The other two tomcat instances are mostly idle, having indexed/merged their respective set of ARCs almost completely over the weekend. We are experimenting with different setups that involve many variables, such as the varying size of ARC files, non-wayback load on the machine, etc., so it's difficult to give a more accurate performance comparison without controlling the variables more. We modified slightly org/archive/wayback/cdx/indexer/ IndexPipeline.class so that the indexing, queuing, and merging are running in separate threads, and sleeping at different intervals. And with this, 3 indexing threads are running. Lastly, I was not able to view the CVS at http:// crawltools.archive.org:8080/cruisecontrol/buildresults/HEAD-archive- access. "Firefox can't establish a connection to the server at crawltools.archive.org:8080." Thank you again, Alex > Hi, > > We have a project with about 48000 ARC files, and would like inputs > on the best way to implement the wayback machine 0.6.0 > > Our setup is Tomcat 5.5.17, JDK 1.5, 1GB memory for JVM. We have > only 6000 ARCs indexed at this point over a 1 week period. We would > like to increase this rate significantly. > > > Some questions we have are: > > 1. Suggested environment setup for this number of ARC files and > greater. > > Your current setup should be fine for this, but when the distributed index option is available, it would be advisable to move to this configuration. > 2. Parallel indexing option for the current version or additional > tools that will allow for this. > > The pipeline-client command line tool has a new option to generate a flat-file version of the index data on STDOUT. This process could be executed in parallel across multiple nodes, and their outputs sorted, and merged together to form a single flat-file. This flat-file can be used today with the BDBJE option, by manually placing the file into the "toBeMerged" directory on the host holding the index. We've seen acceptable performance inserting large sorted files in this manner. With the new flat-file binary searching ResourceIndex code, this sorted flat-file could be used as-is, bypassing the BDBJE altogether. I'll let you know when it's checked in. > 3. The index is tied to the machine name. How to avoid this. > > Not sure what you mean. Do you mean there is data internal to the BDBJE that is aware of the host where it was created and cannot be used on other hosts? Can you elaborate? > 4. Is it possible to have multiple wayback installations, each with > its own JVM, use the same arc files and/or index. > > Yes. We have a couple of installations that include front end UIs for Proxy, Timeline, and Archival URL replay modes on top of the same index, where each installation uses a RemoteCDXIndex. I'll add some documentation to the User Manual outlining this configuration in the next day or two. > 5. The user manual at http://archive-access.sourceforge.net/ > projects/wayback/user_manual.html mentions a non- > LocalBDBResourceIndex resource implementation that communicates > with a remote wayback installation. The user manual does not cover > the preparation of the index data. What are the steps for this > setup, including index data preparation. > > As mentioned in #4, I'll outline this configuration in the User Manual, but the basics: set up one webapp with a LocalBDBResourceIndex, making sure it has a QueryUI with the QueryXMLUI jsps set up. This will allow HTTP-XML queries of the index. Then you set up one or more webapps, using whatever replay modes you prefer, using the RemoteCDXIndex ResourceIndex implementation to connect to the HTTP-XML exported ResourceIndex. > 6. Is there a limitation to the number of ARCs wayback will handle. > > With the 0.8.0 features, we expect the WM to be able to scale to arbitrarily large numbers of ARC files. Generating indexes for larger installations will be handled offline, and will be a manual process until the 1.0.0 release. Thanks for the feedback and questions. We're very interested in your experiences and making this software as easy to use as possible. |