Thread: [Archive-access-cvs] SF.net SVN: archive-access:[2532] trunk/archive-access/projects/wayback/ dist/

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Revision: 2532
          http://archive-access.svn.sourceforge.net/archive-access/?rev=2532&view=rev
Author:   bradtofel
Date:     2008-08-09 02:36:45 +0000 (Sat, 09 Aug 2008)

Log Message:
-----------
DOC update for 1.4 features.

Modified Paths:
--------------
    trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml

Added Paths:
-----------
    trunk/archive-access/projects/wayback/dist/src/site/xdoc/resource_store.xml

Modified: trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml
===================================================================

--- trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml	2008-08-09 01:20:56 UTC (rev 2531)
+++ trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml	2008-08-09 02:36:45 UTC (rev 2532)
@@ -53,7 +53,7 @@
 	      <p>
 	        Once you have downloaded the .tar.gz file from 
 	        sourceforge, you will need to unpack the file to access the
-	        webapp file, <b>wayback.war</b>.
+	        webapp file, <b>wayback-webapp-1.4.0.war</b>.
 	      </p>
 		    <p>
 	        Installation and configuration of this software involves the
@@ -110,177 +110,24 @@
 	class=&quot;org.archive.wayback.webapp.WaybackCollection&quot;&gt;
 	&lt;property name=&quot;resourceStore&quot; ... /&gt;
 	&lt;property name=&quot;resourceIndex&quot; ... /&gt;
+	&lt;property name=&quot;shutdownables&quot; ... /&gt;
 &lt;/bean&gt;
 
         </pre>
       </p>
       <p>
-        The resourceStore property refers to a bean implementing org.archive.wayback.ResourceStore.
+        The resourceStore property refers to a bean implementing 
+        <a href="resource_store.html">org.archive.wayback.ResourceStore</a>.
       </p>
       <p>
-        The resourceIndex property refers to a bean implementing org.archive.wayback.ResourceIndex.
+        The resourceIndex property refers to a bean implementing 
+        <a href="resource_index.html">org.archive.wayback.ResourceIndex</a>.
       </p>
-		</section>
+      <p>
+        The shutdownables property refers to a list of beans implementing org.archive.wayback.Shutdownable, typically worker Threads performing automatic updates of the Collection.
+      </p>
+	</section>
 
-
-
-    <section name="org.archive.wayback.ResourceStore implementations">
-
-
-      <subsection name="LocalResourceStore">
-	      <p>
-					This implementation works well for small
-					collections, where all the ARC/WARC files can be placed in a single 
-					directory on the same computer running the wayback application.
-					Using NFS or another network filesystem technology and symbolic 
-					links can allow this implementation to deal with files in
-					multiple directories, or across multiple storage nodes. This
-					implementation also includes the capability to run a background
-					thread to automatically notice new ARC/WARC files appearing, index
-					those files, and hand off the index data for merging with
-					a BDBResourceIndex. When using automatic indexing, any files added to
-					the 'dataDir' will automatically be indexed and queued for merging 
-					with the ResourceIndex. Please see documentation for the 
-					BDBResourceIndex for information on configuring automatic merging of
-					indexed data with a BDBResourceIndex.
-        </p>
-        <p>
-          The XML configuration template for a LocalResourceStore follows:
-          <pre>
-
-&lt;property name="resourceStore"&gt;
-  &lt;bean class="org.archive.wayback.resourcestore.LocalResourceStore"
-    init-method="init"&gt;
-
-    &lt;property name="dataDir" value="/tmp/wayback/arcs/" /&gt;
-
-    &lt;property name="indexThread"&gt;
-      &lt;bean class="org.archive.wayback.resourcestore.AutoIndexThread"&gt;
-        &lt;property name="queuedDir" value="/tmp/wayback/arc-indexer/queued" /&gt;
-        &lt;property name="workDir" value="/tmp/wayback/arc-indexer/work" /&gt;
-        &lt;property name="runInterval" value="10000" /&gt;
-        &lt;property name="indexClient"&gt;
-          &lt;bean class="org.archive.wayback.resourceindex.indexer.IndexClient"&gt;
-            &lt;property name="tmpDir" value="/tmp/wayback/arc-indexer/tmp" /&gt;
-            &lt;property name="target" value="/tmp/wayback/index-data/incoming" /&gt;
-          &lt;/bean&gt;
-        &lt;/property&gt;
-      &lt;/bean&gt;
-    &lt;/property&gt;
-  &lt;/bean&gt;
-&lt;/property&gt;
-
-		      </pre>
-        </p>
-        <p>
-          Required configuration:
-          <ul>
-            <li>
-              <b>
-                dataDir
-              </b>
-              is the local directory where ARC files will be 
-              located.
-            </li>
-          </ul>
-        </p>
-        <p>
-          Optional configuration (only needed if the indexThread property-bean
-          is specified, for automatic indexing)
-          <ul>
-            <li>
-              <b>
-                queuedDir
-              </b>
-              names a local directory where the indexer will maintain state
-              about ARC files that have already been indexed.
-            </li>
-            <li>
-              <b>
-                workDir
-              </b>
-              names a local directory where the indexer will maintain state
-              about ARC files that are about to be indexed.
-            </li>
-            <li>
-              <b>
-                runInterval
-              </b>
-              indicates the number of milliseconds between polling arcDir
-              for newly created ARC files. Default is 10000.
-            </li>
-            <li>
-              <b>
-                tmpDir
-              </b>
-              names a local directory where index data will be stored
-              temporarily before handing off to <b>target</b>.
-            </li>
-            <li>
-              <b>
-                target
-              </b>
-              names:
-              <ol>
-                <li>
-                  a local directory where an BDBIndexUpdater is configured to
-                  look for new index data to be merged with a BDBIndex.
-                </li>
-                <li>
-                  a remote http:// URL where index data should be PUT, for 
-                  merging with a remote BDBIndex.
-                </li>
-              </ol>
-            </li>
-          </ul>
-        </p>
-        <p>
-          <b>Note:</b> upgrading from Wayback 1.0 to 1.2 requires changing
-          ResourceStore implementations from <b>LocalARCResourceStore</b> to
-          <b>LocalResourceStore</b>. <b>LocalARCResourceStore</b> is now
-          deprecated.
-        </p>
-      </subsection>
-
-
-      <subsection name="Http11ResourceStore">
-        <p>
-	        This implementation allows the wayback application to access 
-	        documents in remote ARC/WARC files via HTTP 1.1, and scales to
-	        millions of ARC/WARC files.
-        </p>
-	      <p>
-	        The XML configuration template for an Http11ResourceStore follows:
-	        <pre>
-
-&lt;property name="resourceStore"&gt;
-  &lt;bean class="org.archive.wayback.resourcestore.Http11ResourceStore"&gt;
-    &lt;property name="urlPrefix" value="http://localhost:8080/arcproxy/" /&gt;
-  &lt;/bean&gt;
-&lt;/property&gt;
-
-	        </pre>
-	      </p>
-	      <p>
-	        Required configuration:
-	        <ul>
-	          <li>
-	            <b>
-	              urlPrefix
-	            </b>
-	            this is the http:// prefix where ARC/WARC files are exported with
-	            an ArcProxy installation. See elsewhere in this document for 
-	            information about setting up an ArcProxy.
-	          </li>
-	        </ul>
-	      </p>
-	    </subsection>
-
-
-	  </section>
-
-
-
     <section name="org.archive.wayback.ResourceIndex implementations">
 
 

Added: trunk/archive-access/projects/wayback/dist/src/site/xdoc/resource_store.xml
===================================================================
--- trunk/archive-access/projects/wayback/dist/src/site/xdoc/resource_store.xml	                        (rev 0)
+++ trunk/archive-access/projects/wayback/dist/src/site/xdoc/resource_store.xml	2008-08-09 02:36:45 UTC (rev 2532)
@@ -0,0 +1,143 @@
+<?xml version="1.0" encoding="ISO-8859-1"?>
+<document>
+  <properties>
+    <title>Resource Store Configuration</title>
+    <author email="brad at archive dot org">Brad Tofel</author>
+    <revision>$$Id$$</revision>
+  </properties>
+  
+  <body>
+    <section name="ResourceStore configuration options">
+      <subsection name="FileLocationDB">
+        <p>
+          The Location Database provides a mapping between ARC/WARC file names 
+          and the absolution location of those ARC/WARC files. Absolute 
+          location, in this case, can refer to either HTTP URLs or absolute 
+          paths to files on the local file system.
+        </p>
+        <p>
+          Whenever locations are added for a new filename that was not 
+          previously present in the location database, a record (in this case a
+          line) is added to a log file. This log file can then be used to 
+          determine which files have been seen by the location database. The
+          ResourceFileLocationDatabase interface includes methods to retrieve
+          the current length of this log file, and to return an iterator with
+          all records between two points in the log. This interface allows an
+          observer to poll the location database to create events when new files
+          are added to the underlying database. 
+        </p>
+      </subsection>
+      <subsection name="Automatic Indexing Components">
+          <p>
+            Wayback includes 5 Thread/Worker classes to enable automatic indexing
+            of new content:
+            <img src="images/AutoIndexing.png" />
+            <ul>
+              <li>
+                <b>ResourceFileSourceUpdater</b> is responsible for repeatedly
+                scanning one or more ResourceFileSource instances, creating 
+                manifests of the files seen in each, and handing the manifests
+                off to the ResourceFileLocationDBUpdater. In the future, for
+                larger installations, with 100s to 1000s of machines holding
+                ARC/WARC files, multiple instances of this component may run in
+                parallel.
+              </li>
+                <li>
+                <b>ResourceFileLocationDBUpdater</b> is responsible for noticing
+                new manifests appearing in an incoming directory, and merging
+                the contents of those manifests with the actual location database,
+                which is currently implemented using a BDBJE database.
+              </li>
+                <li>
+                <b>IndexQueueUpdater</b> is responsible for polling the location
+                database log, and adding newly discovered ARC/WARC files to the
+                IndexQueue. 
+              </li>
+                <li>
+                <b>IndexWorker</b> is responsible for polling the IndexQueue, and
+                when file names are present in the queue, creating an index of
+                all resources in the ARC/WARC file, and handing the results to
+                the LocalResourceIndexUpdater. In the future, for larger
+                installations, multiple instances of this component may run in
+                parallel on multiple hosts, or this entire component may be 
+                replaced by a distributed Hadoop indexing implementation.
+              </li>
+              <li>
+                <b>LocalResourceIndexUpdater</b> is responsible for noticing new
+                index result files appearing in an incoming directory, and merging
+                those results with an existing LocalResourceIndex. Currently the
+                only provided LocalResourceIndex that can be updated based on an
+                underlying BDBJE database, but future implementation may maintain
+                a set of sorted CDX files, or a combination of CDX files and a 
+                BDBJE database. 
+              </li>
+            </ul>
+          </p>
+      </subsection>
+    </section>
+
+    <section name="org.archive.wayback.ResourceStore implementations">
+      <p>
+        Wayback allows for several configurations enabling diverse collection
+        sizes and distribution of ARC/WARC files across many local directories
+        or across many servers. For most configurations, the default
+        LocationDBResourceStore will suffice, but Wayback is distributed with
+        2 additional classes, FileProxy and SimpleResourceStore, which
+        provide an opportunity to insert a single HTTP caching server between
+        the Wayback service and an ARC/WARC storage cluster.
+      </p>
+
+      <subsection name="LocationDBResourceStore">
+        <p>
+          This implementation uses a LocationDB to convert ARC/WARC filenames
+          into absolute paths, or HTTP URLs. The underlying LocationDB can be
+          managed by the automatic indexing threads as described above, or it
+          can be manually managed with the <i>location-client</i> command line
+          tool. Be sure to enable the 
+          org.archive.wayback.resourcestore.locationdb.FileProxyServlet
+          if you plan to manage the LocationDB manually.
+        </p>
+      </subsection>
+      <subsection name="SimpleResourceStore">
+      	<p>
+      	  This configuration depends on all ARC/WARC files appearing within a
+      	  single HTTP 1.1 exported root directory, or within a single local
+      	  directory. ARC/WARC file names are appended to a common prefix, either
+      	  a local directory on the host running Wayback, or under a single 
+      	  remote directory.
+      	</p>
+      	<p>
+          The FileProxyServlet can be used to make all ARC/WARC files accessible
+          within a single HTTP directory, acting as a reverse proxy to the 
+          actual host holding the ARC/WARC files. The FileProxyServlet uses a
+          LocationDB to translate requested ARC/WARC filenames into the actual
+          location of each file.
+      	</p>
+      </subsection> 
+    </section>
+    <section name="Telling Wayback where to look for your ARC/WARC files">
+      <p>
+        When using the automatic indexing functionality, you need to provide a
+        list of ResourceFileSource objects to the ResourceFileSourceUpdater
+        class. Wayback currently contains 2 ResourceFileSource implementations:
+        <ul>
+          <li>
+            <b>DirectoryResourceFileSource</b> will recursively scan a local
+            directory for ARC/WARC files (ending with: .arc, .arc.gz, .warc,
+            or .warc.gz). The 'name' property of each
+            DirectoryResourceFileSource must be unique, and consist of valid
+            filename characters.
+          </li>
+          <li>
+            <b>JspUrlResourceFileSource</b> is a highly experimental 
+            implementation which executes a local .jsp file, passing the 'url'
+            parameter to the .jsp. The local .jsp is expected to produce output
+            of the form (NAME URL) for all ARC/WARC files appearing under the
+            argument url prefix, presumably by parsing the directory index HTML
+            from the server hosting 'url'.
+          </li>
+        </ul>
+      </p>
+    </section>
+  </body>
+</document>
\ No newline at end of file


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.




Thread: [Archive-access-cvs] SF.net SVN: archive-access:[2532] trunk/archive-access/projects/wayback/ dist/

archive-access-cvs