Menu
▾
▴
archive-access-cvs — CVS commits
You can subscribe to this list here.
| 2005 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
(10) |
Sep
(36) |
Oct
(339) |
Nov
(103) |
Dec
(152) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2006 |
Jan
(141) |
Feb
(102) |
Mar
(125) |
Apr
(203) |
May
(57) |
Jun
(30) |
Jul
(139) |
Aug
(46) |
Sep
(64) |
Oct
(105) |
Nov
(34) |
Dec
(162) |
| 2007 |
Jan
(81) |
Feb
(57) |
Mar
(141) |
Apr
(72) |
May
(9) |
Jun
(1) |
Jul
(144) |
Aug
(88) |
Sep
(40) |
Oct
(43) |
Nov
(34) |
Dec
(20) |
| 2008 |
Jan
(44) |
Feb
(45) |
Mar
(16) |
Apr
(36) |
May
(8) |
Jun
(77) |
Jul
(177) |
Aug
(66) |
Sep
(8) |
Oct
(33) |
Nov
(13) |
Dec
(37) |
| 2009 |
Jan
(2) |
Feb
(5) |
Mar
(8) |
Apr
|
May
(36) |
Jun
(19) |
Jul
(46) |
Aug
(8) |
Sep
(1) |
Oct
(66) |
Nov
(61) |
Dec
(10) |
| 2010 |
Jan
(13) |
Feb
(16) |
Mar
(38) |
Apr
(76) |
May
(47) |
Jun
(32) |
Jul
(35) |
Aug
(45) |
Sep
(20) |
Oct
(61) |
Nov
(24) |
Dec
(16) |
| 2011 |
Jan
(22) |
Feb
(34) |
Mar
(11) |
Apr
(8) |
May
(24) |
Jun
(23) |
Jul
(11) |
Aug
(42) |
Sep
(81) |
Oct
(48) |
Nov
(21) |
Dec
(20) |
| 2012 |
Jan
(30) |
Feb
(25) |
Mar
(4) |
Apr
(6) |
May
(1) |
Jun
(5) |
Jul
(5) |
Aug
(8) |
Sep
(6) |
Oct
(6) |
Nov
|
Dec
|
|
From: <bra...@us...> - 2008-07-14 01:18:55
|
Revision: 2430
http://archive-access.svn.sourceforge.net/archive-access/?rev=2430&view=rev
Author: bradtofel
Date: 2008-07-13 18:19:00 -0700 (Sun, 13 Jul 2008)
Log Message:
-----------
Removed Paths:
-------------
branches/wayback-1_2_2/wayback-1_2_1/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-14 01:18:33
|
Revision: 2429
http://archive-access.svn.sourceforge.net/archive-access/?rev=2429&view=rev
Author: bradtofel
Date: 2008-07-13 18:18:43 -0700 (Sun, 13 Jul 2008)
Log Message:
-----------
Added Paths:
-----------
branches/wayback-1_2_2/wayback-1_2_1/
Copied: branches/wayback-1_2_2/wayback-1_2_1 (from rev 2428, branches/wayback-1_2_1)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-14 01:17:57
|
Revision: 2428
http://archive-access.svn.sourceforge.net/archive-access/?rev=2428&view=rev
Author: bradtofel
Date: 2008-07-13 18:18:06 -0700 (Sun, 13 Jul 2008)
Log Message:
-----------
Added Paths:
-----------
branches/wayback-1_2_2/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mi...@us...> - 2008-07-11 14:00:47
|
Revision: 2427
http://archive-access.svn.sourceforge.net/archive-access/?rev=2427&view=rev
Author: miklosh
Date: 2008-07-11 07:00:57 -0700 (Fri, 11 Jul 2008)
Log Message:
-----------
Initial commit of the image search contrib.
Added Paths:
-----------
trunk/archive-access/projects/nutchwax/imagesearch/
trunk/archive-access/projects/nutchwax/imagesearch/README.txt
trunk/archive-access/projects/nutchwax/imagesearch/bin/
trunk/archive-access/projects/nutchwax/imagesearch/build.xml
trunk/archive-access/projects/nutchwax/imagesearch/conf/
trunk/archive-access/projects/nutchwax/imagesearch/lib/
trunk/archive-access/projects/nutchwax/imagesearch/src/
trunk/archive-access/projects/nutchwax/imagesearch/src/java/
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHit.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHitQueue.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHits.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageIndexer.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageProcessor.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageSearch.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageSearcherBean.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageWritable.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/StoredImage.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ThumbnailGenerator.java
trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/WrappedWritable.java
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/build-plugin.xml
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/build.xml
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/plugin.xml
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/java/
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/java/org/
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/java/org/archive/
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/java/org/archive/nutchwax/
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/java/org/archive/nutchwax/imagesearch/
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/java/org/archive/nutchwax/imagesearch/plugin/
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/java/org/archive/nutchwax/imagesearch/plugin/ImageIndexingFilter.java
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/java/org/archive/nutchwax/imagesearch/plugin/ImageParseFilter.java
trunk/archive-access/projects/nutchwax/imagesearch/src/plugin/src/java/org/archive/nutchwax/imagesearch/plugin/ImageParser.java
Added: trunk/archive-access/projects/nutchwax/imagesearch/README.txt
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/README.txt (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/README.txt 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,59 @@
+Nutch(WAX) Image Search Contrib
+===============================
+
+Getting the source
+------------------
+Check out Nutch-1.0-dev as usually, then check-out the image search
+contrib into Nutch's "contrib" directory.
+
+ $ cd contrib
+ $ svn checkout http://archive-access.svn.sourceforge.net/svnroot/archive-access/trunk/archive-access/projects/nutchwax/imagesearch
+
+This will create a sub-directory named "imagesearch" containing the
+sources for the image search contrib.
+
+
+Configuring
+-----------
+Enable the 'image-search' plugin in Nutch's configuration by appending
+it to the 'plugin.includes' property. This registers three plugins:
+ o ImageParseFilter (HTML parse filter)
+ o ImageParser (fake JPEG and GIF parser)
+ o ImageIndexingFilter
+
+If you are using NutchWAX 0.12 or newer, make sure you add the following
+line to the 'indexingfilter.order' property:
+
+ org.archive.nutchwax.imagesearch.plugin.ImageIndexingFilter
+
+
+Build and install
+-----------------
+Build the contrib by executing the 'ant' build command in
+
+ nutch/contrib/imagesearch
+
+as you normally would.
+
+For example
+
+ $ cd nutch/contrib/imagesearch
+ $ ant tar
+
+This command will build all of Nutch, then the image search add-ons and
+finally will package everything up into the "nutch-1.0-dev.tar.gz"
+release package.
+
+Then install the "nutch-1.0-dev.tar.gz" tarball as normal.
+
+
+Searching
+---------
+After performing the usual steps to import or fetch the files, invert
+the links and index the documents, you can search the resulting indexes
+for images by:
+
+ bin/nutch org.archive.nutchwax.imagesearch.ImageSearcherBean product
+
+This calls the ImageSearcherBean to execute a simple keyword search for
+"product".
Added: trunk/archive-access/projects/nutchwax/imagesearch/build.xml
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/build.xml (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/build.xml 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,138 @@
+<?xml version="1.0"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project name="nutchwax-imagesearch" default="job">
+
+ <property name="nutch.dir" value="../../" />
+
+ <property name="src.dir" value="src" />
+ <property name="lib.dir" value="lib" />
+ <property name="build.dir" value="${nutch.dir}/build" />
+ <!-- HACK: Need to import default.properties like Nutch does -->
+ <property name="dist.dir" value="${build.dir}/nutch-1.0-dev" />
+
+ <target name="nutch-compile-core">
+ <ant dir="${nutch.dir}" target="compile-core" inheritAll="false" />
+ </target>
+
+ <target name="nutch-compile-plugins">
+ <ant dir="${nutch.dir}" target="compile-plugins" inheritAll="false" />
+ </target>
+
+ <target name="compile-core" depends="nutch-compile-core">
+ <javac
+ destdir="${build.dir}/classes"
+ debug="true"
+ verbose="false"
+ source="1.5"
+ target="1.5"
+ encoding="UTF-8"
+ fork="true"
+ nowarn="true"
+ deprecation="false">
+ <src path="${src.dir}/java" />
+ <include name="**/*.java" />
+ <classpath>
+ <pathelement location="${build.dir}/classes" />
+ <fileset dir="${lib.dir}">
+ <include name="*.jar"/>
+ </fileset>
+ <fileset dir="${nutch.dir}/lib">
+ <include name="*.jar"/>
+ </fileset>
+ </classpath>
+ </javac>
+ </target>
+
+ <target name="compile-plugins">
+ <ant dir="src/plugin" target="deploy" inheritAll="false" />
+ </target>
+
+ <!--
+ These targets all call down to the corresponding target in the
+ Nutch build.xml file. This way all of the 'ant' build commands
+ can be executed from this directory and everything should get
+ built as expected.
+ -->
+ <target name="compile" depends="compile-core, compile-plugins, nutch-compile-plugins">
+ </target>
+
+ <target name="jar" depends="compile-core">
+ <ant dir="${nutch.dir}" target="jar" inheritAll="false" />
+ </target>
+
+ <target name="job" depends="compile">
+ <ant dir="${nutch.dir}" target="job" inheritAll="false" />
+ </target>
+
+ <target name="war" depends="compile">
+ <ant dir="${nutch.dir}" target="war" inheritAll="false" />
+ </target>
+
+ <target name="javadoc" depends="compile">
+ <ant dir="${nutch.dir}" target="javadoc" inheritAll="false" />
+ </target>
+
+ <target name="tar" depends="package">
+ <ant dir="${nutch.dir}" target="tar" inheritAll="false" />
+ </target>
+
+ <target name="clean">
+ <ant dir="${nutch.dir}" target="clean" inheritAll="false" />
+ </target>
+
+ <!-- This one does a little more after calling down to the relevant
+ Nutch target. After Nutch has copied everything into the
+ distribution directory, we add our script, libraries, etc.
+
+ Rather than over-write the standard Nutch configuration files,
+ we place ours in a newly created directory
+
+ contrib/imagesearch/conf
+
+ and let the individual user decide whether or not to
+ incorporate our modifications.
+ -->
+ <target name="package" depends="jar, job, war, javadoc">
+ <ant dir="${nutch.dir}" target="package" inheritAll="false" />
+
+ <copy todir="${dist.dir}/lib" includeEmptyDirs="false">
+ <fileset dir="lib"/>
+ </copy>
+
+ <copy todir="${dist.dir}/bin">
+ <fileset dir="bin"/>
+ </copy>
+
+ <chmod perm="ugo+x" type="file">
+ <fileset dir="${dist.dir}/bin"/>
+ </chmod>
+
+ <mkdir dir="${dist.dir}/contrib/imagesearch/conf"/>
+ <copy todir="${dist.dir}/contrib/imagesearch/conf">
+ <fileset dir="conf" />
+ </copy>
+
+ <copy todir="${dist.dir}/contrib/imagesearch">
+ <fileset dir=".">
+ <include name="*.txt" />
+ </fileset>
+ </copy>
+
+ </target>
+
+</project>
Added: trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHit.java
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHit.java (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHit.java 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.archive.nutchwax.imagesearch;
+
+public class ImageHit {
+ public int doc;
+ public float docScore;
+
+ public String imageId;
+ public String url;
+
+ public float docSim;
+ public float proximity;
+ public float score;
+
+ public ImageHit(String id, String url, int doc) {
+ this.imageId = id;
+ this.url = url;
+ this.doc = doc;
+ }
+}
Added: trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHitQueue.java
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHitQueue.java (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHitQueue.java 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,37 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.archive.nutchwax.imagesearch;
+
+import org.apache.lucene.util.PriorityQueue;
+
+final class ImageHitQueue extends PriorityQueue {
+
+ ImageHitQueue(int size) {
+ initialize(size);
+ }
+
+ protected final boolean lessThan(Object a, Object b) {
+ ImageHit hitA = (ImageHit)a;
+ ImageHit hitB = (ImageHit)b;
+ if (hitA.score == hitB.score) {
+ return hitA.doc > hitB.doc;
+ } else {
+ return hitA.score < hitB.score;
+ }
+ }
+}
Added: trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHits.java
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHits.java (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageHits.java 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.archive.nutchwax.imagesearch;
+
+/** A set of image hits matching a query. Adapted from Nutch's Hits class. */
+public final class ImageHits {
+
+ private long total;
+ private boolean totalIsExact = true;
+ private ImageHit[] top;
+
+ public ImageHits() {
+ }
+
+ public ImageHits(long total, ImageHit[] top) {
+ this.total = total;
+ this.top = top;
+ }
+
+ /** Returns the total number of hits for this query. This may be an estimate
+ * when (@link #totalIsExact()} is false. */
+ public long getTotal() {
+ return total;
+ }
+
+ /** True if {@link #getTotal()} gives the exact number of hits, or false if
+ * it is only an estimate of the total number of hits. */
+ public boolean totalIsExact() {
+ return totalIsExact;
+ }
+
+ /** Set {@link #totalIsExact()}. */
+ public void setTotalIsExact(boolean isExact) {
+ totalIsExact = isExact;
+ }
+
+ /** Returns the number of hits included in this current listing. */
+ public int getLength() {
+ return top.length;
+ }
+
+ /** Returns the <code>i</code><sup>th</sup> hit in this list. */
+ public ImageHit getHit(int i) {
+ return top[i];
+ }
+
+ /** Returns a subset of the hit objects. */
+ public ImageHit[] getHits(int start, int length) {
+ ImageHit[] results = new ImageHit[length];
+ for (int i = 0; i < length; i++) {
+ results[i] = top[start + i];
+ }
+ return results;
+ }
+}
Added: trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageIndexer.java
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageIndexer.java (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageIndexer.java 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,432 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.archive.nutchwax.imagesearch;
+
+import java.io.*;
+import java.util.*;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import org.apache.hadoop.io.*;
+import org.apache.hadoop.fs.*;
+import org.apache.hadoop.conf.*;
+import org.apache.hadoop.mapred.*;
+import org.apache.hadoop.util.*;
+import org.apache.nutch.parse.*;
+import org.apache.nutch.analysis.*;
+
+import org.apache.nutch.scoring.ScoringFilterException;
+import org.apache.nutch.scoring.ScoringFilters;
+import org.apache.nutch.util.LogUtil;
+import org.apache.nutch.util.NutchConfiguration;
+import org.apache.nutch.util.NutchJob;
+
+import org.apache.nutch.crawl.CrawlDatum;
+import org.apache.nutch.crawl.CrawlDb;
+import org.apache.nutch.crawl.Inlinks;
+import org.apache.nutch.crawl.LinkDb;
+import org.apache.nutch.crawl.NutchWritable;
+
+import org.apache.lucene.index.*;
+import org.apache.lucene.document.*;
+import org.apache.nutch.indexer.IndexingException;
+import org.apache.nutch.indexer.IndexingFilters;
+import org.apache.nutch.indexer.NutchSimilarity;
+import org.apache.nutch.metadata.Metadata;
+import org.apache.nutch.metadata.Nutch;
+import org.apache.nutch.net.protocols.Response;
+
+/** Create indexes for segments. */
+public class ImageIndexer extends Configured implements Tool,
+ Reducer<Text, WrappedWritable, Text, Writable>, Mapper<Text, Writable, Text, WrappedWritable> {
+
+ public static final String DONE_NAME = "index.done";
+ public static final Log LOG = LogFactory.getLog(ImageIndexer.class);
+
+ /** A utility class used to pass a lucene document from Indexer.reduce
+ * to Indexer.OutputFormat.
+ * Note: Despite its name, it can't properly wrap a lucene document - it
+ * doesn't know how to serialize/deserialize a lucene document.
+ */
+ private static class LuceneDocumentWrapper implements Writable {
+
+ private Document doc;
+
+ public LuceneDocumentWrapper(Document doc) {
+ this.doc = doc;
+ }
+
+ public Document get() {
+ return doc;
+ }
+
+ public void readFields(DataInput in) throws IOException {
+ // intentionally left blank
+ }
+
+ public void write(DataOutput out) throws IOException {
+ // intentionally left blank
+ }
+ }
+
+ /** Unwrap Lucene Documents created by reduce and add them to an index. */
+ public static class OutputFormat
+ extends org.apache.hadoop.mapred.OutputFormatBase<WritableComparable, LuceneDocumentWrapper> {
+
+ public RecordWriter<WritableComparable, LuceneDocumentWrapper> getRecordWriter(final FileSystem fs, JobConf job,
+ String name, final Progressable progress) throws IOException {
+ final Path perm = new Path(job.getOutputPath(), name);
+ final Path temp =
+ job.getLocalPath("index/_" + Integer.toString(new Random().nextInt()));
+
+ fs.delete(perm); // delete old, if any
+
+ final AnalyzerFactory factory = new AnalyzerFactory(job);
+ final IndexWriter writer = // build locally first
+ new IndexWriter(fs.startLocalOutput(perm, temp).toString(),
+ new NutchDocumentAnalyzer(job), true);
+
+ writer.setMergeFactor(job.getInt("indexer.mergeFactor", 10));
+ writer.setMaxBufferedDocs(job.getInt("indexer.minMergeDocs", 100));
+ writer.setMaxMergeDocs(job.getInt("indexer.maxMergeDocs", Integer.MAX_VALUE));
+ writer.setTermIndexInterval(job.getInt("indexer.termIndexInterval", 128));
+ writer.setMaxFieldLength(job.getInt("indexer.max.tokens", 10000));
+ writer.setInfoStream(LogUtil.getInfoStream(LOG));
+ writer.setUseCompoundFile(false);
+ writer.setSimilarity(new NutchSimilarity());
+
+ return new RecordWriter<WritableComparable, LuceneDocumentWrapper>() {
+
+ boolean closed;
+
+ public void write(WritableComparable key, LuceneDocumentWrapper value)
+ throws IOException { // unwrap & index doc
+ Document doc = value.get();
+ NutchAnalyzer analyzer = factory.get(doc.get("lang"));
+ if (LOG.isInfoEnabled()) {
+ LOG.info(" Indexing [" + doc.getField("url").stringValue() + "]" +
+ " with analyzer " + analyzer +
+ " (" + doc.get("lang") + ")");
+ }
+ writer.addDocument(doc, analyzer);
+ progress.progress();
+ }
+
+ public void close(final Reporter reporter) throws IOException {
+ // spawn a thread to give progress heartbeats
+ Thread prog = new Thread() {
+
+ public void run() {
+ while (!closed) {
+ try {
+ reporter.setStatus("closing");
+ Thread.sleep(1000);
+ } catch (InterruptedException e) {
+ continue;
+ } catch (Throwable e) {
+ return;
+ }
+ }
+ }
+ };
+
+ try {
+ prog.start();
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Optimizing index.");
+ }
+ // optimize & close index
+ writer.optimize();
+ writer.close();
+ fs.completeLocalOutput(perm, temp); // copy to dfs
+ fs.createNewFile(new Path(perm, DONE_NAME));
+ } finally {
+ closed = true;
+ }
+ }
+ };
+ }
+ }
+ private IndexingFilters filters;
+ private ScoringFilters scfilters;
+
+ public ImageIndexer() {
+ }
+
+ public ImageIndexer(Configuration conf) {
+ setConf(conf);
+ }
+
+ public void configure(JobConf job) {
+ setConf(job);
+ this.filters = new IndexingFilters(getConf());
+ this.scfilters = new ScoringFilters(getConf());
+ }
+
+ public void close() {
+ }
+
+ /**
+ * Copies key/value pairs from one metadata container to another.
+ * Overwrites the destination if the source has a value with greater length.
+ *
+ * @param from Metadata to copy from
+ * @param to target metadata container
+ */
+ private void mergeMetadata(Metadata from, Metadata to) {
+ String[] names = from.names();
+ for (String name : names) {
+ String newValue = from.get(name);
+ String value = to.get(name);
+ if (value != null) {
+ if (newValue.length() > value.length()) {
+ to.set(name, newValue);
+ }
+ } else {
+ to.add(name, newValue);
+ }
+ }
+ }
+
+ public void reduce(Text key, Iterator<WrappedWritable> values,
+ OutputCollector<Text, Writable> output, Reporter reporter)
+ throws IOException {
+ Inlinks inlinks = null;
+ CrawlDatum dbDatum = null;
+ CrawlDatum fetchDatum = null;
+ ParseData parseData = null;
+ ParseText parseText = null;
+
+ Metadata metadata = null;
+ Metadata contentMetadata = null;
+ String segmentName = null;
+ String signature = null;
+ while (values.hasNext()) {
+ Writable value = values.next().get();
+ if (value instanceof ImageWritable) {
+ ImageWritable imgData = (ImageWritable) value;
+ Metadata imgMeta = imgData.getMetadata();
+ if (metadata == null) {
+ metadata = imgMeta;
+ } else {
+ mergeMetadata(imgMeta, metadata);
+ }
+ } else if (value instanceof Inlinks) {
+ inlinks = (Inlinks) value;
+ } else if (value instanceof CrawlDatum) {
+ CrawlDatum datum = (CrawlDatum) value;
+ if (CrawlDatum.hasDbStatus(datum)) {
+ dbDatum = datum;
+ } else if (CrawlDatum.hasFetchStatus(datum)) {
+ // don't index unmodified (empty) pages
+ if (datum.getStatus() != CrawlDatum.STATUS_FETCH_NOTMODIFIED) {
+ fetchDatum = datum;
+ }
+ } else if (CrawlDatum.STATUS_LINKED == datum.getStatus() ||
+ CrawlDatum.STATUS_SIGNATURE == datum.getStatus()) {
+ continue;
+ } else {
+ throw new RuntimeException("Unexpected status: " + datum.getStatus());
+ }
+ } else if (value instanceof ParseData) {
+ if (parseData != null) {
+ ParseData newParse = (ParseData) value;
+ Metadata parseMeta = newParse.getParseMeta();
+ // Check if this is the parse meta from ImageParseFilter
+ // If so, use its parse meta, otherwise use the content meta
+ if (parseMeta.get(ImageSearch.PARENT_URL_KEY) != null) {
+ mergeMetadata(parseMeta, metadata);
+ } else {
+ contentMetadata = newParse.getContentMeta();
+ }
+ } else {
+ parseData = (ParseData) value;
+ metadata = parseData.getParseMeta();
+ contentMetadata = parseData.getContentMeta();
+ }
+ } else if (value instanceof ParseText) {
+ ParseText newParseText = (ParseText) value;
+ if (parseText == null || (parseText != null &&
+ parseText.getText().length() < newParseText.getText().length())) {
+ parseText = (ParseText) value;
+ }
+ } else if (LOG.isWarnEnabled()) {
+ LOG.warn("Unrecognized type: " + value.getClass());
+ }
+ // Save segment name and signature
+ if (contentMetadata != null) {
+ if (segmentName == null || signature == null) {
+ String stringValue = contentMetadata.get(Nutch.SEGMENT_NAME_KEY);
+ if (stringValue != null) {
+ segmentName = stringValue;
+ }
+ stringValue = contentMetadata.get(Nutch.SIGNATURE_KEY);
+ if (stringValue != null) {
+ signature = stringValue;
+ }
+ }
+ }
+ }
+
+ if (fetchDatum == null || dbDatum == null || parseText == null || parseData == null) {
+ return; // only have inlinks
+ }
+ if (!parseData.getStatus().isSuccess() ||
+ fetchDatum.getStatus() != CrawlDatum.STATUS_FETCH_SUCCESS) {
+ return;
+ }
+
+ // Skip possibly non-images
+ if (metadata.get(ImageSearch.PARENT_URL_KEY) == null) {
+ return;
+ }
+ // Make sure segment name and signature are set
+ contentMetadata.set(Nutch.SEGMENT_NAME_KEY, segmentName);
+ contentMetadata.set(Nutch.SIGNATURE_KEY, signature);
+
+ Document doc = new Document();
+
+ // add segment, used to map from merged index back to segment files
+ doc.add(new Field("segment", contentMetadata.get(Nutch.SEGMENT_NAME_KEY),
+ Field.Store.YES, Field.Index.NO));
+
+ // add digest, used by dedup
+ doc.add(new Field("digest", contentMetadata.get(Nutch.SIGNATURE_KEY),
+ Field.Store.YES, Field.Index.NO));
+
+ ParseData combinedParseData = new ParseData(parseData.getStatus(),
+ parseData.getTitle(), parseData.getOutlinks(), contentMetadata,
+ metadata);
+
+ Parse parse = new ParseImpl(parseText, combinedParseData);
+ try {
+ // extract information from dbDatum and pass it to
+ // fetchDatum so that indexing filters can use it
+ Text url = (Text) dbDatum.getMetaData().get(Nutch.WRITABLE_REPR_URL_KEY);
+ if (url != null) {
+ fetchDatum.getMetaData().put(Nutch.WRITABLE_REPR_URL_KEY, url);
+ }
+ // run indexing filters
+ doc = this.filters.filter(doc, parse, key, fetchDatum, inlinks);
+ } catch (IndexingException e) {
+ if (LOG.isWarnEnabled()) {
+ LOG.warn("Error indexing " + key + ": " + e);
+ }
+ return;
+ }
+
+ // skip documents discarded by indexing filters
+ if (doc == null) {
+ return;
+ }
+
+ float boost = 1.0f;
+ // run scoring filters
+ try {
+ boost = this.scfilters.indexerScore((Text) key, doc, dbDatum,
+ fetchDatum, parse, inlinks, boost);
+ } catch (ScoringFilterException e) {
+ if (LOG.isWarnEnabled()) {
+ LOG.warn("Error calculating score " + key + ": " + e);
+ }
+ return;
+ }
+ // apply boost to all indexed fields.
+ doc.setBoost(boost);
+ // store boost for use by explain and dedup
+ doc.add(new Field("boost", Float.toString(boost),
+ Field.Store.YES, Field.Index.NO));
+
+ output.collect(key, new LuceneDocumentWrapper(doc));
+ }
+
+ public void index(Path indexDir, Path crawlDb, Path linkDb, Path[] segments)
+ throws IOException {
+
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ImageIndexer: starting");
+ LOG.info("ImageIndexer: linkdb: " + linkDb);
+ }
+
+ JobConf job = new NutchJob(getConf());
+ job.setJobName("index " + indexDir);
+
+ for (int i = 0; i < segments.length; i++) {
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ImageIndexer: adding segment: " + segments[i]);
+ }
+ job.addInputPath(new Path(segments[i], CrawlDatum.FETCH_DIR_NAME));
+ job.addInputPath(new Path(segments[i], CrawlDatum.PARSE_DIR_NAME));
+ job.addInputPath(new Path(segments[i], ParseData.DIR_NAME));
+ job.addInputPath(new Path(segments[i], ParseText.DIR_NAME));
+ job.addInputPath(new Path(segments[i], ImageWritable.IMAGE_DATA_DIR));
+ }
+
+ job.addInputPath(new Path(crawlDb, CrawlDb.CURRENT_NAME));
+ job.addInputPath(new Path(linkDb, LinkDb.CURRENT_NAME));
+ job.setInputFormat(SequenceFileInputFormat.class);
+
+ job.setMapperClass(ImageIndexer.class);
+ job.setMapOutputKeyClass(Text.class);
+ job.setMapOutputValueClass(WrappedWritable.class);
+ job.setReducerClass(ImageIndexer.class);
+
+ job.setOutputPath(indexDir);
+ job.setOutputFormat(OutputFormat.class);
+ job.setOutputKeyClass(Text.class);
+ job.setOutputValueClass(NutchWritable.class);
+
+ JobClient.runJob(job);
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ImageIndexer: done");
+ }
+ }
+
+ public static void main(String[] args) throws Exception {
+ int res = ToolRunner.run(NutchConfiguration.create(), new ImageIndexer(), args);
+ System.exit(res);
+ }
+
+ public int run(String[] args) throws Exception {
+
+ if (args.length < 4) {
+ System.err.println("Usage: <index> <crawldb> <linkdb> <segment> ...");
+ return -1;
+ }
+
+ Path[] segments = new Path[args.length - 3];
+ for (int i = 3; i < args.length; i++) {
+ segments[i - 3] = new Path(args[i]);
+ }
+
+ try {
+ index(new Path(args[0]), new Path(args[1]), new Path(args[2]),
+ segments);
+ return 0;
+ } catch (Exception e) {
+ LOG.fatal("ImageIndexer: " + StringUtils.stringifyException(e));
+ return -1;
+ }
+ }
+
+ public void map(Text key, Writable value,
+ OutputCollector<Text, WrappedWritable> output, Reporter reporter) throws IOException {
+ output.collect(key, new WrappedWritable(value));
+ }
+}
Added: trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageProcessor.java
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageProcessor.java (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageProcessor.java 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,136 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.archive.nutchwax.imagesearch;
+
+import java.io.IOException;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.mapred.JobClient;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.MapFileOutputFormat;
+import org.apache.hadoop.mapred.Mapper;
+import org.apache.hadoop.mapred.OutputCollector;
+import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapred.SequenceFileInputFormat;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.nutch.metadata.Metadata;
+import org.apache.nutch.protocol.Content;
+import org.apache.nutch.util.NutchConfiguration;
+import org.apache.nutch.util.NutchJob;
+
+public class ImageProcessor extends Configured implements Tool,
+ Mapper<Text, Content, Text, ImageWritable> {
+
+ private static final Log LOG = LogFactory.getLog(ImageProcessor.class);
+
+ private int thumbQuality;
+ private int thumbMaxSize;
+
+ ImageProcessor() {}
+ ImageProcessor(Configuration conf) {
+ setConf(conf);
+ }
+
+ public void map(Text key, Content content,
+ OutputCollector<Text, ImageWritable> output,
+ Reporter reporter) throws IOException {
+
+ Metadata metadata = new Metadata();
+ // Check content type
+ if (!content.getContentType().contains("image/")) {
+ return;
+ }
+
+ // Generate thumbnail
+ byte[] data = content.getContent();
+ StoredImage thumb = ThumbnailGenerator.generateThumbnail(data,
+ thumbMaxSize, thumbMaxSize, thumbQuality, metadata);
+
+ // Create and setup an ImageWritable
+ ImageWritable image = new ImageWritable(key.toString());
+ image.setMetadata(metadata);
+ image.setThumbnail(thumb);
+
+ output.collect(key, image);
+ }
+
+ public void processImageContent(Path segment)
+ throws IOException {
+
+ JobConf job = new NutchJob(getConf());
+ job.setJobName("ImageProcessor " + segment);
+
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ImageProcessor: processing " + segment);
+ }
+ job.addInputPath(new Path(segment, Content.DIR_NAME));
+
+ job.setInputFormat(SequenceFileInputFormat.class);
+ job.setMapperClass(ImageProcessor.class);
+
+ job.setOutputPath(new Path(segment, ImageWritable.IMAGE_DATA_DIR));
+ job.setOutputFormat(MapFileOutputFormat.class);
+ job.setOutputKeyClass(Text.class);
+ job.setOutputValueClass(ImageWritable.class);
+
+ JobClient.runJob(job);
+
+ if (LOG.isInfoEnabled()) {
+ LOG.info("ImageProcessor: done");
+ }
+ }
+
+ public static void main(String[] args) throws Exception {
+ int res = ToolRunner.run(NutchConfiguration.create(),
+ new ImageProcessor(), args);
+ System.exit(res);
+ }
+ public int run(String[] args) throws Exception {
+
+ if (args.length == 0) {
+ System.err.println("Usage: imageprocessor <segment>");
+ return -1;
+ }
+
+ Path segment = new Path(args[0]);
+ try {
+ processImageContent(segment);
+ return 0;
+ } catch (Exception e) {
+ LOG.fatal("ImageProcessor: " + StringUtils.stringifyException(e));
+ return -1;
+ }
+ }
+
+ private Configuration conf;
+ public void configure(JobConf conf) {
+ setConf(conf);
+
+ this.thumbQuality = conf.getInt("imagesearcher.thumbnail.quality", 50);
+ this.thumbMaxSize = conf.getInt("imagesearcher.thumbnail.maxSize", 100);
+ }
+
+ public void close() throws IOException {
+ }
+}
Added: trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageSearch.java
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageSearch.java (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageSearch.java 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.archive.nutchwax.imagesearch;
+
+public class ImageSearch {
+ public static final String PARENT_URL_KEY = "parent_url";
+ public static final String ALT_TEXT_KEY = "alt";
+
+ public static final String IMAGE_IDS_KEY = "image_ids";
+ public static final String IMAGE_POS_KEY = "image_pos";
+ public static final String IMAGE_URLS_KEY = "image_urls";
+ public static final String HAS_IMAGE_KEY = "has_image";
+}
Added: trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageSearcherBean.java
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageSearcherBean.java (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageSearcherBean.java 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,346 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.archive.nutchwax.imagesearch;
+
+import java.io.File;
+import java.io.IOException;
+import java.lang.Math;
+import java.util.Iterator;
+import java.util.Vector;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.MultiReader;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.spans.SpanNearQuery;
+import org.apache.lucene.search.spans.SpanQuery;
+import org.apache.lucene.search.spans.SpanTermQuery;
+import org.apache.lucene.search.spans.Spans;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.FSDirectory;
+import org.apache.nutch.indexer.FsDirectory;
+import org.apache.nutch.indexer.Indexer;
+import org.apache.nutch.util.NutchConfiguration;
+
+public class ImageSearcherBean {
+
+ public static final Log LOG = LogFactory.getLog(ImageSearcherBean.class);
+
+ private IndexReader reader;
+
+ private Path baseDir;
+ private Configuration conf;
+ private FileSystem fs;
+
+ private int distThreshold; // Maximum allowed distance of image from hit
+ // to be considered
+
+ /** Construct given a configuration. */
+ public ImageSearcherBean(Configuration conf) throws IOException {
+ this.conf = conf;
+ this.fs = FileSystem.get(conf);
+ this.baseDir = new Path(conf.get("searcher.dir", "crawl"));
+ this.distThreshold = conf.getInt("imagesearch.maxDist", 300);
+
+ // Try to load unmerged indexes
+ Path indexesDir = new Path(baseDir, "indexes");
+ if (this.fs.exists(indexesDir)) {
+ Vector<Path> doneDirs = new Vector<Path>();
+ Path[] dirs = fs.listPaths(indexesDir, new PathFilter() {
+
+ public boolean accept(Path f) {
+ try {
+ if (fs.isDirectory(f)) {
+ return true;
+ }
+ } catch (IOException ioe) {
+ }
+ return false;
+ }
+ });
+ for (Path dir : dirs) {
+ Path indexdone = new Path(dir, Indexer.DONE_NAME);
+ if (fs.isFile(indexdone)) {
+ doneDirs.add(dir);
+ }
+ }
+ dirs = new Path[doneDirs.size()];
+ Iterator<Path> it = doneDirs.iterator();
+ int i = 0;
+ while (it.hasNext()) {
+ dirs[i++] = it.next();
+ }
+ init(dirs);
+ } else {
+ Path[] indexDir = {new Path(baseDir, "index")};
+ init(indexDir);
+ }
+ }
+
+ /** Init given a set of indexes or just one index. */
+ public void init(Path[] indexes) throws IOException {
+ IndexReader[] indexReaders = new IndexReader[indexes.length];
+ for (int i = 0; i < indexes.length; i++) {
+ indexReaders[i] = IndexReader.open(getDirectory(indexes[i]));
+ }
+ if (indexes.length > 1) {
+ this.reader = new MultiReader(indexReaders);
+ } else {
+ this.reader = IndexReader.open(getDirectory(indexes[0]));
+ }
+ }
+
+ private Directory getDirectory(Path file) throws IOException {
+ if ("file".equals(this.fs.getUri().getScheme())) {
+ Path qualified = file.makeQualified(FileSystem.getLocal(conf));
+ File fsLocal = new File(qualified.toUri());
+ return FSDirectory.getDirectory(fsLocal.getAbsolutePath());
+ } else {
+ return new FsDirectory(this.fs, file, false, this.conf);
+ }
+ }
+
+ public void close() throws IOException {
+ if (reader != null) {
+ reader.close();
+ }
+ }
+
+ public IndexReader getReader() {
+ return reader;
+ }
+
+ /**
+ * Calculate the score for an image hit.
+ * @param hit found hit
+ * @param doc parent document
+ * @return float score
+ */
+ private float scoreHit(ImageHit hit, Document doc) {
+ float a = 0.2f;
+ float b = 0.1f;
+ return a*hit.docScore + (1.0f-a)*(b*hit.docSim + (1.0f-b)*hit.proximity);
+ }
+
+ /**
+ * Find query-related images in the content of documents based on proximity.
+ *
+ * @param queryTerms
+ * @param hitCollector
+ * @throws java.io.IOException
+ */
+ private long getImagesFromContent(Term[] queryTerms, ImageHitQueue hitCollector,
+ int maxHits)
+ throws IOException {
+
+ // Construct SpanQuery
+ SpanQuery[] clauses = new SpanTermQuery[queryTerms.length];
+ for (int i=0; i<queryTerms.length; i++) {
+ clauses[i] = new SpanTermQuery(queryTerms[i]);
+ }
+ SpanNearQuery snq = new SpanNearQuery(clauses, queryTerms.length+1, false);
+ Spans spans = snq.getSpans(reader);
+
+ // Per document info
+ Document doc = null;
+ int currentDoc = -1;
+ int numDocImages = 0;
+ int[] imagePositions = null;
+ String[] imageIds = null;
+ String[] imageUrls = null;
+ float docBoost = 1.0f;
+ float docSim = 0.0f;
+ float maxDist = Float.MAX_VALUE;
+ float minScore = 0.0f;
+
+ long totalHits = 0;
+
+ boolean more = spans.next();
+ while (more) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("currentDoc "+currentDoc);
+ }
+ if (currentDoc != spans.doc()) {
+ currentDoc = spans.doc();
+ doc = reader.document(currentDoc);
+ // Skip document with no images
+ if ("0".equals(doc.getField(ImageSearch.HAS_IMAGE_KEY).stringValue())) {
+ while (more && spans.doc() == currentDoc) {
+ more = spans.next();
+ }
+ continue;
+ }
+
+ // Get document's global score
+ docBoost = doc.getBoost();
+
+ // Get image positions
+ String posField = doc.getField(ImageSearch.IMAGE_POS_KEY).stringValue();
+ String[] positions = posField.split(":");
+ imagePositions = new int[positions.length];
+ numDocImages = positions.length;
+ for (int i = 0; i < numDocImages; i++) {
+ imagePositions[i] = Integer.parseInt(positions[i]);
+ }
+ maxDist = (float)imagePositions[numDocImages-1];
+
+ // Get image ids
+ String idField = doc.getField(ImageSearch.IMAGE_IDS_KEY).stringValue();
+ imageIds = idField.split(":");
+
+ // Get image urls
+ String urlField = doc.getField(ImageSearch.IMAGE_URLS_KEY).stringValue();
+ imageUrls = urlField.split(" ");
+ }
+
+ int pos = 0;
+ int end = 0;
+ int imgIndex = 0;
+ int prevDist = Integer.MAX_VALUE;
+ while (more && spans.doc() == currentDoc) {
+ if (imgIndex >= numDocImages) {
+ more = spans.next();
+ continue;
+ }
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("sp " + spans.start() + " " + spans.end());
+ }
+ pos = spans.start();
+ end = spans.end();
+ int dist = Math.abs(imagePositions[imgIndex] - pos) + (end-pos);
+ int nextDist = imgIndex < numDocImages-1 ?
+ Math.abs(imagePositions[imgIndex + 1] - pos) + (end-pos) : Integer.MAX_VALUE;
+ /*if (prevDist < dist) {
+ more = spans.next();
+ prevDist = dist;
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("p<d");
+ }
+ continue;
+ }*/
+ // Advance image pointer till a nearer image can be found
+ while (imgIndex < numDocImages && nextDist <= dist) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("adv " + nextDist + " " + dist + " id " + imageUrls[imgIndex].substring(imageUrls[imgIndex].lastIndexOf("/")));
+ }
+ dist = nextDist;
+ imgIndex++;
+ nextDist = imgIndex < numDocImages-1 ?
+ Math.abs(imagePositions[imgIndex+1] - pos) + (end-pos) : Integer.MAX_VALUE;
+ }
+ // Check if this image is in the allowed proximity of the span
+ if (dist > distThreshold) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("d>t: " + dist);
+ }
+ more = spans.next();
+ continue;
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("hit " + imageUrls[imgIndex].substring(imageUrls[imgIndex].lastIndexOf("/")) + " " + dist + " next " + nextDist);
+ }
+ // Found hit
+ ImageHit newHit = new ImageHit(imageIds[imgIndex], imageUrls[imgIndex], currentDoc);
+ newHit.docSim = docSim;
+ newHit.docScore = docBoost;
+ newHit.proximity = Math.min(1.0f, 1.0f-((float)dist/maxDist));
+ newHit.score = scoreHit(newHit, doc);
+
+ if (hitCollector.size() < maxHits || newHit.score >= minScore) {
+ hitCollector.insert(newHit);
+ minScore = ((ImageHit)hitCollector.top()).score;
+
+ prevDist = dist;
+ imgIndex++;
+ }
+ totalHits++;
+ more = spans.next();
+ }
+ }
+
+ return totalHits;
+ }
+
+ /**
+ * Search for images matching the query.
+ *
+ * @param query query
+ * @param maxHits maximum number of hits to retrieve
+ * @return ImageHits the matching hits
+ * @throws java.io.IOException
+ */
+ public ImageHits search(String query, int maxHits) throws IOException {
+ String[] keywords = query.split("\\s");
+ if (keywords == null) {
+ return new ImageHits(0, new ImageHit[0]);
+ }
+
+ // Create query term array
+ Term[] queryTerms = new Term[keywords.length];
+ for (int i=0; i<queryTerms.length; i++) {
+ queryTerms[i] = new Term("content", keywords[i]);
+ }
+
+ ImageHitQueue hitQueue = new ImageHitQueue(maxHits);
+ long totalHits = getImagesFromContent(queryTerms, hitQueue, maxHits);
+
+ // Extract top results
+ ImageHit[] resultSet = new ImageHit[hitQueue.size()];
+ for (int i = resultSet.length - 1; i >= 0; i--) {
+ resultSet[i] = (ImageHit) hitQueue.pop();
+ }
+
+ return new ImageHits(totalHits, resultSet);
+ }
+
+ /** For debugging purposes. */
+ public static void main(String[] args) throws Exception {
+ if (args.length == 0) {
+ System.err.println("Usage: ImageSearcherBean <query>");
+ System.exit(-1);
+ }
+
+ Configuration conf = NutchConfiguration.create();
+ ImageSearcherBean isb = new ImageSearcherBean(conf);
+
+ // Construct query string
+ StringBuffer sb = new StringBuffer();
+ for (String arg : args) {
+ if (sb.length() > 0) {
+ sb.append(' ');
+ }
+ sb.append(arg);
+ }
+ // Conduct search
+ int maxHits = 10;
+ ImageHits hits = isb.search(sb.toString(), maxHits);
+ // Show results
+ System.out.println("Total hits: " + hits.getTotal());
+ ImageHit[] top = hits.getHits(0,
+ hits.getTotal() >= maxHits ? maxHits : (int)hits.getTotal());
+ for (ImageHit hit : top) {
+ System.out.println(hit.score + " " + hit.url + " " + hit.imageId);
+ }
+ }
+}
Added: trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageWritable.java
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageWritable.java (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/ImageWritable.java 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.archive.nutchwax.imagesearch;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.io.MD5Hash;
+import org.apache.hadoop.io.Writable;
+import org.apache.nutch.metadata.Metadata;
+
+public class ImageWritable implements Writable {
+
+ public static final String IMAGE_DATA_DIR = "image_data";
+
+ private MD5Hash id;
+ private Metadata metadata;
+ private StoredImage thumbnail;
+
+ public ImageWritable() {}
+
+ public ImageWritable(String url) {
+ this.id = MD5Hash.digest(url);
+ this.metadata = new Metadata();
+ }
+
+ public Metadata getMetadata() {
+ return metadata;
+ }
+
+ public void setMetadata(Metadata metadata) {
+ this.metadata = metadata;
+ }
+
+ public void setThumbnail(StoredImage thumbnail) {
+ this.thumbnail = thumbnail;
+ }
+
+ public StoredImage getThumbnail() {
+ return thumbnail;
+ }
+
+ public void write(DataOutput out) throws IOException {
+ id.write(out);
+ metadata.write(out);
+ if (thumbnail != null) {
+ out.writeBoolean(true);
+ thumbnail.write(out);
+ } else {
+ out.writeBoolean(false);
+ }
+ }
+
+ public void readFields(DataInput in) throws IOException {
+ id = MD5Hash.read(in);
+ metadata = new Metadata();
+ metadata.readFields(in);
+ thumbnail = new StoredImage();
+ if (in.readBoolean()) {
+ thumbnail.readFields(in);
+ }
+ }
+}
Added: trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/StoredImage.java
===================================================================
--- trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/StoredImage.java (rev 0)
+++ trunk/archive-access/projects/nutchwax/imagesearch/src/java/org/archive/nutchwax/imagesearch/StoredImage.java 2008-07-11 14:00:57 UTC (rev 2427)
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.archive.nutchwax.imagesearch;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.io.Writable;
+
+/** Represents binary image data as a Writable. */
+public class StoredImage implements Writable {
+
+ public static final byte TYPE_JPEG = 'j';
+ public static final byte T...
[truncated message content] |
|
From: <bra...@us...> - 2008-07-10 18:46:26
|
Revision: 2426
http://archive-access.svn.sourceforge.net/archive-access/?rev=2426&view=rev
Author: bradtofel
Date: 2008-07-10 11:46:15 -0700 (Thu, 10 Jul 2008)
Log Message:
-----------
Removed/moved => org.archive.wayback.util.bdb
Removed Paths:
-------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-10 18:41:47
|
Revision: 2425
http://archive-access.svn.sourceforge.net/archive-access/?rev=2425&view=rev
Author: bradtofel
Date: 2008-07-10 11:41:53 -0700 (Thu, 10 Jul 2008)
Log Message:
-----------
REFACTOR: moved org.archive.wayback.bdb.* to org.archive.wayback.util.bdb.*
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/ui/AdministrativeExclusionAuthority.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/BDBIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/BDBRecordToSearchResultAdapter.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/SearchResultToBDBRecordAdapter.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/BDBResourceFileLocationDB.java
Added Paths:
-----------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecord.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecordIterator.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecordSet.java
Removed Paths:
-------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecord.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordIterator.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordSet.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/ui/AdministrativeExclusionAuthority.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/ui/AdministrativeExclusionAuthority.java 2008-07-09 00:30:15 UTC (rev 2424)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/ui/AdministrativeExclusionAuthority.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -27,10 +27,10 @@
import java.util.ArrayList;
import org.apache.commons.httpclient.URIException;
-import org.archive.wayback.bdb.BDBRecord;
-import org.archive.wayback.bdb.BDBRecordSet;
-import org.archive.wayback.bdb.BDBRecordIterator;
import org.archive.wayback.surt.SURTTokenizer;
+import org.archive.wayback.util.bdb.BDBRecord;
+import org.archive.wayback.util.bdb.BDBRecordIterator;
+import org.archive.wayback.util.bdb.BDBRecordSet;
import com.sleepycat.je.DatabaseException;
Deleted: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecord.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecord.java 2008-07-09 00:30:15 UTC (rev 2424)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecord.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -1,58 +0,0 @@
-/* BDBRecord
- *
- * $Id$
- *
- * Created on 1:43:15 PM May 15, 2006.
- *
- * Copyright (C) 2006 Internet Archive.
- *
- * This file is part of wayback.
- *
- * wayback is free software; you can redistribute it and/or modify
- * it under the terms of the GNU Lesser Public License as published by
- * the Free Software Foundation; either version 2.1 of the License, or
- * any later version.
- *
- * wayback is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU Lesser Public License for more details.
- *
- * You should have received a copy of the GNU Lesser Public License
- * along with wayback; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
- */
-package org.archive.wayback.bdb;
-
-import com.sleepycat.je.DatabaseEntry;
-
-/**
- *
- *
- * @author brad
- * @version $Date$, $Revision$
- */
-public class BDBRecord {
- private DatabaseEntry key;
- private DatabaseEntry value;
- /**
- * @param key
- * @param value
- */
- public BDBRecord(DatabaseEntry key, DatabaseEntry value) {
- this.key = key;
- this.value = value;
- }
- /**
- * @return Returns the key.
- */
- public DatabaseEntry getKey() {
- return key;
- }
- /**
- * @return Returns the value.
- */
- public DatabaseEntry getValue() {
- return value;
- }
-}
Deleted: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordIterator.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordIterator.java 2008-07-09 00:30:15 UTC (rev 2424)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordIterator.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -1,155 +0,0 @@
-/* BDBRecordIterator
- *
- * $Id$
- *
- * Created on 1:27:24 PM May 15, 2006.
- *
- * Copyright (C) 2006 Internet Archive.
- *
- * This file is part of wayback.
- *
- * wayback is free software; you can redistribute it and/or modify
- * it under the terms of the GNU Lesser Public License as published by
- * the Free Software Foundation; either version 2.1 of the License, or
- * any later version.
- *
- * wayback is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU Lesser Public License for more details.
- *
- * You should have received a copy of the GNU Lesser Public License
- * along with wayback; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
- */
-package org.archive.wayback.bdb;
-
-import java.util.NoSuchElementException;
-
-import org.archive.wayback.util.CloseableIterator;
-
-import com.sleepycat.je.Cursor;
-import com.sleepycat.je.DatabaseEntry;
-import com.sleepycat.je.DatabaseException;
-import com.sleepycat.je.LockMode;
-import com.sleepycat.je.OperationStatus;
-
-/**
- *
- *
- * @author brad
- * @version $Date$, $Revision$
- */
-public class BDBRecordIterator implements CloseableIterator<BDBRecord> {
-
- DatabaseEntry key;
- DatabaseEntry value;
- boolean hitLast;
- boolean gotNext;
- Cursor cursor;
- boolean backward;
- BDBRecord record;
-
- /**
- * @param cursor
- * @param search
- * @throws DatabaseException
- */
- public BDBRecordIterator(Cursor cursor,String search)
- throws DatabaseException {
- initialize(cursor,search,false);
- }
- /**
- * @param cursor
- * @param search
- * @param backward
- * @throws DatabaseException
- */
- public BDBRecordIterator(Cursor cursor,String search,boolean backward)
- throws DatabaseException {
- initialize(cursor,search,backward);
- }
- private void initialize(Cursor cursor,String search, boolean backward)
- throws DatabaseException {
- this.cursor = cursor;
- this.backward = backward;
- key = new DatabaseEntry();
- value = new DatabaseEntry();
- key.setData(search.getBytes());
- key.setPartial(false);
- OperationStatus status = cursor.getSearchKeyRange(key, value,
- LockMode.DEFAULT);
- if(backward && (status == OperationStatus.SUCCESS)) {
- // if we are in reverse, immediately back up one record:
- status = cursor.getPrev(key, value, LockMode.DEFAULT);
- }
- if(status == OperationStatus.SUCCESS) {
- gotNext = true;
- }
- record = new BDBRecord(key,value);
- }
-
- /* (non-Javadoc)
- * @see java.util.Iterator#hasNext()
- */
- public boolean hasNext() {
- if(hitLast) {
- return false;
- }
- if(cursor == null) {
- return false;
- }
- if(!gotNext) {
- OperationStatus status;
- // attempt to get the next:
- try {
- if(backward) {
- status = cursor.getPrev(key, value, LockMode.DEFAULT);
- } else {
- status = cursor.getNext(key, value, LockMode.DEFAULT);
- }
- if(status == OperationStatus.SUCCESS) {
- gotNext = true;
- } else {
- close();
- }
- } catch (DatabaseException e) {
- // SLOP: throw a runtime?
- e.printStackTrace();
- close();
- }
- }
- return gotNext;
- }
-
- public void close() {
- if(!hitLast) {
- hitLast = true;
- try {
- cursor.close();
- } catch (DatabaseException e) {
- // TODO what to do?
- // let's just eat it for now..
- e.printStackTrace();
- }
- }
- }
-
- /* (non-Javadoc)
- * @see java.util.Iterator#next()
- */
- public BDBRecord next() {
- if(!gotNext) {
- throw new NoSuchElementException();
- }
- gotNext = false;
- return record;
- }
-
- /* (non-Javadoc)
- * @see java.util.Iterator#remove()
- */
- public void remove() {
- throw new UnsupportedOperationException();
- }
-}
Deleted: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordSet.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordSet.java 2008-07-09 00:30:15 UTC (rev 2424)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordSet.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -1,263 +0,0 @@
-/* BDBRecordGenerator
- *
- * $Id$
- *
- * Created on 1:22:39 PM May 15, 2006.
- *
- * Copyright (C) 2006 Internet Archive.
- *
- * This file is part of wayback.
- *
- * wayback is free software; you can redistribute it and/or modify
- * it under the terms of the GNU Lesser Public License as published by
- * the Free Software Foundation; either version 2.1 of the License, or
- * any later version.
- *
- * wayback is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU Lesser Public License for more details.
- *
- * You should have received a copy of the GNU Lesser Public License
- * along with wayback; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
- */
-package org.archive.wayback.bdb;
-
-import java.io.File;
-import java.io.UnsupportedEncodingException;
-import java.util.Iterator;
-
-import com.sleepycat.je.Cursor;
-import com.sleepycat.je.Database;
-import com.sleepycat.je.DatabaseConfig;
-import com.sleepycat.je.DatabaseEntry;
-import com.sleepycat.je.DatabaseException;
-import com.sleepycat.je.Environment;
-import com.sleepycat.je.EnvironmentConfig;
-import com.sleepycat.je.LockMode;
-import com.sleepycat.je.OperationStatus;
-import com.sleepycat.je.Transaction;
-
-/**
- *
- *
- * @author brad
- * @version $Date$, $Revision$
- */
-public class BDBRecordSet {
- /**
- * Maximum BDBJE file size
- */
- private final static String JE_LOG_FILEMAX = "256000000";
- /**
- * path to directory containing the BDBJE files
- */
- private String path;
-
- /**
- * name of BDBJE db within the path directory
- */
- private String dbName;
-
- /**
- * BDBJE Environment
- */
- Environment env = null;
-
- /**
- * BDBJE Database
- */
- Database db = null;
-
- /**
- * @param thePath Directory where BDBJE files are stored
- * @param theDbName Name of files in thePath
- * @throws DatabaseException
- */
- public void initializeDB(final String thePath, final String theDbName)
- throws DatabaseException {
- path = thePath;
- dbName = theDbName;
-
- EnvironmentConfig environmentConfig = new EnvironmentConfig();
- environmentConfig.setAllowCreate(true);
- environmentConfig.setTransactional(true);
- environmentConfig.setConfigParam("je.log.fileMax",JE_LOG_FILEMAX);
- File file = new File(path);
- if(!file.isDirectory()) {
- if(!file.mkdirs()) {
- throw new DatabaseException("failed mkdirs(" + path + ")");
- }
- }
- env = new Environment(file, environmentConfig);
- DatabaseConfig databaseConfig = new DatabaseConfig();
- databaseConfig.setAllowCreate(true);
- databaseConfig.setTransactional(true);
- // perform other database configurations
-
- db = env.openDatabase(null, dbName, databaseConfig);
- }
-
- /**
- * shut down the BDB.
- *
- * @throws DatabaseException
- */
- public synchronized void shutdownDB() throws DatabaseException {
-
- if (db != null) {
- db.close();
- db = null;
- }
-
- if (env != null) {
- env.close();
- env = null;
- }
- }
-
- /**
- * @param s
- * @return byte array representation of String s in UTF-8
- */
- public static byte[] stringToBytes(String s) {
- try {
- return s.getBytes("UTF-8");
- } catch (UnsupportedEncodingException e) {
- // no UTF-8, huh?
- e.printStackTrace();
- return s.getBytes();
- }
- }
- /**
- * @param ba
- * @return String of UTF-8 encoded bytes ba
- */
- public static String bytesToString(byte[] ba) {
- try {
- return new String(ba,"UTF-8");
- } catch (UnsupportedEncodingException e) {
- // not likely..
- e.printStackTrace();
- return new String(ba);
- }
- }
-
- /**
- * @param startKey
- * @return iterator for BDBRecords
- * @throws DatabaseException
- */
- public BDBRecordIterator recordIterator(final String startKey)
- throws DatabaseException {
- return recordIterator(startKey,true);
- }
- /**
- * @param startKey
- * @param forward
- * @return iterator for BDBRecords
- * @throws DatabaseException
- */
- public BDBRecordIterator recordIterator(final String startKey,
- final boolean forward) throws DatabaseException {
- Cursor cursor = db.openCursor(null, null);
- return new BDBRecordIterator(cursor,startKey,!forward);
- }
-
- /**
- * @param itr
- */
- public void insertRecords(final Iterator<BDBRecord> itr) {
- OperationStatus status = null;
- try {
- Transaction txn = env.beginTransaction(null, null);
- try {
- Cursor cursor = db.openCursor(txn, null);
- while (itr.hasNext()) {
- BDBRecord record = (BDBRecord) itr.next();
- status = cursor.put(record.getKey(), record.getValue());
- if (status != OperationStatus.SUCCESS) {
- throw new RuntimeException("put() non-success status");
- }
- }
- cursor.close();
- txn.commit();
- } catch (DatabaseException e) {
- if(txn != null) {
- txn.abort();
- }
- e.printStackTrace();
- }
- } catch (DatabaseException e) {
- e.printStackTrace();
- }
- }
-
- /**
- * persistantly store key-value pair
- * @param keyStr
- * @param valueStr
- * @throws DatabaseException
- */
- public void put(String keyStr, String valueStr) throws DatabaseException {
- DatabaseEntry key = new DatabaseEntry(stringToBytes(keyStr));
- DatabaseEntry data = new DatabaseEntry(stringToBytes(valueStr));
- db.put(null, key, data);
- }
-
- /**
- * retrieve the value assoicated with keyStr from persistant storage
- * @param keyStr
- * @return String value associated with key, or null if no key is found
- * or an error occurs
- * @throws DatabaseException
- */
- public String get(String keyStr) throws DatabaseException {
- String result = null;
- DatabaseEntry key = new DatabaseEntry(stringToBytes(keyStr));
- DatabaseEntry data = new DatabaseEntry();
- if (db.get(null, key, data, LockMode.DEFAULT)
- == OperationStatus.SUCCESS) {
-
- result = bytesToString(data.getData());
- }
- return result;
- }
-
- /**
- * @param keyStr
- * @throws DatabaseException
- */
- public void delete(String keyStr) throws DatabaseException {
- db.delete(null,new DatabaseEntry(stringToBytes(keyStr)));
- }
-
- /**
- * @return Returns the dbName.
- */
- public String getDbName() {
- return dbName;
- }
-
- /**
- * @return Returns the path.
- */
- public String getPath() {
- return path;
- }
-
- /**
- * @param path the path to set
- */
- public void setPath(String path) {
- this.path = path;
- }
-
- /**
- * @param dbName the dbName to set
- */
- public void setDbName(String dbName) {
- this.dbName = dbName;
- }
-}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/BDBIndex.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/BDBIndex.java 2008-07-09 00:30:15 UTC (rev 2424)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/BDBIndex.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -31,8 +31,6 @@
import java.util.Iterator;
import org.archive.wayback.UrlCanonicalizer;
-import org.archive.wayback.bdb.BDBRecord;
-import org.archive.wayback.bdb.BDBRecordSet;
import org.archive.wayback.core.CaptureSearchResult;
import org.archive.wayback.exception.ConfigurationException;
import org.archive.wayback.exception.ResourceIndexNotAvailableException;
@@ -42,6 +40,8 @@
import org.archive.wayback.util.AdaptedIterator;
import org.archive.wayback.util.Adapter;
import org.archive.wayback.util.CloseableIterator;
+import org.archive.wayback.util.bdb.BDBRecord;
+import org.archive.wayback.util.bdb.BDBRecordSet;
import org.archive.wayback.util.flatfile.RecordIterator;
import org.archive.wayback.util.url.AggressiveUrlCanonicalizer;
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/BDBRecordToSearchResultAdapter.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/BDBRecordToSearchResultAdapter.java 2008-07-09 00:30:15 UTC (rev 2424)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/BDBRecordToSearchResultAdapter.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -26,10 +26,10 @@
import java.io.UnsupportedEncodingException;
-import org.archive.wayback.bdb.BDBRecord;
import org.archive.wayback.core.CaptureSearchResult;
import org.archive.wayback.resourceindex.cdx.CDXLineToSearchResultAdapter;
import org.archive.wayback.util.Adapter;
+import org.archive.wayback.util.bdb.BDBRecord;
/**
* Adapter that converts a BDBRecord into a SearchResult
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/SearchResultToBDBRecordAdapter.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/SearchResultToBDBRecordAdapter.java 2008-07-09 00:30:15 UTC (rev 2424)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/bdb/SearchResultToBDBRecordAdapter.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -28,10 +28,10 @@
import org.apache.commons.httpclient.URIException;
import org.archive.wayback.UrlCanonicalizer;
-import org.archive.wayback.bdb.BDBRecord;
-import org.archive.wayback.bdb.BDBRecordSet;
import org.archive.wayback.core.CaptureSearchResult;
import org.archive.wayback.util.Adapter;
+import org.archive.wayback.util.bdb.BDBRecord;
+import org.archive.wayback.util.bdb.BDBRecordSet;
import com.sleepycat.je.DatabaseEntry;
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/BDBResourceFileLocationDB.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/BDBResourceFileLocationDB.java 2008-07-09 00:30:15 UTC (rev 2424)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/BDBResourceFileLocationDB.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -28,10 +28,10 @@
import java.io.IOException;
import java.io.InputStreamReader;
-import org.archive.wayback.bdb.BDBRecordSet;
import org.archive.wayback.resourcestore.locationdb.ResourceFileLocationDB;
import org.archive.wayback.resourcestore.locationdb.ResourceFileLocationDBLog;
import org.archive.wayback.util.CloseableIterator;
+import org.archive.wayback.util.bdb.BDBRecordSet;
import com.sleepycat.je.DatabaseException;
Copied: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecord.java (from rev 2418, trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecord.java)
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecord.java (rev 0)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecord.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -0,0 +1,58 @@
+/* BDBRecord
+ *
+ * $Id$
+ *
+ * Created on 1:43:15 PM May 15, 2006.
+ *
+ * Copyright (C) 2006 Internet Archive.
+ *
+ * This file is part of wayback.
+ *
+ * wayback is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser Public License as published by
+ * the Free Software Foundation; either version 2.1 of the License, or
+ * any later version.
+ *
+ * wayback is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser Public License
+ * along with wayback; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+package org.archive.wayback.util.bdb;
+
+import com.sleepycat.je.DatabaseEntry;
+
+/**
+ *
+ *
+ * @author brad
+ * @version $Date$, $Revision$
+ */
+public class BDBRecord {
+ private DatabaseEntry key;
+ private DatabaseEntry value;
+ /**
+ * @param key
+ * @param value
+ */
+ public BDBRecord(DatabaseEntry key, DatabaseEntry value) {
+ this.key = key;
+ this.value = value;
+ }
+ /**
+ * @return Returns the key.
+ */
+ public DatabaseEntry getKey() {
+ return key;
+ }
+ /**
+ * @return Returns the value.
+ */
+ public DatabaseEntry getValue() {
+ return value;
+ }
+}
Copied: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecordIterator.java (from rev 2418, trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordIterator.java)
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecordIterator.java (rev 0)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecordIterator.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -0,0 +1,155 @@
+/* BDBRecordIterator
+ *
+ * $Id$
+ *
+ * Created on 1:27:24 PM May 15, 2006.
+ *
+ * Copyright (C) 2006 Internet Archive.
+ *
+ * This file is part of wayback.
+ *
+ * wayback is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser Public License as published by
+ * the Free Software Foundation; either version 2.1 of the License, or
+ * any later version.
+ *
+ * wayback is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser Public License
+ * along with wayback; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+package org.archive.wayback.util.bdb;
+
+import java.util.NoSuchElementException;
+
+import org.archive.wayback.util.CloseableIterator;
+
+import com.sleepycat.je.Cursor;
+import com.sleepycat.je.DatabaseEntry;
+import com.sleepycat.je.DatabaseException;
+import com.sleepycat.je.LockMode;
+import com.sleepycat.je.OperationStatus;
+
+/**
+ *
+ *
+ * @author brad
+ * @version $Date$, $Revision$
+ */
+public class BDBRecordIterator implements CloseableIterator<BDBRecord> {
+
+ DatabaseEntry key;
+ DatabaseEntry value;
+ boolean hitLast;
+ boolean gotNext;
+ Cursor cursor;
+ boolean backward;
+ BDBRecord record;
+
+ /**
+ * @param cursor
+ * @param search
+ * @throws DatabaseException
+ */
+ public BDBRecordIterator(Cursor cursor,String search)
+ throws DatabaseException {
+ initialize(cursor,search,false);
+ }
+ /**
+ * @param cursor
+ * @param search
+ * @param backward
+ * @throws DatabaseException
+ */
+ public BDBRecordIterator(Cursor cursor,String search,boolean backward)
+ throws DatabaseException {
+ initialize(cursor,search,backward);
+ }
+ private void initialize(Cursor cursor,String search, boolean backward)
+ throws DatabaseException {
+ this.cursor = cursor;
+ this.backward = backward;
+ key = new DatabaseEntry();
+ value = new DatabaseEntry();
+ key.setData(search.getBytes());
+ key.setPartial(false);
+ OperationStatus status = cursor.getSearchKeyRange(key, value,
+ LockMode.DEFAULT);
+ if(backward && (status == OperationStatus.SUCCESS)) {
+ // if we are in reverse, immediately back up one record:
+ status = cursor.getPrev(key, value, LockMode.DEFAULT);
+ }
+ if(status == OperationStatus.SUCCESS) {
+ gotNext = true;
+ }
+ record = new BDBRecord(key,value);
+ }
+
+ /* (non-Javadoc)
+ * @see java.util.Iterator#hasNext()
+ */
+ public boolean hasNext() {
+ if(hitLast) {
+ return false;
+ }
+ if(cursor == null) {
+ return false;
+ }
+ if(!gotNext) {
+ OperationStatus status;
+ // attempt to get the next:
+ try {
+ if(backward) {
+ status = cursor.getPrev(key, value, LockMode.DEFAULT);
+ } else {
+ status = cursor.getNext(key, value, LockMode.DEFAULT);
+ }
+ if(status == OperationStatus.SUCCESS) {
+ gotNext = true;
+ } else {
+ close();
+ }
+ } catch (DatabaseException e) {
+ // SLOP: throw a runtime?
+ e.printStackTrace();
+ close();
+ }
+ }
+ return gotNext;
+ }
+
+ public void close() {
+ if(!hitLast) {
+ hitLast = true;
+ try {
+ cursor.close();
+ } catch (DatabaseException e) {
+ // TODO what to do?
+ // let's just eat it for now..
+ e.printStackTrace();
+ }
+ }
+ }
+
+ /* (non-Javadoc)
+ * @see java.util.Iterator#next()
+ */
+ public BDBRecord next() {
+ if(!gotNext) {
+ throw new NoSuchElementException();
+ }
+ gotNext = false;
+ return record;
+ }
+
+ /* (non-Javadoc)
+ * @see java.util.Iterator#remove()
+ */
+ public void remove() {
+ throw new UnsupportedOperationException();
+ }
+}
Copied: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecordSet.java (from rev 2418, trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/bdb/BDBRecordSet.java)
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecordSet.java (rev 0)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/util/bdb/BDBRecordSet.java 2008-07-10 18:41:53 UTC (rev 2425)
@@ -0,0 +1,264 @@
+/* BDBRecordGenerator
+ *
+ * $Id$
+ *
+ * Created on 1:22:39 PM May 15, 2006.
+ *
+ * Copyright (C) 2006 Internet Archive.
+ *
+ * This file is part of wayback.
+ *
+ * wayback is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser Public License as published by
+ * the Free Software Foundation; either version 2.1 of the License, or
+ * any later version.
+ *
+ * wayback is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser Public License
+ * along with wayback; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+package org.archive.wayback.util.bdb;
+
+import java.io.File;
+import java.io.UnsupportedEncodingException;
+import java.util.Iterator;
+
+
+import com.sleepycat.je.Cursor;
+import com.sleepycat.je.Database;
+import com.sleepycat.je.DatabaseConfig;
+import com.sleepycat.je.DatabaseEntry;
+import com.sleepycat.je.DatabaseException;
+import com.sleepycat.je.Environment;
+import com.sleepycat.je.EnvironmentConfig;
+import com.sleepycat.je.LockMode;
+import com.sleepycat.je.OperationStatus;
+import com.sleepycat.je.Transaction;
+
+/**
+ *
+ *
+ * @author brad
+ * @version $Date$, $Revision$
+ */
+public class BDBRecordSet {
+ /**
+ * Maximum BDBJE file size
+ */
+ private final static String JE_LOG_FILEMAX = "256000000";
+ /**
+ * path to directory containing the BDBJE files
+ */
+ private String path;
+
+ /**
+ * name of BDBJE db within the path directory
+ */
+ private String dbName;
+
+ /**
+ * BDBJE Environment
+ */
+ Environment env = null;
+
+ /**
+ * BDBJE Database
+ */
+ Database db = null;
+
+ /**
+ * @param thePath Directory where BDBJE files are stored
+ * @param theDbName Name of files in thePath
+ * @throws DatabaseException
+ */
+ public void initializeDB(final String thePath, final String theDbName)
+ throws DatabaseException {
+ path = thePath;
+ dbName = theDbName;
+
+ EnvironmentConfig environmentConfig = new EnvironmentConfig();
+ environmentConfig.setAllowCreate(true);
+ environmentConfig.setTransactional(true);
+ environmentConfig.setConfigParam("je.log.fileMax",JE_LOG_FILEMAX);
+ File file = new File(path);
+ if(!file.isDirectory()) {
+ if(!file.mkdirs()) {
+ throw new DatabaseException("failed mkdirs(" + path + ")");
+ }
+ }
+ env = new Environment(file, environmentConfig);
+ DatabaseConfig databaseConfig = new DatabaseConfig();
+ databaseConfig.setAllowCreate(true);
+ databaseConfig.setTransactional(true);
+ // perform other database configurations
+
+ db = env.openDatabase(null, dbName, databaseConfig);
+ }
+
+ /**
+ * shut down the BDB.
+ *
+ * @throws DatabaseException
+ */
+ public synchronized void shutdownDB() throws DatabaseException {
+
+ if (db != null) {
+ db.close();
+ db = null;
+ }
+
+ if (env != null) {
+ env.close();
+ env = null;
+ }
+ }
+
+ /**
+ * @param s
+ * @return byte array representation of String s in UTF-8
+ */
+ public static byte[] stringToBytes(String s) {
+ try {
+ return s.getBytes("UTF-8");
+ } catch (UnsupportedEncodingException e) {
+ // no UTF-8, huh?
+ e.printStackTrace();
+ return s.getBytes();
+ }
+ }
+ /**
+ * @param ba
+ * @return String of UTF-8 encoded bytes ba
+ */
+ public static String bytesToString(byte[] ba) {
+ try {
+ return new String(ba,"UTF-8");
+ } catch (UnsupportedEncodingException e) {
+ // not likely..
+ e.printStackTrace();
+ return new String(ba);
+ }
+ }
+
+ /**
+ * @param startKey
+ * @return iterator for BDBRecords
+ * @throws DatabaseException
+ */
+ public BDBRecordIterator recordIterator(final String startKey)
+ throws DatabaseException {
+ return recordIterator(startKey,true);
+ }
+ /**
+ * @param startKey
+ * @param forward
+ * @return iterator for BDBRecords
+ * @throws DatabaseException
+ */
+ public BDBRecordIterator recordIterator(final String startKey,
+ final boolean forward) throws DatabaseException {
+ Cursor cursor = db.openCursor(null, null);
+ return new BDBRecordIterator(cursor,startKey,!forward);
+ }
+
+ /**
+ * @param itr
+ */
+ public void insertRecords(final Iterator<BDBRecord> itr) {
+ OperationStatus status = null;
+ try {
+ Transaction txn = env.beginTransaction(null, null);
+ try {
+ Cursor cursor = db.openCursor(txn, null);
+ while (itr.hasNext()) {
+ BDBRecord record = (BDBRecord) itr.next();
+ status = cursor.put(record.getKey(), record.getValue());
+ if (status != OperationStatus.SUCCESS) {
+ throw new RuntimeException("put() non-success status");
+ }
+ }
+ cursor.close();
+ txn.commit();
+ } catch (DatabaseException e) {
+ if(txn != null) {
+ txn.abort();
+ }
+ e.printStackTrace();
+ }
+ } catch (DatabaseException e) {
+ e.printStackTrace();
+ }
+ }
+
+ /**
+ * persistantly store key-value pair
+ * @param keyStr
+ * @param valueStr
+ * @throws DatabaseException
+ */
+ public void put(String keyStr, String valueStr) throws DatabaseException {
+ DatabaseEntry key = new DatabaseEntry(stringToBytes(keyStr));
+ DatabaseEntry data = new DatabaseEntry(stringToBytes(valueStr));
+ db.put(null, key, data);
+ }
+
+ /**
+ * retrieve the value assoicated with keyStr from persistant storage
+ * @param keyStr
+ * @return String value associated with key, or null if no key is found
+ * or an error occurs
+ * @throws DatabaseException
+ */
+ public String get(String keyStr) throws DatabaseException {
+ String result = null;
+ DatabaseEntry key = new DatabaseEntry(stringToBytes(keyStr));
+ DatabaseEntry data = new DatabaseEntry();
+ if (db.get(null, key, data, LockMode.DEFAULT)
+ == OperationStatus.SUCCESS) {
+
+ result = bytesToString(data.getData());
+ }
+ return result;
+ }
+
+ /**
+ * @param keyStr
+ * @throws DatabaseException
+ */
+ public void delete(String keyStr) throws DatabaseException {
+ db.delete(null,new DatabaseEntry(stringToBytes(keyStr)));
+ }
+
+ /**
+ * @return Returns the dbName.
+ */
+ public String getDbName() {
+ return dbName;
+ }
+
+ /**
+ * @return Returns the path.
+ */
+ public String getPath() {
+ return path;
+ }
+
+ /**
+ * @param path the path to set
+ */
+ public void setPath(String path) {
+ this.path = path;
+ }
+
+ /**
+ * @param dbName the dbName to set
+ */
+ public void setDbName(String dbName) {
+ this.dbName = dbName;
+ }
+}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-09 00:30:06
|
Revision: 2424
http://archive-access.svn.sourceforge.net/archive-access/?rev=2424&view=rev
Author: bradtofel
Date: 2008-07-08 17:30:15 -0700 (Tue, 08 Jul 2008)
Log Message:
-----------
FEATURE: added Shutdownable interface, which all worker threads now implement.
added configurable list of Shutdownable objects to WaybackCollection, which will be shutdown() when the WaybackCollection is shutdown()
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/updater/LocalResourceIndexUpdater.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexQueueUpdater.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexWorker.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/ResourceFileLocationDBUpdater.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/resourcefile/ResourceFileSourceUpdater.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/WaybackCollection.java
Added Paths:
-----------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/Shutdownable.java
Added: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/Shutdownable.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/Shutdownable.java (rev 0)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/Shutdownable.java 2008-07-09 00:30:15 UTC (rev 2424)
@@ -0,0 +1,37 @@
+/* Shutdownable
+ *
+ * $Id$
+ *
+ * Created on 4:54:13 PM Jul 8, 2008.
+ *
+ * Copyright (C) 2008 Internet Archive.
+ *
+ * This file is part of wayback.
+ *
+ * wayback is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser Public License as published by
+ * the Free Software Foundation; either version 2.1 of the License, or
+ * any later version.
+ *
+ * wayback is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser Public License
+ * along with wayback; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+package org.archive.wayback;
+
+import java.io.IOException;
+
+/**
+ *
+ *
+ * @author brad
+ * @version $Date$, $Revision$
+ */
+public interface Shutdownable {
+ public void shutdown() throws IOException;
+}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/updater/LocalResourceIndexUpdater.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/updater/LocalResourceIndexUpdater.java 2008-07-09 00:26:19 UTC (rev 2423)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/updater/LocalResourceIndexUpdater.java 2008-07-09 00:30:15 UTC (rev 2424)
@@ -28,6 +28,7 @@
import java.io.IOException;
import java.util.logging.Logger;
+import org.archive.wayback.Shutdownable;
import org.archive.wayback.core.CaptureSearchResult;
import org.archive.wayback.exception.ConfigurationException;
import org.archive.wayback.resourceindex.LocalResourceIndex;
@@ -48,7 +49,7 @@
* @author brad
* @version $Date$, $Revision$
*/
-public class LocalResourceIndexUpdater {
+public class LocalResourceIndexUpdater implements Shutdownable {
/**
* Logger for this class
*/
@@ -71,7 +72,7 @@
* Thread object of update thread -- also is flag indicating if the thread
* has already been started. Access to it is synchronized.
*/
- private Thread updateThread = null;
+ private Thread thread = null;
/**
* start the background index merging thread
@@ -88,10 +89,21 @@
throw new ConfigurationException("No incoming");
}
if(runInterval > 0) {
- updateThread = new UpdateThread(this,runInterval);
- updateThread.start();
+ thread = new UpdateThread(this,runInterval);
+ thread.start();
}
}
+
+ public void shutdown() {
+ if(thread != null) {
+ thread.interrupt();
+ try {
+ thread.join(1000);
+ } catch (InterruptedException e) {
+ e.printStackTrace();
+ }
+ }
+ }
private boolean mergeFile(File cdxFile) {
boolean added = false;
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexQueueUpdater.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexQueueUpdater.java 2008-07-09 00:26:19 UTC (rev 2423)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexQueueUpdater.java 2008-07-09 00:30:15 UTC (rev 2424)
@@ -31,6 +31,7 @@
import java.io.PrintWriter;
import java.util.logging.Logger;
+import org.archive.wayback.Shutdownable;
import org.archive.wayback.resourcestore.locationdb.ResourceFileLocationDB;
import org.archive.wayback.util.CloseableIterator;
import org.archive.wayback.util.DirMaker;
@@ -45,7 +46,7 @@
* @author brad
* @version $Date$, $Revision$
*/
-public class IndexQueueUpdater {
+public class IndexQueueUpdater implements Shutdownable {
private static final Logger LOGGER =
Logger.getLogger(IndexQueueUpdater.class.getName());
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexWorker.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexWorker.java 2008-07-09 00:26:19 UTC (rev 2423)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexWorker.java 2008-07-09 00:30:15 UTC (rev 2424)
@@ -27,6 +27,7 @@
import java.io.IOException;
import java.util.logging.Logger;
+import org.archive.wayback.Shutdownable;
import org.archive.wayback.UrlCanonicalizer;
import org.archive.wayback.core.CaptureSearchResult;
import org.archive.wayback.resourceindex.updater.IndexClient;
@@ -45,7 +46,7 @@
* @author brad
* @version $Date$, $Revision$
*/
-public class IndexWorker {
+public class IndexWorker implements Shutdownable {
private static final Logger LOGGER =
Logger.getLogger(IndexWorker.class.getName());
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/ResourceFileLocationDBUpdater.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/ResourceFileLocationDBUpdater.java 2008-07-09 00:26:19 UTC (rev 2423)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/ResourceFileLocationDBUpdater.java 2008-07-09 00:30:15 UTC (rev 2424)
@@ -29,6 +29,7 @@
import java.util.Iterator;
import java.util.logging.Logger;
+import org.archive.wayback.Shutdownable;
import org.archive.wayback.resourcestore.resourcefile.ResourceFileList;
import org.archive.wayback.resourcestore.resourcefile.ResourceFileLocation;
import org.archive.wayback.util.DirMaker;
@@ -52,7 +53,7 @@
* @author brad
* @version $Date$, $Revision$
*/
-public class ResourceFileLocationDBUpdater {
+public class ResourceFileLocationDBUpdater implements Shutdownable {
private static final Logger LOGGER =
Logger.getLogger(ResourceFileLocationDBUpdater.class.getName());
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/resourcefile/ResourceFileSourceUpdater.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/resourcefile/ResourceFileSourceUpdater.java 2008-07-09 00:26:19 UTC (rev 2423)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/resourcefile/ResourceFileSourceUpdater.java 2008-07-09 00:30:15 UTC (rev 2424)
@@ -29,6 +29,7 @@
import java.util.List;
import java.util.logging.Logger;
+import org.archive.wayback.Shutdownable;
import org.archive.wayback.resourcestore.locationdb.ResourceFileLocationDBUpdater;
import org.archive.wayback.util.DirMaker;
@@ -44,7 +45,7 @@
* @author brad
* @version $Date$, $Revision$
*/
-public class ResourceFileSourceUpdater {
+public class ResourceFileSourceUpdater implements Shutdownable {
private static final Logger LOGGER =
Logger.getLogger(ResourceFileSourceUpdater.class.getName());
private List<ResourceFileSource> sources = null;
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/WaybackCollection.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/WaybackCollection.java 2008-07-09 00:26:19 UTC (rev 2423)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/WaybackCollection.java 2008-07-09 00:30:15 UTC (rev 2424)
@@ -25,9 +25,11 @@
package org.archive.wayback.webapp;
import java.io.IOException;
+import java.util.List;
import org.archive.wayback.ResourceIndex;
import org.archive.wayback.ResourceStore;
+import org.archive.wayback.Shutdownable;
import org.archive.wayback.exception.ConfigurationException;
/**
@@ -40,7 +42,27 @@
public class WaybackCollection {
private ResourceStore resourceStore = null;
private ResourceIndex resourceIndex = null;
+ private List<Shutdownable> shutdownables = null;
private boolean shutdownDone = false;
+
+ public void shutdown() throws IOException {
+ if(shutdownDone) {
+ return;
+ }
+ if(resourceStore != null) {
+ resourceStore.shutdown();
+ }
+ if(resourceIndex != null) {
+ resourceIndex.shutdown();
+ }
+ if(shutdownables != null) {
+ for(Shutdownable s : shutdownables) {
+ s.shutdown();
+ }
+ }
+ shutdownDone = true;
+ }
+
public ResourceStore getResourceStore() throws ConfigurationException {
if(resourceStore == null) {
throw new ConfigurationException("No resourceStore declared");
@@ -59,16 +81,12 @@
public void setResourceIndex(ResourceIndex resourceIndex) {
this.resourceIndex = resourceIndex;
}
- public void shutdown() throws IOException {
- if(shutdownDone) {
- return;
- }
- if(resourceStore != null) {
- resourceStore.shutdown();
- }
- if(resourceIndex != null) {
- resourceIndex.shutdown();
- }
- shutdownDone = true;
+
+ public List<Shutdownable> getShutdownables() {
+ return shutdownables;
}
+
+ public void setShutdownables(List<Shutdownable> shutdownables) {
+ this.shutdownables = shutdownables;
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-09 00:26:10
|
Revision: 2423
http://archive-access.svn.sourceforge.net/archive-access/?rev=2423&view=rev
Author: bradtofel
Date: 2008-07-08 17:26:19 -0700 (Tue, 08 Jul 2008)
Log Message:
-----------
INTERFACE: now use CaptureSearchResults.markClosest()
COMMENT: javadoc update
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java 2008-07-08 23:46:16 UTC (rev 2422)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java 2008-07-09 00:26:19 UTC (rev 2423)
@@ -61,11 +61,15 @@
* within a ServletContext, including holding references to the
* implementation instances of the primary Wayback classes:
*
- * ResourceIndex
- * ResourceStore
- * QueryUI
- * ReplayUI
+ * RequestParser
+ * ResourceIndex(via WaybackCollection)
+ * ResourceStore(via WaybackCollection)
+ * QueryRenderer
+ * ReplayDispatcher
+ * ExceptionRenderer
+ * ResultURIConverter
*
+ *
* @author brad
* @version $Date$, $Revision$
*/
@@ -331,8 +335,7 @@
SearchResults results = collection.getResourceIndex().query(wbRequest);
if(results instanceof CaptureSearchResults) {
CaptureSearchResults cResults = (CaptureSearchResults) results;
- CaptureSearchResult closest = cResults.getClosest(wbRequest);
- closest.setClosest(true);
+ cResults.markClosest(wbRequest);
query.renderCaptureResults(httpRequest,httpResponse,wbRequest,
cResults,uriConverter);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-08 23:46:24
|
Revision: 2422
http://archive-access.svn.sourceforge.net/archive-access/?rev=2422&view=rev
Author: bradtofel
Date: 2008-07-08 16:46:16 -0700 (Tue, 08 Jul 2008)
Log Message:
-----------
COMMENT: javadoc update
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-08 23:45:08 UTC (rev 2421)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-08 23:46:16 UTC (rev 2422)
@@ -162,8 +162,9 @@
public static final String REQUEST_START_DATE = "startdate";
/**
- * GUARANTEED PRESENT for Replay requests only, no meaning for Query
- * requests.
+ * GUARANTEED PRESENT for Replay requests only. If present for Query
+ * requests, then it will be interpreted as a partial timestamp for missing
+ * REQUEST_START_DATE and REQUEST_END_DATE fields.
* Original (RAW/possibly partial) 14-digit timestamp of date requested for
* Replay
*/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
Revision: 2421
http://archive-access.svn.sourceforge.net/archive-access/?rev=2421&view=rev
Author: bradtofel
Date: 2008-07-08 16:45:08 -0700 (Tue, 08 Jul 2008)
Log Message:
-----------
INTERFACE: now works with NutchWax 0.12.1 xml results.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/NutchResourceIndex.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/NutchResourceIndex.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/NutchResourceIndex.java 2008-07-08 22:03:16 UTC (rev 2420)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/NutchResourceIndex.java 2008-07-08 23:45:08 UTC (rev 2421)
@@ -28,6 +28,8 @@
import java.io.IOException;
import java.io.UnsupportedEncodingException;
+import java.util.ArrayList;
+import java.util.List;
import java.util.logging.Logger;
import javax.xml.parsers.DocumentBuilder;
@@ -71,11 +73,14 @@
private DocumentBuilder builder;
private static final String NUTCH_ARCNAME = "arcname";
private static final String NUTCH_ARCOFFSET = "arcoffset";
- private static final String NUTCH_ARCDATE = "tstamp";
- private static final String NUTCH_ARCDATE_ALT = "arcdate";
+// private static final String NUTCH_FILENAME = "filename";
+// private static final String NUTCH_FILEOFFSET = "fileoffset";
+ private static final String NUTCH_ARCDATE = "date";
+// private static final String NUTCH_ARCDATE_ALT = "arcdate";
private static final String NUTCH_DIGEST = "digest";
- private static final String NUTCH_PRIMARY_TYPE = "primaryType";
- private static final String NUTCH_SUB_TYPE = "subType";
+ private static final String NUTCH_MIME_TYPE = "type";
+// private static final String NUTCH_PRIMARY_TYPE = "primaryType";
+// private static final String NUTCH_SUB_TYPE = "subType";
// private static final String NUTCH_CAPTURE_HOST = "site";
private static final String NUTCH_CAPTURE_URL = "link";
@@ -117,6 +122,7 @@
Document document = null;
try {
// HTTP Request + parse
+ LOGGER.info("Requesting OpenSearch: " + requestUrl);
document = getHttpDocument(requestUrl);
} catch (IOException e) {
// TODO: better error for user:
@@ -132,8 +138,10 @@
if(wbRequest.isReplayRequest() || wbRequest.isCaptureQueryRequest()) {
results = new CaptureSearchResults();
} else {
- // TODO: this is wrong, but needs exploration into what NutchWax can actually do.
- throw new BadQueryException("Unable to perform path prefix requests with this index type");
+ // TODO: this is wrong, but needs exploration into what NutchWax
+ // can actually do.
+ throw new BadQueryException("Unable to perform path " +
+ "prefix requests with this index type");
}
NodeList channel = getSearchChannel(document);
NodeList nodes = getSearchItems(document);
@@ -154,8 +162,12 @@
Element e = (Element) nodes.item(i);
- CaptureSearchResult result = elementToSearchResult(e);
- results.addSearchResult(result);
+ List<CaptureSearchResult> resultsList = itemToSearchResults(e);
+ if(resultsList != null) {
+ for(CaptureSearchResult result : resultsList) {
+ results.addSearchResult(result);
+ }
+ }
}
Element channelElement = (Element) channel.item(0);
@@ -179,42 +191,45 @@
return results;
}
- private CaptureSearchResult elementToSearchResult(Element e)
+ private List<CaptureSearchResult> itemToSearchResults(Element e)
throws ResourceIndexNotAvailableException {
- CaptureSearchResult result = new CaptureSearchResult();
+ String fileName = getNodeNutchContent(e,NUTCH_ARCNAME);
+ String httpCode = NUTCH_DEFAULT_HTTP_CODE;
+ String digest = getNodeNutchContent(e,NUTCH_DIGEST);
+ String mimeType = getNodeNutchContent(e,NUTCH_MIME_TYPE);
+ String offsetStr = getNodeNutchContent(e,NUTCH_ARCOFFSET);
+ long offset = 0;
+ if(offsetStr != null && offsetStr.length() > 0) {
+ offset = Long.parseLong(offsetStr);
+ }
+ String redirectUrl = NUTCH_DEFAULT_REDIRECT_URL;
+ String originalUrl = getNodeContent(e,NUTCH_CAPTURE_URL);
+ String urlKey = originalUrl;
+
+ NodeList nodes = e.getElementsByTagNameNS(NUTCH_NS,NUTCH_ARCDATE);
+ int numDates = nodes.getLength();
+ ArrayList<CaptureSearchResult> results = null;
- result.setFile(getNodeNutchContent(e,NUTCH_ARCNAME));
+ if(numDates > 0) {
+ results = new ArrayList<CaptureSearchResult>();
- // The date in nutchwax is now named 'tstamp' and its
- // 17 characters rather than 14. Pass first 14 only.
- String d = getNodeNutchContent(e,NUTCH_ARCDATE);
- if(d == null) {
- d = getNodeNutchContent(e,NUTCH_ARCDATE_ALT);
- }
- if(d == null) {
- throw new ResourceIndexNotAvailableException("Missing arcdate field in search results");
- }
- if (d.length() == 17) {
- d = d.substring(0, 14);
- }
- result.setCaptureTimestamp(d);
-
- //result.put(WaybackConstants.RESULT_HTTP_CODE,getNodeContent(e,""));
- result.setHttpCode(NUTCH_DEFAULT_HTTP_CODE);
- result.setDigest(getNodeNutchContent(e,NUTCH_DIGEST));
-
- result.setMimeType(getNodeNutchContent(e,NUTCH_PRIMARY_TYPE) + "/" +
- getNodeNutchContent(e,NUTCH_SUB_TYPE));
-
- result.setOffset(Long.parseLong(getNodeNutchContent(e,NUTCH_ARCOFFSET)));
-
- result.setRedirectUrl(NUTCH_DEFAULT_REDIRECT_URL);
- result.setCaptureTimestamp(getNodeContent(e,NUTCH_CAPTURE_URL));
- result.setOriginalUrl(getNodeContent(e,NUTCH_CAPTURE_URL));
- result.setUrlKey(getNodeContent(e,NUTCH_CAPTURE_URL));
-
- return result;
+ for(int i = 0; i < numDates; i++) {
+ String captureDate = nodes.item(i).getTextContent();
+ CaptureSearchResult result = new CaptureSearchResult();
+ result.setFile(fileName);
+ result.setCaptureTimestamp(captureDate);
+ result.setHttpCode(httpCode);
+ result.setDigest(digest);
+ result.setMimeType(mimeType);
+ result.setOffset(offset);
+ result.setRedirectUrl(redirectUrl);
+ result.setOriginalUrl(originalUrl);
+ result.setUrlKey(urlKey);
+ results.add(result);
+ }
+ }
+ return results;
}
protected NodeList getSearchChannel(Document d) {
@@ -271,13 +286,13 @@
ms.append("date%3A").append(startDateStr).append('-').append(endDateStr);
ms.append('+');
// Add 'url:URL'.
- if(wbRequest.isUrlQueryRequest()) {
+// if(wbRequest.isUrlQueryRequest()) {
ms.append("url%3A");
- } else {
- ms.append("exacturl%3A");
- }
+// } else {
+// ms.append("exacturl%3A");
+// }
try {
- ms.append(java.net.URLEncoder.encode(urlStr, "UTF-8"));
+ ms.append(java.net.URLEncoder.encode("\""+urlStr+"\"", "UTF-8"));
} catch (UnsupportedEncodingException e) {
throw new BadQueryException(e.toString());
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-08 22:03:13
|
Revision: 2420
http://archive-access.svn.sourceforge.net/archive-access/?rev=2420&view=rev
Author: bradtofel
Date: 2008-07-08 15:03:16 -0700 (Tue, 08 Jul 2008)
Log Message:
-----------
BUGFIX(unreported): was performing opposite the logic needed for boolean flag tests.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-08 06:06:41 UTC (rev 2419)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-08 22:03:16 UTC (rev 2420)
@@ -473,7 +473,7 @@
}
private boolean getBoolean(String key) {
String value = get(key);
- return(value == null || !value.equals(REQUEST_YES));
+ return(value != null && value.equals(REQUEST_YES));
}
/**
* @param key
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-08 06:06:33
|
Revision: 2419
http://archive-access.svn.sourceforge.net/archive-access/?rev=2419&view=rev
Author: bradtofel
Date: 2008-07-07 23:06:41 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
REFACTOR: WaybackRequest now has get/set for all standard request values.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/ArchivalUrlReplayDispatcher.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDatePrefixQueryRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDateRangeQueryRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDatePrefixQueryRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDateRangeQueryRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/HTTPAuthBooleanOperator.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/IPMatchesBooleanOperator.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/UIResults.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixReplayDispatcher.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/BaseExceptionRenderer.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyReplayRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/Renderer.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/UIQueryResults.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/FormRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/OpenSearchRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/LocalResourceIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/NutchResourceIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/RemoteResourceIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/distributed/AlphaPartitionedIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java
trunk/archive-access/projects/wayback/wayback-core/src/test/java/org/archive/wayback/resourceindex/distributed/AlphaPartitionedIndexTest.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/ArchivalUrlReplayDispatcher.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/ArchivalUrlReplayDispatcher.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/ArchivalUrlReplayDispatcher.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -75,7 +75,7 @@
// if the result is not for the exact date requested, redirect to the
// exact date. some capture dates are not 14 digits, only compare as
// many digits as are in the result date:
- String reqDateStr = wbRequest.get(WaybackRequest.REQUEST_DATE);
+ String reqDateStr = wbRequest.getReplayTimestamp();
String resDateStr = result.getCaptureTimestamp();
if(!resDateStr.equals(reqDateStr.substring(0, resDateStr.length()))) {
return redirect;
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDatePrefixQueryRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDatePrefixQueryRequestParser.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDatePrefixQueryRequestParser.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -27,7 +27,6 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import org.apache.commons.httpclient.URIException;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -66,15 +65,10 @@
startDate = Timestamp.parseBefore(dateStr).getDateStr();
endDate = Timestamp.parseAfter(dateStr).getDateStr();
}
- wbRequest.put(WaybackRequest.REQUEST_START_DATE,startDate);
- wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
- wbRequest.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_URL_QUERY);
- try {
- wbRequest.setRequestUrl(urlStr);
- } catch (URIException e) {
- wbRequest = null;
- }
+ wbRequest.setStartTimestamp(startDate);
+ wbRequest.setEndTimestamp(endDate);
+ wbRequest.setCaptureQueryRequest();
+ wbRequest.setRequestUrl(urlStr);
}
return wbRequest;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDateRangeQueryRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDateRangeQueryRequestParser.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDateRangeQueryRequestParser.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -27,7 +27,6 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import org.apache.commons.httpclient.URIException;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -61,15 +60,10 @@
String startDate = Timestamp.parseBefore(startDateStr).getDateStr();
String endDate = Timestamp.parseAfter(endDateStr).getDateStr();
- wbRequest.put(WaybackRequest.REQUEST_START_DATE,startDate);
- wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
- wbRequest.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_URL_QUERY);
- try {
- wbRequest.setRequestUrl(urlStr);
- } catch (URIException e) {
- wbRequest = null;
- }
+ wbRequest.setStartTimestamp(startDate);
+ wbRequest.setEndTimestamp(endDate);
+ wbRequest.setCaptureQueryRequest();
+ wbRequest.setRequestUrl(urlStr);
}
return wbRequest;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDatePrefixQueryRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDatePrefixQueryRequestParser.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDatePrefixQueryRequestParser.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -27,7 +27,6 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import org.apache.commons.httpclient.URIException;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -66,17 +65,11 @@
endDate = Timestamp.parseAfter(dateStr).getDateStr();
}
- wbRequest.put(WaybackRequest.REQUEST_START_DATE,
- startDate);
- wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
+ wbRequest.setStartTimestamp(startDate);
+ wbRequest.setEndTimestamp(endDate);
- wbRequest.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_URL_PREFIX_QUERY);
- try {
- wbRequest.setRequestUrl(urlStr);
- } catch (URIException e) {
- wbRequest = null;
- }
+ wbRequest.setUrlQueryRequest();
+ wbRequest.setRequestUrl(urlStr);
}
return wbRequest;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDateRangeQueryRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDateRangeQueryRequestParser.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDateRangeQueryRequestParser.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -27,7 +27,6 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import org.apache.commons.httpclient.URIException;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -58,17 +57,11 @@
String urlStr = matcher.group(3);
String startDate = Timestamp.parseBefore(startDateStr).getDateStr();
String endDate = Timestamp.parseAfter(endDateStr).getDateStr();
- wbRequest.put(WaybackRequest.REQUEST_START_DATE,
- startDate);
- wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
+ wbRequest.setStartTimestamp(startDate);
+ wbRequest.setEndTimestamp(endDate);
- wbRequest.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_URL_PREFIX_QUERY);
- try {
- wbRequest.setRequestUrl(urlStr);
- } catch (URIException e) {
- wbRequest = null;
- }
+ wbRequest.setUrlQueryRequest();
+ wbRequest.setRequestUrl(urlStr);
}
return wbRequest;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -24,12 +24,9 @@
*/
package org.archive.wayback.archivalurl.requestparser;
-import java.util.logging.Logger;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-
-import org.apache.commons.httpclient.URIException;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -42,8 +39,6 @@
* @version $Date$, $Revision$
*/
public class ReplayRequestParser extends PathRequestParser {
- private static final Logger LOGGER = Logger.getLogger(
- ReplayRequestParser.class.getName());
/**
* Regex which parses Archival URL replay requests into timestamp + url
*/
@@ -65,8 +60,6 @@
// based upon amount given (2001 => 20010101... - 20011231...)
// AND assume the user asked for the LATEST possible date
// within that range...
- //
- // ...don't ask me, I just work here.
String startDate = null;
String endDate = null;
@@ -76,36 +69,22 @@
} else {
// classic behavior:
- // startDate = Timestamp.parseBefore(dateStr).getDateStr();
- // endDate = Timestamp.parseAfter(dateStr).getDateStr();
- // dateStr = endDate;
+ startDate = Timestamp.parseBefore(dateStr).getDateStr();
+ endDate = Timestamp.parseAfter(dateStr).getDateStr();
+ dateStr = endDate;
- // "better" behavior:
- startDate = getEarliestTimestamp();
- endDate = getLatestTimestamp();
- dateStr = Timestamp.parseAfter(dateStr).getDateStr();
+ // maybe "better" behavior:
+// startDate = getEarliestTimestamp();
+// endDate = getLatestTimestamp();
+// dateStr = Timestamp.parseAfter(dateStr).getDateStr();
}
- wbRequest.put(WaybackRequest.REQUEST_DATE, dateStr);
- wbRequest.put(WaybackRequest.REQUEST_START_DATE, startDate);
- wbRequest.put(WaybackRequest.REQUEST_END_DATE, endDate);
+ wbRequest.setReplayTimestamp(dateStr);
+ wbRequest.setStartTimestamp(startDate);
+ wbRequest.setEndTimestamp(endDate);
- wbRequest.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_REPLAY_QUERY);
-
- try {
-// String wbPrefix = wbRequest.getDefaultWaybackPrefix();
-// if (urlStr.startsWith(wbPrefix)) {
-// wbRequest.setBetterRequestURI(urlStr);
-// }
- wbRequest.setRequestUrl(urlStr);
- } catch (URIException e) {
- if(urlStr != null) {
- LOGGER.severe("Failed parse of url(" + urlStr + ")");
- }
- e.printStackTrace();
- wbRequest = null;
- }
+ wbRequest.setReplayRequest();
+ wbRequest.setRequestUrl(urlStr);
}
return wbRequest;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/HTTPAuthBooleanOperator.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/HTTPAuthBooleanOperator.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/HTTPAuthBooleanOperator.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -11,7 +11,7 @@
if(allowedUsers == null) {
return false;
}
- String currentUser = value.get(WaybackRequest.REQUEST_REMOTE_USER);
+ String currentUser = value.getRemoteUser();
if(currentUser == null) {
return false;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/IPMatchesBooleanOperator.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/IPMatchesBooleanOperator.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/IPMatchesBooleanOperator.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -33,7 +33,7 @@
if(allowedRanges == null) {
return false;
}
- String ipString = value.get(WaybackRequest.REQUEST_REMOTE_ADDRESS);
+ String ipString = value.getRemoteIPAddress();
if(ipString == null) {
return false;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -109,16 +109,12 @@
CaptureSearchResult closest = null;
long closestDistance = 0;
CaptureSearchResult cur = null;
- String anchorDate = wbRequest.get(WaybackRequest.REQUEST_ANCHOR_DATE);
+ String anchorDate = wbRequest.getAnchorTimestamp();
long maxWindow = -1;
- long wantTime = Timestamp.parseBefore(wbRequest
- .get(WaybackRequest.REQUEST_EXACT_DATE)).getDate().getTime();
+ long wantTime = wbRequest.getReplayDate().getTime();
if(anchorDate != null) {
wantTime = Timestamp.parseBefore(anchorDate).getDate().getTime();
- String anchorWindow = wbRequest.get(WaybackRequest.REQUEST_ANCHOR_WINDOW);
- if(anchorWindow != null) {
- maxWindow = Long.parseLong(anchorWindow);
- }
+ maxWindow = wbRequest.getAnchorWindow();
}
Iterator<CaptureSearchResult> itr = results.iterator();
@@ -132,7 +128,7 @@
closestDistance = curDistance;
}
}
- if(err && (maxWindow != -1)) {
+ if(err && (maxWindow > 0)) {
if(closestDistance > maxWindow) {
throw new AnchorWindowTooSmallException("Closest is " +
closestDistance + " seconds away, Window is " +
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/UIResults.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/UIResults.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/UIResults.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -28,7 +28,6 @@
import javax.servlet.http.HttpServletRequest;
-import org.apache.commons.httpclient.URIException;
import org.archive.wayback.ResultURIConverter;
import org.archive.wayback.util.StringFormatter;
import org.archive.wayback.webapp.AccessPoint;
@@ -72,14 +71,8 @@
public String makeCaptureQueryUrl(String url) {
WaybackRequest newWBR = wbRequest.clone();
- newWBR.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_URL_QUERY);
- try {
- newWBR.setRequestUrl(url);
- } catch (URIException e) {
- // should not happen...
- e.printStackTrace();
- }
+ newWBR.setCaptureQueryRequest();
+ newWBR.setRequestUrl(url);
return newWBR.getContextPrefix() + "query?" +
newWBR.getQueryArguments(1);
}
@@ -214,7 +207,7 @@
*/
public String getContextConfig(final String configName) {
String configValue = null;
- AccessPoint context = getWbRequest().getContext();
+ AccessPoint context = getWbRequest().getAccessPoint();
if(context != null) {
Properties configs = context.getConfigs();
if(configs != null) {
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -23,6 +23,7 @@
package org.archive.wayback.core;
+import java.util.Date;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Locale;
@@ -34,9 +35,6 @@
import javax.servlet.http.Cookie;
import javax.servlet.http.HttpServletRequest;
-import org.apache.commons.httpclient.URIException;
-import org.archive.net.UURI;
-import org.archive.net.UURIFactory;
import org.archive.wayback.requestparser.OpenSearchRequestParser;
import org.archive.wayback.util.ObjectFilter;
import org.archive.wayback.util.StringFormatter;
@@ -51,185 +49,315 @@
*/
public class WaybackRequest {
- public final static String REQUEST_ANCHOR_DATE = "request.anchordate";
- public final static String REQUEST_ANCHOR_WINDOW = "request.anchorwindow";
-
+ /**
+ * indicates the number of requests per page, only makes sense for
+ * Capture/Url queries.
+ */
private int resultsPerPage = 10;
-
+ /**
+ * indicates the specific page of results to show, for paginated requests,
+ * only makes sense for Capture/Url queries.
+ */
private int pageNum = 1;
-
+ /**
+ * absolute URL prefix to the AccessPoint which received this request
+ */
private String contextPrefix = null;
+ /**
+ * absolute URL prefix to the Server(webapp) which received this request
+ */
private String serverPrefix = null;
- private AccessPoint context = null;
+ /**
+ * reference to the AccessPoint which received this request.
+ */
+ private AccessPoint accessPoint = null;
+ /**
+ * custom CaptureSearchResult Filter to use for this specific request. Can
+ * be null, and is sometimes useful to allow an AccessPoint to have specific
+ * and possibly variable filters.
+ */
private ObjectFilter<CaptureSearchResult> exclusionFilter = null;
-
- private HashMap<String,String> filters = new HashMap<String,String>();
-
- private StringFormatter formatter = null;
/**
- * Request: Authorization Type: "BASIC", "SSL", or "" if none.
+ * StringFormatter object set up with the users specific Locale, and the
+ * Wayback UI ResourceBundle prepared for use, simplifying UI generation
+ * somewhat.
*/
- public static final String REQUEST_AUTH_TYPE = "requestauthtype";
+ private StringFormatter formatter = null;
/**
- * Request: Wayback Context: the string context used in the request,
- * if applicable.
+ * generic String-to-String map of various request filters and type
+ * information. See constants below for keys & values.
*/
- public static final String REQUEST_WAYBACK_CONTEXT = "waybackcontext";
+ private HashMap<String,String> filters = new HashMap<String,String>();
+
+
+ /*
+ * **********************
+ * REQUEST TYPE CONSTANTS
+ * **********************
+ */
/**
- * Request: Wayback Port: the port the remote user connected to for this
- * request.
+ * specifies the TYPE of the this particular request. One of:
+ * *) REQUEST_REPLAY_QUERY
+ * *) REQUEST_CAPTURE_QUERY
+ * *) REQUEST_URL_QUERY
*/
- public static final String REQUEST_WAYBACK_PORT = "waybackport";
+ public static final String REQUEST_TYPE = "type";
/**
- * Request: Wayback Hostname: the string "Host:" HTTP header
+ * REQUEST_TYPE option indicating a request for Replay of the Resource
+ * matching REQUEST_URL closest in time to REQUEST_DATE
*/
- public static final String REQUEST_WAYBACK_HOSTNAME = "waybackhostname";
+ public static final String REQUEST_REPLAY_QUERY = "replay";
/**
- * Request: Remote Address, string IP address: "127.0.0.1"
+ * REQUEST_TYPE option indicating a query against the ResourceIndex for
+ * captures of URLs matching the REQUEST_URL
*/
- public static final String REQUEST_REMOTE_ADDRESS = "remoteaddress";
+ public static final String REQUEST_CAPTURE_QUERY = "capturequery";
/**
- * Request: auto resolution (TimeLine mode)
+ * REQUEST_TYPE option indicating a query against the ResourceIndex for
+ * summaries of URLs prefixed with the REQUEST_URL
*/
- public static final String REQUEST_RESOLUTION_AUTO = "auto";
+ public static final String REQUEST_URL_QUERY = "urlquery";
+ /*
+ * **********************
+ * /REQUEST TYPE CONSTANTS
+ * **********************
+ */
+
+
+ /*
+ * ******************
+ * URL/DATE CONSTANTS
+ * ******************
+ */
/**
- * Request: year resolution (TimeLine mode)
+ * GUARANTEED PRESENT: Original(RAW) URL or URL prefix requested, before any
+ * cleanup/fixing
*/
- public static final String REQUEST_RESOLUTION_YEARS = "years";
+ public static final String REQUEST_URL = "url";
+
/**
- * Request: two-month resolution (TimeLine mode)
+ * Cleaned up version of original requested URL or URL prefix, as performed
+ * by UURIFactory.
*/
- public static final String REQUEST_RESOLUTION_TWO_MONTHS = "twomonths";
+// public static final String REQUEST_URL_CLEANED = "cleanedurl";
+
/**
- * Request: month resolution (TimeLine mode)
+ * GUARANTEED PRESENT: omit results after this 14-digit String timestamp.
+ * Possibly created from:
+ * 1) specified directly in request
+ * 2) a partial REQUEST_DATE (latest possible given a prefix)
+ * 3) RequestParser default
+ * 4) 14-digit representation of the moment the request was recieved.
*/
- public static final String REQUEST_RESOLUTION_MONTHS = "months";
+ public static final String REQUEST_END_DATE = "enddate";
+
/**
- * Request: day resolution (TimeLine mode)
+ * GUARANTEED PRESENT: omit results before this 14-digit String timestamp.
+ * Possibly created from:
+ * 1) specified directly in request
+ * 2) a partial REQUEST_DATE (earliest possible given a prefix)
+ * 3) RequestParser default
+ * 4) 14-digit representation of midnight Jan 1, 1996.
*/
- public static final String REQUEST_RESOLUTION_DAYS = "days";
+ public static final String REQUEST_START_DATE = "startdate";
+
/**
- * Request: hour resolution (TimeLine mode)
+ * GUARANTEED PRESENT for Replay requests only, no meaning for Query
+ * requests.
+ * Original (RAW/possibly partial) 14-digit timestamp of date requested for
+ * Replay
*/
- public static final String REQUEST_RESOLUTION_HOURS = "hours";
+ public static final String REQUEST_DATE = "date";
+
/**
- * Request: replay actual document or metadata for document: "yes" means
- * replay metadata only, not the actual document: (TimeLine mode)
+ * GUARANTEED PRESENT for Replay requests only, no meaning for Query
+ * requests.
+ * Cleaned up version of original REQUEST_DATE, padded to 14 digits assuming
+ * the
*/
- public static final String REQUEST_META_MODE = "metamode";
+ public static final String REQUEST_EXACT_DATE = "exactdate";
+
/**
- * Request: resolution of results to be displayed: (TimeLine mode)
+ * Indicates user only wants results that exactly match the hostname within
+ * REQUEST_URL -- no canonicalization.
*/
- public static final String REQUEST_RESOLUTION = "resolution";
+ public static final String REQUEST_EXACT_HOST_ONLY = "requestexacthost";
+
/**
- * Request: closest type request
+ * indicates positive value for any request boolean flag.
*/
- public static final String REQUEST_CLOSEST_QUERY = "urlclosestquery";
+ public static final String REQUEST_YES = "yes";
+
/**
- * Request: replay type request
+ * Replay-Only: indicates the date to tend towards when computing closest
+ * matches within time. Used to prevent "time drift" while surfing from a
+ * particular date.
*/
- public static final String REQUEST_REPLAY_QUERY = "replay";
+ public final static String REQUEST_ANCHOR_DATE = "request.anchordate";
+
/**
- * Request: urlprefixquery type request
+ * Replay-Only: String representation of number of seconds. Used only in
+ * conjunction with REQUEST_ANCHOR_DATE, and indicates that documents more
+ * than this many seconds should not be shown in a replay session. Useful
+ * for QA purposes, to ensure that all content within a replay session was
+ * crawled near a particular point, the REQUEST_ANCHOR_DATE.
*/
- public static final String REQUEST_URL_PREFIX_QUERY = "urlprefixquery";
+ public final static String REQUEST_ANCHOR_WINDOW = "request.anchorwindow";
+ /*
+ * ******************
+ * /URL/DATE CONSTANTS
+ * ******************
+ */
+
+
+ /*
+ * *******************************
+ * OUTPUT TYPE CONSTANTS
+ * *******************************
+ */
/**
- * Request: urlquery type request
+ * Request: replay actual document or metadata for document: "yes" means
+ * replay metadata only, not the actual document: (TimeLine mode)
*/
- public static final String REQUEST_URL_QUERY = "urlquery";
+ public static final String REQUEST_META_MODE = "metamode";
/**
* Request: xml data requested
*/
public static final String REQUEST_XML_DATA = "xmldata";
+ /*
+ * *******************************
+ * /OUTPUT TYPE CONSTANTS
+ * *******************************
+ */
+
+ /*
+ * *******************************
+ * CONTEXT & ACCESSPOINT CONSTANTS
+ * *******************************
+ */
/**
- * Request: defines type - urlquery, urlprefixquery, or replay
+ * the string (webapp) context that received this request
*/
- public static final String REQUEST_TYPE = "type";
+ public static final String REQUEST_WAYBACK_CONTEXT = "waybackcontext";
/**
- * Request: URL of referrer, if supplied, or "" if not
+ * the port the remote user connected to for this request
*/
+ public static final String REQUEST_WAYBACK_PORT = "waybackport";
+ /*
+ * *******************************
+ * /CONTEXT & ACCESSPOINT CONSTANTS
+ * *******************************
+ */
+
+ /*
+ * *****************************
+ * HTTP HEADER/REQUEST CONSTANTS
+ * *****************************
+ */
+ /**
+ * incoming requests HTTP "Host:" header, or null
+ */
+ public static final String REQUEST_WAYBACK_HOSTNAME = "waybackhostname";
+ /**
+ * incoming requests HTTP "Referer:" header, or null
+ */
public static final String REQUEST_REFERER_URL = "refererurl";
/**
- * Request: Original URL or URL prefix requested.
- * This version differs from @{link {@link REQUEST_URL} in that its
- * the URL before it was passed via the UURIFactory cleanup.
+ * Remote Address that connected to this webapp to create the request
+ * string IP address: "127.0.0.1"
*/
- public static final String REQUEST_URL_CLEANED = "cleanedurl";
+ public static final String REQUEST_REMOTE_ADDRESS = "remoteaddress";
/**
- * Request: URL or URL prefix requested
+ * Remote User or null if the request did not contain auth info.
+ * see HttpServletRequest.getRemoteUser()
*/
- public static final String REQUEST_URL = "url";
+ public static final String REQUEST_REMOTE_USER = "requestremoteuser";
+
/**
- * Request: (replay) find closest result to this 14-digit timestamp
+ * User Locale name: Best Guess at users requested locale.
+ * see ServletRequest.getLocale().getDisplayLanguage()
*/
- public static final String REQUEST_EXACT_DATE = "exactdate";
+ public static final String REQUEST_LOCALE_LANG = "requestlocalelang";
/**
- * Request: filter results after this 14-digit timestamp
+ * Authorization Type: "BASIC", "SSL", or null if none.
+ * see HttpServletRequest.getAuthType()
*/
- public static final String REQUEST_END_DATE = "enddate";
+ public static final String REQUEST_AUTH_TYPE = "requestauthtype";
+ /*
+ * ***********************
+ * /HTTP HEADER/REQUEST CONSTANTS
+ * ***********************
+ */
+
+ /*
+ * ***********************
+ * TIMELINE MODE CONSTANTS
+ * ***********************
+ */
/**
- * Request: filter results before this 14-digit timestamp
+ * resolution of results to be displayed: (TimeLine mode)
*/
- public static final String REQUEST_START_DATE = "startdate";
+ public static final String REQUEST_RESOLUTION = "resolution";
/**
- * Request: (query) filter results to those prefixed with this (possibly
- * partial) 14-digit timestamp
+ * auto resolution (TimeLine mode)
*/
- public static final String REQUEST_DATE = "date";
+ public static final String REQUEST_RESOLUTION_AUTO = "auto";
/**
- * Request: Remote User or "" if the request did not contain auth info.
+ * year resolution (TimeLine mode)
*/
- public static final String REQUEST_REMOTE_USER = "requestremoteuser";
+ public static final String REQUEST_RESOLUTION_YEARS = "years";
/**
- * Request: Best Guess at users requested locale.
+ * two-month resolution (TimeLine mode)
*/
- public static final String REQUEST_LOCALE_LANG = "requestlocalelang";
+ public static final String REQUEST_RESOLUTION_TWO_MONTHS = "twomonths";
/**
- * Request: Indicates user only wants results that exactly match the
- * requested hostname -- no canonicalization.
+ * month resolution (TimeLine mode)
*/
- public static final String REQUEST_EXACT_HOST_ONLY = "requestexacthost";
+ public static final String REQUEST_RESOLUTION_MONTHS = "months";
/**
- * Request: indicates positive value for any request boolean flag.
+ * day resolution (TimeLine mode)
*/
- public static final String REQUEST_YES = "yes";
+ public static final String REQUEST_RESOLUTION_DAYS = "days";
+ /**
+ * hour resolution (TimeLine mode)
+ */
+ public static final String REQUEST_RESOLUTION_HOURS = "hours";
+ /*
+ * ***********************
+ * /TIMELINE MODE CONSTANTS
+ * ***********************
+ */
+
private static String UI_RESOURCE_BUNDLE_NAME = "WaybackUI";
+ /**
+ * set of filter keys that are not forwarded to subsequent paginated
+ * requests.
+ */
private final static String standardHeaders[] = {
- WaybackRequest.REQUEST_REFERER_URL,
- WaybackRequest.REQUEST_REMOTE_ADDRESS,
- WaybackRequest.REQUEST_WAYBACK_HOSTNAME,
- WaybackRequest.REQUEST_WAYBACK_PORT,
- WaybackRequest.REQUEST_WAYBACK_CONTEXT,
- WaybackRequest.REQUEST_AUTH_TYPE,
- WaybackRequest.REQUEST_REMOTE_USER,
- WaybackRequest.REQUEST_LOCALE_LANG };
+ REQUEST_REFERER_URL,
+ REQUEST_REMOTE_ADDRESS,
+ REQUEST_WAYBACK_HOSTNAME,
+ REQUEST_WAYBACK_PORT,
+ REQUEST_WAYBACK_CONTEXT,
+ REQUEST_AUTH_TYPE,
+ REQUEST_REMOTE_USER,
+ REQUEST_LOCALE_LANG };
/**
- * Constructor, possibly/probably this should BE a Properties, instead of
- * HAVEing a Properties...
+ * @return Returns the resultsPerPage.
*/
- public WaybackRequest() {
- super();
+ public int getResultsPerPage() {
+ return resultsPerPage;
}
/**
- * @return true if REQUEST_TYPE is set, and is set to REQUEST_REPLAY_QUERY
+ * @param resultsPerPage
+ * The resultsPerPage to set.
*/
- public boolean isReplayRequest() {
- String type = get(WaybackRequest.REQUEST_TYPE);
- if(type != null && type.equals(WaybackRequest.REQUEST_REPLAY_QUERY)) {
- return true;
- }
- return false;
+ public void setResultsPerPage(int resultsPerPage) {
+ this.resultsPerPage = resultsPerPage;
}
- /**
- * @return true if true if REQUEST_TYPE is not set, or is set to a value
- * other than REQUEST_REPLAY_QUERY
- */
- public boolean isQueryRequest() {
- return !isReplayRequest();
- }
/**
* @return Returns the pageNum.
@@ -247,34 +375,82 @@
}
/**
- * @return Returns the resultsPerPage.
+ * @param prefix
*/
- public int getResultsPerPage() {
- return resultsPerPage;
+ public void setContextPrefix(String prefix) {
+ contextPrefix = prefix;
}
/**
- * @param resultsPerPage
- * The resultsPerPage to set.
+ * Construct an absolute URL that points to the root of the context that
+ * received the request, including a trailing "/".
+ *
+ * @return String absolute URL pointing to the Context root where the
+ * request was received.
*/
- public void setResultsPerPage(int resultsPerPage) {
- this.resultsPerPage = resultsPerPage;
+ public String getContextPrefix() {
+ if(contextPrefix == null) {
+ return "";
+ }
+ return contextPrefix;
}
/**
- * @param key
- * @return boolean, true if the request contains key 'key'
+ * @param prefix
*/
- public boolean containsKey(String key) {
- return filters.containsKey(key);
+ public void setServerPrefix(String prefix) {
+ serverPrefix = prefix;
}
/**
+ * @param prefix
+ * @return an absolute String URL that will point to the root of the
+ * server that is handling the request.
+ */
+ public String getServerPrefix() {
+ if(serverPrefix == null) {
+ return "";
+ }
+ return serverPrefix;
+ }
+ /**
+ * @return the accessPoint
+ */
+ public AccessPoint getAccessPoint() {
+ return accessPoint;
+ }
+
+ /**
+ * @param accessPoint the accessPoint to set
+ */
+ public void setAccessPoint(AccessPoint accessPoint) {
+ this.accessPoint = accessPoint;
+ }
+
+ public ObjectFilter<CaptureSearchResult> getExclusionFilter() {
+ return exclusionFilter;
+ }
+
+ public void setExclusionFilter(ObjectFilter<CaptureSearchResult> exclusionFilter) {
+ this.exclusionFilter = exclusionFilter;
+ }
+
+ /**
+ * @return StringFormatter based on user request info
+ */
+ public StringFormatter getFormatter() {
+ if(formatter == null) {
+ setLocale(Locale.getAvailableLocales()[0]);
+ }
+ return formatter;
+ }
+
+ /**
* @param key
* @return String value for key 'key', or null if no value exists
*/
public String get(String key) {
- return (String) filters.get(key);
+ return filters.get(key);
}
/**
@@ -284,14 +460,238 @@
public void put(String key, String value) {
filters.put(key, value);
}
+ public void remove(String key) {
+ filters.remove(key);
+ }
- private String emptyIfNull(String arg) {
- if (arg == null) {
- return "";
+ private void setBoolean(String key, boolean value) {
+ if(value) {
+ put(key,REQUEST_YES);
+ } else {
+ remove(key);
}
- return arg;
+ }
+ private boolean getBoolean(String key) {
+ String value = get(key);
+ return(value == null || !value.equals(REQUEST_YES));
+ }
+ /**
+ * @param key
+ * @return boolean, true if the request contains key 'key'
+ * @deprecated
+ */
+ public boolean containsKey(String key) {
+ return filters.containsKey(key);
}
+
+ private void putUnlessNull(String key, String val) {
+ if (val != null) {
+ put(key,val);
+ }
+ }
+
+ private boolean isRequestType(String requestType) {
+ String type = get(REQUEST_TYPE);
+ if(type != null && type.equals(requestType)) {
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * @return true if this is a Replay request
+ */
+ public boolean isReplayRequest() {
+ return isRequestType(REQUEST_REPLAY_QUERY);
+ }
+ /**
+ * marks this request as a Replay request
+ */
+ public void setReplayRequest() {
+ put(REQUEST_TYPE,REQUEST_REPLAY_QUERY);
+ }
+ /**
+ * @return true if this is a Capture Query request
+ */
+ public boolean isCaptureQueryRequest() {
+ return isRequestType(REQUEST_CAPTURE_QUERY);
+ }
+ /**
+ * marks this request as a Replay request
+ */
+ public void setCaptureQueryRequest() {
+ put(REQUEST_TYPE,REQUEST_CAPTURE_QUERY);
+ }
+ /**
+ * @return true if this is an Url Query request
+ */
+ public boolean isUrlQueryRequest() {
+ return isRequestType(REQUEST_URL_QUERY);
+ }
+ /**
+ * marks this request as a Replay request
+ */
+ public void setUrlQueryRequest() {
+ put(REQUEST_TYPE,REQUEST_URL_QUERY);
+ }
+
+ public String getRequestUrl() {
+ return get(REQUEST_URL);
+ }
+ /**
+ * Set the request URL.
+ * @param urlStr Request URL.
+ */
+ public void setRequestUrl(String urlStr) {
+ // TODO: fix this to use other schemes
+ if (!urlStr.startsWith("http://")) {
+ if(urlStr.startsWith("http:/")) {
+ urlStr = "http://" + urlStr.substring(6);
+ } else {
+ urlStr = "http://" + urlStr;
+ }
+ }
+// UURI requestURI = UURIFactory.getInstance(urlStr);
+// put(REQUEST_URL_CLEANED, requestURI.toString());
+ put(REQUEST_URL, urlStr);
+ }
+ public String getEndTimestamp() {
+ return get(REQUEST_END_DATE);
+ }
+ public Date getEndDate() {
+ return Timestamp.parseAfter(get(REQUEST_END_DATE)).getDate();
+ }
+ public void setEndDate(Date date) {
+ put(REQUEST_END_DATE,new Timestamp(date).getDateStr());
+ }
+ public void setEndTimestamp(String timestamp) {
+ put(REQUEST_END_DATE,timestamp);
+ }
+
+ public String getStartTimestamp() {
+ return get(REQUEST_START_DATE);
+ }
+ public Date getStartDate() {
+ return Timestamp.parseBefore(get(REQUEST_START_DATE)).getDate();
+ }
+ public void setStartDate(Date date) {
+ put(REQUEST_START_DATE,new Timestamp(date).getDateStr());
+ }
+ public void setStartTimestamp(String timestamp) {
+ put(REQUEST_START_DATE,timestamp);
+ }
+
+ public String getReplayTimestamp() {
+ return get(REQUEST_DATE);
+ }
+ public Date getReplayDate() {
+ return Timestamp.parseAfter(get(REQUEST_DATE)).getDate();
+ }
+ public void setReplayDate(Date date) {
+ put(REQUEST_DATE,new Timestamp(date).getDateStr());
+ }
+ public void setReplayTimestamp(String timestamp) {
+ put(REQUEST_DATE,timestamp);
+ }
+
+ public void setExactHost(boolean isExactHost) {
+ setBoolean(REQUEST_EXACT_HOST_ONLY,isExactHost);
+ }
+ public boolean isExactHost() {
+ return getBoolean(REQUEST_EXACT_HOST_ONLY);
+ }
+
+ public String getAnchorTimestamp() {
+ return get(REQUEST_ANCHOR_DATE);
+ }
+ public Date getAnchorDate() {
+ return Timestamp.parseAfter(get(REQUEST_ANCHOR_DATE)).getDate();
+ }
+ public void setAnchorDate(Date date) {
+ put(REQUEST_ANCHOR_DATE,new Timestamp(date).getDateStr());
+ }
+ public void setAnchorTimestamp(String timestamp) {
+ put(REQUEST_ANCHOR_DATE,timestamp);
+ }
+
+ public long getAnchorWindow() {
+ String seconds = get(REQUEST_ANCHOR_WINDOW);
+ if(seconds == null) {
+ return 0;
+ }
+ return Long.parseLong(seconds);
+ }
+ public void setAnchorWindow(long seconds) {
+ put(REQUEST_ANCHOR_WINDOW,String.valueOf(seconds));;
+ }
+
+ public void setMetaMode(boolean isMetaMode) {
+ setBoolean(REQUEST_META_MODE,isMetaMode);
+ }
+ public boolean isMetaMode() {
+ return getBoolean(REQUEST_META_MODE);
+ }
+
+ public void setXMLMode(boolean isXMLMode) {
+ setBoolean(REQUEST_XML_DATA,isXMLMode);
+ }
+ public boolean isXMLMode() {
+ return getBoolean(REQUEST_XML_DATA);
+ }
+
+ public String getWaybackContext() {
+ return get(REQUEST_WAYBACK_CONTEXT);
+ }
+ public int getWaybackPort() {
+ String port = get(REQUEST_WAYBACK_PORT);
+ if(port == null) {
+ return 0;
+ }
+ return Integer.parseInt(port);
+ }
+
+ public String getWaybackHostname() {
+ return get(REQUEST_WAYBACK_HOSTNAME);
+ }
+ public String getRefererUrl() {
+ return get(REQUEST_REFERER_URL);
+ }
+ public String getRemoteIPAddress() {
+ return get(REQUEST_REMOTE_ADDRESS);
+ }
+ public String getRemoteUser() {
+ return get(REQUEST_REMOTE_USER);
+ }
+ public String getLocaleLanguage() {
+ return get(REQUEST_LOCALE_LANG);
+ }
+ public String getAuthType() {
+ return get(REQUEST_AUTH_TYPE);
+ }
+
+ public String getTimelineResolution() {
+ return get(REQUEST_RESOLUTION);
+ }
+ public void setTimelineAutoResolution() {
+ put(REQUEST_RESOLUTION,REQUEST_RESOLUTION_AUTO);
+ }
+ public void setTimelineYearResolution() {
+ put(REQUEST_RESOLUTION,REQUEST_RESOLUTION_YEARS);
+ }
+ public void setTimelineTwoMonthResolution() {
+ put(REQUEST_RESOLUTION,REQUEST_RESOLUTION_TWO_MONTHS);
+ }
+ public void setTimelineMonthResolution() {
+ put(REQUEST_RESOLUTION,REQUEST_RESOLUTION_MONTHS);
+ }
+ public void setTimelineDayResolution() {
+ put(REQUEST_RESOLUTION,REQUEST_RESOLUTION_DAYS);
+ }
+ public void setTimelineHourResolution() {
+ put(REQUEST_RESOLUTION,REQUEST_RESOLUTION_HOURS);
+ }
+
/**
* Set the Locale for the request, which impacts UI Strings
* @param l
@@ -301,14 +701,6 @@
formatter = new StringFormatter(b,l);
}
- private String getUserLocale(HttpServletRequest httpRequest) {
- Locale l = httpRequest.getLocale();
- ResourceBundle b = ResourceBundle.getBundle(UI_RESOURCE_BUNDLE_NAME,
- httpRequest.getLocale());
- formatter = new StringFormatter(b,l);
- return emptyIfNull(httpRequest.getLocale().getDisplayLanguage());
- }
-
/**
* extract REFERER, remote IP and authorization information from the
* HttpServletRequest
@@ -316,23 +708,22 @@
* @param httpRequest
*/
private void extractHttpRequestInfo(HttpServletRequest httpRequest) {
- // attempt to get the HTTP referer if present..
- put(WaybackRequest.REQUEST_REFERER_URL, emptyIfNull(httpRequest
- .getHeader("REFERER")));
- put(WaybackRequest.REQUEST_REMOTE_ADDRESS, emptyIfNull(httpRequest
- .getRemoteAddr()));
- put(WaybackRequest.REQUEST_WAYBACK_HOSTNAME, emptyIfNull(httpRequest
- .getLocalName()));
- put(WaybackRequest.REQUEST_WAYBACK_PORT, String.valueOf(httpRequest
- .getLocalPort()));
- put(WaybackRequest.REQUEST_WAYBACK_CONTEXT, emptyIfNull(httpRequest
- .getContextPath()));
- put(WaybackRequest.REQUEST_AUTH_TYPE, emptyIfNull(httpRequest
- .getAuthType()));
- put(WaybackRequest.REQUEST_REMOTE_USER, emptyIfNull(httpRequest
- .getRemoteUser()));
- put(WaybackRequest.REQUEST_LOCALE_LANG,getUserLocale(httpRequest));
+
+ putUnlessNull(REQUEST_REFERER_URL, httpRequest.getHeader("REFERER"));
+ putUnlessNull(REQUEST_REMOTE_ADDRESS, httpRequest.getRemoteAddr());
+ putUnlessNull(REQUEST_WAYBACK_HOSTNAME, httpRequest.getLocalName());
+ putUnlessNull(REQUEST_AUTH_TYPE, httpRequest.getAuthType());
+ putUnlessNull(REQUEST_REMOTE_USER, httpRequest.getRemoteUser());
+ putUnlessNull(REQUEST_WAYBACK_PORT,
+ String.valueOf(httpRequest.getLocalPort()));
+ putUnlessNull(REQUEST_WAYBACK_CONTEXT, httpRequest.getContextPath());
+ Locale l = httpRequest.getLocale();
+ ResourceBundle b = ResourceBundle.getBundle(UI_RESOURCE_BUNDLE_NAME,
+ httpRequest.getLocale());
+ formatter = new StringFormatter(b,l);
+ putUnlessNull(REQUEST_LOCALE_LANG,l.getDisplayLanguage());
+
Cookie[] cookies = httpRequest.getCookies();
if(cookies != null) {
for(Cookie cookie : cookies) {
@@ -342,46 +733,6 @@
}
/**
- * @param prefix
- */
- public void setServerPrefix(String prefix) {
- serverPrefix = prefix;
- }
-
- /**
- * @param prefix
- * @return an absolute String URL that will point to the root of the
- * server that is handling the request.
- */
- public String getServerPrefix() {
- if(serverPrefix == null) {
- return "";
- }
- return serverPrefix;
- }
-
-
- /**
- * @param prefix
- */
- public void setContextPrefix(String prefix) {
- contextPrefix = prefix;
- }
- /**
- * Construct an absolute URL that points to the root of the context that
- * recieved the request, including a trailing "/".
- *
- * @return String absolute URL pointing to the Context root where the
- * request was revieved.
- */
- public String getContextPrefix() {
- if(contextPrefix == null) {
- return "";
- }
- return contextPrefix;
- }
-
- /**
* attempt to fixup this WaybackRequest, mostly with respect to dates: if
* only "date" was specified, infer start and end dates from it. Also grab
* useful info from the HttpServletRequest, cookies, remote address, etc.
@@ -390,32 +741,32 @@
*/
public void fixup(HttpServletRequest httpRequest) {
extractHttpRequestInfo(httpRequest);
- String startDate = get(WaybackRequest.REQUEST_START_DATE);
- String endDate = get(WaybackRequest.REQUEST_END_DATE);
- String exactDate = get(WaybackRequest.REQUEST_EXACT_DATE);
- String partialDate = get(WaybackRequest.REQUEST_DATE);
+ String startDate = get(REQUEST_START_DATE);
+ String endDate = get(REQUEST_END_DATE);
+ String exactDate = get(REQUEST_EXACT_DATE);
+ String partialDate = get(REQUEST_DATE);
if (partialDate == null) {
partialDate = "";
}
if (startDate == null || startDate.length() == 0) {
- put(WaybackRequest.REQUEST_START_DATE, Timestamp
+ put(REQUEST_START_DATE, Timestamp
.padStartDateStr(partialDate));
} else if (startDate.length() < 14) {
- put(WaybackRequest.REQUEST_START_DATE, Timestamp
+ put(REQUEST_START_DATE, Timestamp
.padStartDateStr(startDate));
}
if (endDate == null || endDate.length() == 0) {
- put(WaybackRequest.REQUEST_END_DATE, Timestamp
+ put(REQUEST_END_DATE, Timestamp
.padEndDateStr(partialDate));
} else if (endDate.length() < 14) {
- put(WaybackRequest.REQUEST_END_DATE, Timestamp
+ put(REQUEST_END_DATE, Timestamp
.padEndDateStr(endDate));
}
if (exactDate == null || exactDate.length() == 0) {
- put(WaybackRequest.REQUEST_EXACT_DATE, Timestamp
+ put(REQUEST_EXACT_DATE, Timestamp
.padEndDateStr(partialDate));
} else if (exactDate.length() < 14) {
- put(WaybackRequest.REQUEST_EXACT_DATE, Timestamp
+ put(REQUEST_EXACT_DATE, Timestamp
.padEndDateStr(exactDate));
}
}
@@ -469,35 +820,6 @@
+ OpenSearchRequestParser.START_PAGE + "=" + pageNum;
}
- /**
- * Set the request URL.
- * Also populates request url cleaned.
- * @param urlStr Request URL.
- * @throws URIException
- */
- public void setRequestUrl(String urlStr) throws URIException {
- if (!urlStr.startsWith("http://")) {
- if(urlStr.startsWith("http:/")) {
- urlStr = "http://" + urlStr.substring(6);
- } else {
- urlStr = "http://" + urlStr;
- }
- }
- // If its not http, next line throws exception. TODO: Fix.
- UURI requestURI = UURIFactory.getInstance(urlStr);
- put(WaybackRequest.REQUEST_URL_CLEANED, requestURI.toString());
- put(WaybackRequest.REQUEST_URL, urlStr);
- }
-
- /**
- * @return StringFormatter based on user request info
- */
- public StringFormatter getFormatter() {
- if(formatter == null) {
- setLocale(Locale.getAvailableLocales()[0]);
- }
- return formatter;
- }
public WaybackRequest clone() {
WaybackRequest wbRequest = new WaybackRequest();
@@ -521,27 +843,10 @@
}
/**
- * @return the context
+ *
+ * @return
+ * @deprecated
*/
- public AccessPoint getContext() {
- return context;
- }
-
- /**
- * @param context the context to set
- */
- public void setContext(AccessPoint context) {
- this.context = context;
- }
-
- public ObjectFilter<CaptureSearchResult> getExclusionFilter() {
- return exclusionFilter;
- }
-
- public void setExclusionFilter(ObjectFilter<CaptureSearchResult> exclusionFilter) {
- this.exclusionFilter = exclusionFilter;
- }
-
public Set<String> keySet() {
return filters.keySet();
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixReplayDispatcher.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixReplayDispatcher.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixReplayDispatcher.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -59,7 +59,7 @@
// if the result is not for the exact date requested, redirect to the
// exact date. some capture dates are not 14 digits, only compare as
// many digits as are in the result date:
- String reqDateStr = wbRequest.get(WaybackRequest.REQUEST_EXACT_DATE);
+ String reqDateStr = wbRequest.getReplayTimestamp();
String resDateStr = result.getCaptureTimestamp();
if((resDateStr.length() > reqDateStr.length()) ||
!resDateStr.equals(reqDateStr.substring(0, resDateStr.length()))) {
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixRequestParser.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixRequestParser.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -29,7 +29,6 @@
import javax.servlet.http.HttpServletRequest;
-import org.apache.commons.httpclient.URIException;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.exception.BadQueryException;
@@ -89,21 +88,19 @@
String requestUrl = getRequestString(host,httpRequest);
- wbRequest.put(WaybackRequest.REQUEST_EXACT_DATE, dateStr);
- wbRequest.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_REPLAY_QUERY);
- try {
- wbRequest.setRequestUrl(requestUrl);
- } catch (URIException e) {
- e.printStackTrace();
- wbRequest = null;
- }
+ wbRequest.setReplayRequest();
+ wbRequest.setReplayTimestamp(dateStr);
+ wbRequest.setRequestUrl(requestUrl);
+
} else {
Matcher queryMatcher = QUERY_REGEX.matcher(prefix);
if(queryMatcher != null && queryMatcher.matches()) {
wbRequest = new WaybackRequest();
String dateStr = queryMatcher.group(1);
String host = queryMatcher.group(2);
+
+ String requestUrl = getRequestString(host,httpRequest);
+
String startDate;
String endDate;
if(dateStr.length() == 0) {
@@ -113,20 +110,13 @@
startDate = Timestamp.parseBefore(dateStr).getDateStr();
endDate = Timestamp.parseAfter(dateStr).getDateStr();
}
- wbRequest.put(WaybackRequest.REQUEST_START_DATE,startDate);
- wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
- wbRequest.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_URL_QUERY);
-
- String requestUrl = getRequestString(host,httpRequest);
-
- try {
- wbRequest.setRequestUrl(requestUrl);
- } catch (URIException e) {
- e.printStackTrace();
- wbRequest = null;
- }
+ wbRequest.setCaptureQueryRequest();
+ wbRequest.setStartTimestamp(startDate);
+ wbRequest.setEndTimestamp(endDate);
+ wbRequest.setRequestUrl(requestUrl);
}
+ // TODO: what if it doesn't match the QUERY_REGEX?
+ // throw a BadQueryException?
}
}
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/BaseExceptionRenderer.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/BaseExceptionRenderer.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/BaseExceptionRenderer.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -68,13 +68,13 @@
if (wbRequest == null) {
return false;
}
- String referer = wbRequest.get(WaybackRequest.REQUEST_REFERER_URL);
+ String referer = wbRequest.getRefererUrl();
return (referer != null && referer.length() > 0);
}
protected boolean requestIsImage(HttpServletRequest httpRequest,
WaybackRequest wbRequest) {
- String requestUrl = wbRequest.get(WaybackRequest.REQUEST_URL);
+ String requestUrl = wbRequest.getRequestUrl();
if (requestUrl == null)
return false;
Matcher matcher = IMAGE_REGEX.matcher(requestUrl);
@@ -84,14 +84,14 @@
protected boolean requestIsJavascript(HttpServletRequest httpRequest,
WaybackRequest wbRequest) {
- String requestUrl = wbRequest.get(WaybackRequest.REQUEST_URL);
+ String requestUrl = wbRequest.getRequestUrl();
return (requestUrl != null) && requestUrl.endsWith(".js");
}
protected boolean requestIsCSS(HttpServletRequest httpRequest,
WaybackRequest wbRequest) {
- String requestUrl = wbRequest.get(WaybackRequest.REQUEST_URL);
+ String requestUrl = wbRequest.getRequestUrl();
return (requestUrl != null) && requestUrl.endsWith(".css");
}
@@ -101,9 +101,9 @@
// the "standard HTML" response handler:
String jspPath = errorJsp;
- if(wbRequest.isQueryRequest()) {
+ if(!wbRequest.isReplayRequest()) {
- if(wbRequest.containsKey(WaybackRequest.REQUEST_XML_DATA)) {
+ if(wbRequest.isXMLMode()) {
jspPath = xmlErrorJsp;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -80,10 +80,8 @@
boolean bUseOlder) throws URIException {
WaybackRequest req = new WaybackRequest();
req.setRequestUrl(url.toString());
- req.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_CLOSEST_QUERY);
- req.put(WaybackRequest.REQUEST_EXACT_DATE,
- Timestamp.currentTimestamp().getDateStr());
+ req.setReplayRequest();
+ req.setReplayTimestamp(Timestamp.currentTimestamp().getDateStr());
Timestamp earliest = null;
if(bUseOlder) {
earliest = Timestamp.earliestTimestamp();
@@ -91,11 +89,10 @@
Date d = new Date(System.currentTimeMillis() - maxCacheMS);
earliest = new Timestamp(d);
}
- req.put(WaybackRequest.REQUEST_START_DATE,earliest.getDateStr());
+ req.setStartTimestamp(earliest.getDateStr());
// for now, assume all live web requests are only satisfiable by the
// exact host -- no massaging.
- req.put(WaybackRequest.REQUEST_EXACT_HOST_ONLY,
- WaybackRequest.REQUEST_YES);
+ req.setExactHost(true);
return req;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyReplayRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyReplayRequestParser.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyReplayRequestParser.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -28,7 +28,6 @@
import javax.servlet.http.HttpServletRequest;
-import org.apache.commons.httpclient.URIException;
import org.archive.util.InetAddressUtil;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.exception.BadQueryException;
@@ -85,14 +84,8 @@
String requestUrl = requestScheme + "://" + requestServer + requestPath;
wbRequest = new WaybackRequest();
- try {
- wbRequest.setRequestUrl(requestUrl);
- } catch (URIException e) {
- e.printStackTrace();
- return null;
- }
- wbRequest.put(WaybackRequest.REQUEST_TYPE,
- WaybackRequest.REQUEST_REPLAY_QUERY);
+ wbRequest.setRequestUrl(requestUrl);
+ wbRequest.setReplayRequest();
wbRequest.setResultsPerPage(maxRecords);
return wbRequest;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyRequestParser.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyRequestParser.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -70,8 +70,10 @@
String id = httpRequest.getHeader("Proxy-Id");
if (id == null)
id = httpRequest.getRemoteAddr();
- wbRequest.put(WaybackRequest.REQUEST_EXACT_DATE, Timestamp
- .getTimestampForId(httpRequest.getContextPath(), id));
+ // TODO: This is hacky.
+ String replayDateStr = Timestamp.getTimestampForId(
+ httpRequest.getContextPath(), id);
+ wbRequest.setReplayTimestamp(replayDateStr);
wbRequest.fixup(httpRequest);
}
return wbRequest;
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/Renderer.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/Renderer.java 2008-07-07 22:07:22 UTC (rev 2418)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/Renderer.java 2008-07-08 06:06:41 UTC (rev 2419)
@@ -72,10 +72,10 @@
CaptureSearchResults results, ResultURIConverter uriConverter)
throws ServletException, IOException {
- UICaptureQueryResults uiResults = new UICaptureQueryResults(httpRequest, wbRequest,
- results, uriConverter);
+ UICaptureQueryResults uiResults = new UICaptureQueryResults(httpRequest,
+ wbRequest, results, uriConverter);
String jsp = captureJsp;
- if(wbRequest.containsKey(WaybackRequest.REQUEST_XML_DATA)) {
+ if(wbRequest.isXMLMode()) {
jsp = xmlCaptureJsp;
}
@@ -95,7 +95,7 @@
UIUrlQueryResults uiResults = new UIUrlQueryResults(httpRequest, wbRequest,
results, uriConverter);
String jsp = urlJsp;
- if(wbRequest.containsKey(WaybackRequest.REQUEST_XML_DATA)) {
+ if(wbRequest.is...
[truncated message content] |
Revision: 2418
http://archive-access.svn.sourceforge.net/archive-access/?rev=2418&view=rev
Author: bradtofel
Date: 2008-07-07 15:07:22 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
TWEAK: now storing initial requested date as REQUEST_DATE not REQUEST_EXACT_DATE
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java 2008-07-07 22:07:22 UTC (rev 2418)
@@ -86,7 +86,7 @@
dateStr = Timestamp.parseAfter(dateStr).getDateStr();
}
- wbRequest.put(WaybackRequest.REQUEST_EXACT_DATE, dateStr);
+ wbRequest.put(WaybackRequest.REQUEST_DATE, dateStr);
wbRequest.put(WaybackRequest.REQUEST_START_DATE, startDate);
wbRequest.put(WaybackRequest.REQUEST_END_DATE, endDate);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-07 21:57:22
|
Revision: 2417
http://archive-access.svn.sourceforge.net/archive-access/?rev=2417&view=rev
Author: bradtofel
Date: 2008-07-07 14:57:30 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
REFACTOR: moved WaybackConstants.REQUEST* to WaybackRequest class.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/WaybackConstants.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/ArchivalUrlReplayDispatcher.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDatePrefixQueryRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDateRangeQueryRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDatePrefixQueryRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDateRangeQueryRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/HTTPAuthBooleanOperator.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/IPMatchesBooleanOperator.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/SearchResults.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/UIResults.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixReplayDispatcher.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/BaseExceptionRenderer.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyReplayRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/Renderer.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/UIQueryResults.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/resultspartitioner/ResultsPartitionsFactory.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/resultspartitioner/ResultsTimelinePartitionsFactory.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/FormRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/OpenSearchRequestParser.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/LocalResourceIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/NutchResourceIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/RemoteResourceIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/distributed/AlphaPartitionedIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/WaybackConstants.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/WaybackConstants.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/WaybackConstants.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -24,6 +24,7 @@
*/
package org.archive.wayback;
+
/**
*
*
@@ -48,319 +49,6 @@
public static final String DNS_URL_PREFIX = "dns:";
/**
- * Request: (query) filter results to those prefixed with this (possibly
- * partial) 14-digit timestamp
- */
- public static final String REQUEST_DATE = "date";
-
- /**
- * Request: filter results before this 14-digit timestamp
- */
- public static final String REQUEST_START_DATE = "startdate";
-
- /**
- * Request: filter results after this 14-digit timestamp
- */
- public static final String REQUEST_END_DATE = "enddate";
-
- /**
- * Request: (replay) find closest result to this 14-digit timestamp
- */
- public static final String REQUEST_EXACT_DATE = "exactdate";
-
- /**
- * Request: URL or URL prefix requested
- */
- public static final String REQUEST_URL = "url";
-
- /**
- * Request: Original URL or URL prefix requested.
- * This version differs from @{link {@link #REQUEST_URL} in that its
- * the URL before it was passed via the UURIFactory cleanup.
- */
- public static final String REQUEST_URL_CLEANED = "cleanedurl";
-
- /**
- * Request: URL of referrer, if supplied, or "" if not
- */
- public static final String REQUEST_REFERER_URL = "refererurl";
-
- /**
- * Request: defines type - urlquery, urlprefixquery, or replay
- */
- public static final String REQUEST_TYPE = "type";
-
- /**
- * Request: xml data requested
- */
- public static final String REQUEST_XML_DATA = "xmldata";
-
- /**
- * Request: urlquery type request
- */
- public static final String REQUEST_URL_QUERY = "urlquery";
-
- /**
- * Request: urlprefixquery type request
- */
- public static final String REQUEST_URL_PREFIX_QUERY = "urlprefixquery";
-
- /**
- * Request: replay type request
- */
- public static final String REQUEST_REPLAY_QUERY = "replay";
-
- /**
- * Request: closest type request
- */
- public static final String REQUEST_CLOSEST_QUERY = "urlclosestquery";
-
- /**
- * Request: resolution of results to be displayed: (TimeLine mode)
- */
- public static final String REQUEST_RESOLUTION = "resolution";
-
- /**
- * Request: replay actual document or metadata for document: "yes" means
- * replay metadata only, not the actual document: (TimeLine mode)
- */
- public static final String REQUEST_META_MODE = "metamode";
-
- /**
- * Request: hour resolution (TimeLine mode)
- */
- public static final String REQUEST_RESOLUTION_HOURS = "hours";
-
- /**
- * Request: day resolution (TimeLine mode)
- */
- public static final String REQUEST_RESOLUTION_DAYS = "days";
-
- /**
- * Request: month resolution (TimeLine mode)
- */
- public static final String REQUEST_RESOLUTION_MONTHS = "months";
-
- /**
- * Request: two-month resolution (TimeLine mode)
- */
- public static final String REQUEST_RESOLUTION_TWO_MONTHS = "twomonths";
-
- /**
- * Request: year resolution (TimeLine mode)
- */
- public static final String REQUEST_RESOLUTION_YEARS = "years";
-
- /**
- * Request: auto resolution (TimeLine mode)
- */
- public static final String REQUEST_RESOLUTION_AUTO = "auto";
-
- /**
- * Request: Remote Address, string IP address: "127.0.0.1"
- */
- public static final String REQUEST_REMOTE_ADDRESS = "remoteaddress";
-
- /**
- * Request: Wayback Hostname: the string "Host:" HTTP header
- */
- public static final String REQUEST_WAYBACK_HOSTNAME = "waybackhostname";
-
- /**
- * Request: Wayback Port: the port the remote user connected to for this
- * request.
- */
- public static final String REQUEST_WAYBACK_PORT = "waybackport";
-
- /**
- * Request: Wayback Context: the string context used in the request,
- * if applicable.
- */
- public static final String REQUEST_WAYBACK_CONTEXT = "waybackcontext";
-
- /**
- * Request: Authorization Type: "BASIC", "SSL", or "" if none.
- */
- public static final String REQUEST_AUTH_TYPE = "requestauthtype";
-
- /**
- * Request: Remote User or "" if the request did not contain auth info.
- */
- public static final String REQUEST_REMOTE_USER = "requestremoteuser";
-
- /**
- * Request: Best Guess at users requested locale.
- */
- public static final String REQUEST_LOCALE_LANG = "requestlocalelang";
-
- /**
- * Request: Indicates user only wants results that exactly match the
- * requested hostname -- no canonicalization.
- */
- public static final String REQUEST_EXACT_HOST_ONLY = "requestexacthost";
- /**
- * Request: indicates positive value for any request boolean flag.
- */
- public static final String REQUEST_YES = "yes";
-
- /**
- * Results: type of results: "Capture" or "Url"
- */
- public static final String RESULTS_TYPE = "resultstype";
-
- /**
- * Results: indicates SearchResult objects within the SearchResults are of
- * type UrlSearchResults.
- */
- public static final String RESULTS_TYPE_URL = "resultstypeurl";
-
- /**
- * Results: indicates SearchResult objects within the SearchResults are of
- * type CaptureSearchResults.
- */
- public static final String RESULTS_TYPE_CAPTURE = "resultstypecapture";
-
- /**
- * Results: int total number of records matching, not all necc. returned.
- */
- public static final String RESULTS_NUM_RESULTS = "numresults";
-
- /**
- * Results: int first record of all matching returned, 1-based
- */
- public static final String RESULTS_FIRST_RETURNED = "firstreturned";
-
- /**
- * Results: int total number of records *returned* in results
- */
- public static final String RESULTS_NUM_RETURNED = "numreturned";
-
- /**
- * Results: int number of results requested
- */
- public static final String RESULTS_REQUESTED = "resultsrequested";
-
-
- /**
- * Result: as close as possible to original URL of captured document given
- * informatio accessible only within the index.
- *
- * This may not always yield the exact URL requested: some canonicalization
- * operations are irreversible.
- * Example: lowercasing of GET URI encoded arguments
- * http://foo.com/q?a=%3F => foo.com/q?a=%3f
- * Example: rearrangement of GET URI encoded arguments
- * http://foo.com/q?b=1&a=1 => foo.com/q?a=1&b=1
- * Example: removal of sessionID information from request path
- * http://foo.com/SESSION_A2KSM2/i.htm => foo.com/i.htm
- *
- */
- public static final String RESULT_URL = "url";
-
- /**
- * Result: canonicalized(lookup key) form of URL of captured document
- */
- public static final String RESULT_URL_KEY = "urlkey";
-
- /**
- * Result: 14-digit timestamp when document was captured
- */
- public static final String RESULT_CAPTURE_DATE = "capturedate";
-
- /**
- * Result: basename of ARC file containing this document.
- */
- public static final String RESULT_ARC_FILE = "arcfile";
-
- /**
- * Result: compressed byte offset within ARC file where this document's
- * gzip envelope begins.
- */
- public static final String RESULT_OFFSET = "compressedoffset";
-
- /**
- * Result: compressed byte offset within ARC file where this document's
- * gzip envelope Ends.
- */
- public static final String RESULT_END_OFFSET = "compressedendoffset";
-
- /**
- * Result: original exact host from which this document was captured.
- */
- public static final String RESULT_ORIG_HOST = "originalhost";
-
- /**
- * Result: best-guess at mime-type of this document.
- */
- public static final String RESULT_MIME_TYPE = "mimetype";
-
- /**
- * Result: 3-digit integer HTTP response code. may be '0' in some
- * fringe conditions, old ARCs, bug in crawler, etc.
- */
- public static final String RESULT_HTTP_CODE = "httpresponsecode";
-
- /**
- * Result: all or part of the 32-digit hexadecimal MD5 digest of this
- * document
- */
- public static final String RESULT_MD5_DIGEST= "md5digest";
-
- /**
- * Result: URL that this document redirected to, or '-' if it does
- * not redirect
- */
- public static final String RESULT_REDIRECT_URL = "redirecturl";
-
- /**
- * Result: flag within a SearchResult that indicates this is the closest to
- * a particular requested date.
- */
- public static final String RESULT_CLOSEST_INDICATOR = "closest";
- public static final String RESULT_CLOSEST_VALUE = "true";
-
- /**
- * Result: this key being present indicates that this particular capture
- * was not actually stored, and that other values within this SearchResult
- * are actually values from a different record which *should* be identical
- * to this capture, had it been stored.
- */
- public static final String RESULT_DUPLICATE_ANNOTATION = "duplicate";
-
- /**
- * Result: this key is present when the RESULT_DUPLICATE_ANNOTATION is also
- * present, with the value indicating the last date that was actually
- * stored for this duplicate.
- */
- public static final String RESULT_DUPLICATE_STORED_DATE = "duplicate-date";
-
- /**
- * flag indicates that this document was downloaded and verified as
- * identical to a previous capture by digest.
- */
- public static final String RESULT_DUPLICATE_DIGEST = "digest";
-
- /**
- * flag indicates that this document was NOT downloaded, but that the
- * origin server indicated that the document had not changed, based on
- * If-Modified HTTP request headers.
- */
- public static final String RESULT_DUPLICATE_HTTP = "http";
-
-
- /**
- * Name of configuration in web.xml for maximum number of results to return
- * in index searches.
- */
- public static final String MAX_RESULTS_CONFIG_NAME = "maxresults";
-
- /**
- * Name of configuration in web.xml for default number of results to show
- * on each page
- */
- public static final String RESULTS_PER_PAGE_CONFIG_NAME = "resultsperpage";
-
- /**
* HTTP Header for redirection URL
*/
public final static String LOCATION_HTTP_HEADER = "Location";
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/ArchivalUrlReplayDispatcher.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/ArchivalUrlReplayDispatcher.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/ArchivalUrlReplayDispatcher.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -28,7 +28,6 @@
import org.archive.wayback.ReplayDispatcher;
import org.archive.wayback.ReplayRenderer;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.Resource;
import org.archive.wayback.core.CaptureSearchResult;
import org.archive.wayback.core.WaybackRequest;
@@ -76,7 +75,7 @@
// if the result is not for the exact date requested, redirect to the
// exact date. some capture dates are not 14 digits, only compare as
// many digits as are in the result date:
- String reqDateStr = wbRequest.get(WaybackConstants.REQUEST_DATE);
+ String reqDateStr = wbRequest.get(WaybackRequest.REQUEST_DATE);
String resDateStr = result.getCaptureTimestamp();
if(!resDateStr.equals(reqDateStr.substring(0, resDateStr.length()))) {
return redirect;
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDatePrefixQueryRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDatePrefixQueryRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDatePrefixQueryRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -28,7 +28,6 @@
import java.util.regex.Pattern;
import org.apache.commons.httpclient.URIException;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -67,10 +66,10 @@
startDate = Timestamp.parseBefore(dateStr).getDateStr();
endDate = Timestamp.parseAfter(dateStr).getDateStr();
}
- wbRequest.put(WaybackConstants.REQUEST_START_DATE,startDate);
- wbRequest.put(WaybackConstants.REQUEST_END_DATE,endDate);
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_URL_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_START_DATE,startDate);
+ wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_URL_QUERY);
try {
wbRequest.setRequestUrl(urlStr);
} catch (URIException e) {
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDateRangeQueryRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDateRangeQueryRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathDateRangeQueryRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -28,7 +28,6 @@
import java.util.regex.Pattern;
import org.apache.commons.httpclient.URIException;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -62,10 +61,10 @@
String startDate = Timestamp.parseBefore(startDateStr).getDateStr();
String endDate = Timestamp.parseAfter(endDateStr).getDateStr();
- wbRequest.put(WaybackConstants.REQUEST_START_DATE,startDate);
- wbRequest.put(WaybackConstants.REQUEST_END_DATE,endDate);
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_URL_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_START_DATE,startDate);
+ wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_URL_QUERY);
try {
wbRequest.setRequestUrl(urlStr);
} catch (URIException e) {
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDatePrefixQueryRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDatePrefixQueryRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDatePrefixQueryRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -28,7 +28,6 @@
import java.util.regex.Pattern;
import org.apache.commons.httpclient.URIException;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -67,12 +66,12 @@
endDate = Timestamp.parseAfter(dateStr).getDateStr();
}
- wbRequest.put(WaybackConstants.REQUEST_START_DATE,
+ wbRequest.put(WaybackRequest.REQUEST_START_DATE,
startDate);
- wbRequest.put(WaybackConstants.REQUEST_END_DATE,endDate);
+ wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_URL_PREFIX_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_URL_PREFIX_QUERY);
try {
wbRequest.setRequestUrl(urlStr);
} catch (URIException e) {
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDateRangeQueryRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDateRangeQueryRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/PathPrefixDateRangeQueryRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -28,7 +28,6 @@
import java.util.regex.Pattern;
import org.apache.commons.httpclient.URIException;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -59,12 +58,12 @@
String urlStr = matcher.group(3);
String startDate = Timestamp.parseBefore(startDateStr).getDateStr();
String endDate = Timestamp.parseAfter(endDateStr).getDateStr();
- wbRequest.put(WaybackConstants.REQUEST_START_DATE,
+ wbRequest.put(WaybackRequest.REQUEST_START_DATE,
startDate);
- wbRequest.put(WaybackConstants.REQUEST_END_DATE,endDate);
+ wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_URL_PREFIX_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_URL_PREFIX_QUERY);
try {
wbRequest.setRequestUrl(urlStr);
} catch (URIException e) {
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/archivalurl/requestparser/ReplayRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -30,7 +30,6 @@
import org.apache.commons.httpclient.URIException;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.requestparser.PathRequestParser;
@@ -87,12 +86,12 @@
dateStr = Timestamp.parseAfter(dateStr).getDateStr();
}
- wbRequest.put(WaybackConstants.REQUEST_EXACT_DATE, dateStr);
- wbRequest.put(WaybackConstants.REQUEST_START_DATE, startDate);
- wbRequest.put(WaybackConstants.REQUEST_END_DATE, endDate);
+ wbRequest.put(WaybackRequest.REQUEST_EXACT_DATE, dateStr);
+ wbRequest.put(WaybackRequest.REQUEST_START_DATE, startDate);
+ wbRequest.put(WaybackRequest.REQUEST_END_DATE, endDate);
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_REPLAY_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_REPLAY_QUERY);
try {
// String wbPrefix = wbRequest.getDefaultWaybackPrefix();
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/HTTPAuthBooleanOperator.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/HTTPAuthBooleanOperator.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/HTTPAuthBooleanOperator.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -2,7 +2,6 @@
import java.util.List;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.util.operator.BooleanOperator;
@@ -12,7 +11,7 @@
if(allowedUsers == null) {
return false;
}
- String currentUser = value.get(WaybackConstants.REQUEST_REMOTE_USER);
+ String currentUser = value.get(WaybackRequest.REQUEST_REMOTE_USER);
if(currentUser == null) {
return false;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/IPMatchesBooleanOperator.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/IPMatchesBooleanOperator.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/authenticationcontrol/IPMatchesBooleanOperator.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -4,7 +4,6 @@
import java.util.List;
import java.util.logging.Logger;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.util.IPRange;
import org.archive.wayback.util.operator.BooleanOperator;
@@ -34,7 +33,7 @@
if(allowedRanges == null) {
return false;
}
- String ipString = value.get(WaybackConstants.REQUEST_REMOTE_ADDRESS);
+ String ipString = value.get(WaybackRequest.REQUEST_REMOTE_ADDRESS);
if(ipString == null) {
return false;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -28,7 +28,6 @@
import java.util.Date;
import java.util.Iterator;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.exception.AnchorWindowTooSmallException;
/**
@@ -113,7 +112,7 @@
String anchorDate = wbRequest.get(WaybackRequest.REQUEST_ANCHOR_DATE);
long maxWindow = -1;
long wantTime = Timestamp.parseBefore(wbRequest
- .get(WaybackConstants.REQUEST_EXACT_DATE)).getDate().getTime();
+ .get(WaybackRequest.REQUEST_EXACT_DATE)).getDate().getTime();
if(anchorDate != null) {
wantTime = Timestamp.parseBefore(anchorDate).getDate().getTime();
String anchorWindow = wbRequest.get(WaybackRequest.REQUEST_ANCHOR_WINDOW);
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/SearchResults.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/SearchResults.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/SearchResults.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -69,6 +69,23 @@
private long firstReturned = -1;
private long matchingCount = -1;
private long numRequested = -1;
+
+ /**
+ * Results: indicates SearchResult objects within the SearchResults are of
+ * type UrlSearchResults.
+ */
+ public static final String RESULTS_TYPE_URL = "resultstypeurl";
+
+ /**
+ * Results: indicates SearchResult objects within the SearchResults are of
+ * type CaptureSearchResults.
+ */
+ public static final String RESULTS_TYPE_CAPTURE = "resultstypecapture";
+
+ /**
+ * Results: type of results: "Capture" or "Url"
+ */
+ public static final String RESULTS_TYPE = "resultstype";
/**
* @param key
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/UIResults.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/UIResults.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/UIResults.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -30,7 +30,6 @@
import org.apache.commons.httpclient.URIException;
import org.archive.wayback.ResultURIConverter;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.util.StringFormatter;
import org.archive.wayback.webapp.AccessPoint;
@@ -73,8 +72,8 @@
public String makeCaptureQueryUrl(String url) {
WaybackRequest newWBR = wbRequest.clone();
- newWBR.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_URL_QUERY);
+ newWBR.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_URL_QUERY);
try {
newWBR.setRequestUrl(url);
} catch (URIException e) {
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -37,7 +37,6 @@
import org.apache.commons.httpclient.URIException;
import org.archive.net.UURI;
import org.archive.net.UURIFactory;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.requestparser.OpenSearchRequestParser;
import org.archive.wayback.util.ObjectFilter;
import org.archive.wayback.util.StringFormatter;
@@ -67,17 +66,144 @@
private HashMap<String,String> filters = new HashMap<String,String>();
private StringFormatter formatter = null;
+ /**
+ * Request: Authorization Type: "BASIC", "SSL", or "" if none.
+ */
+ public static final String REQUEST_AUTH_TYPE = "requestauthtype";
+ /**
+ * Request: Wayback Context: the string context used in the request,
+ * if applicable.
+ */
+ public static final String REQUEST_WAYBACK_CONTEXT = "waybackcontext";
+ /**
+ * Request: Wayback Port: the port the remote user connected to for this
+ * request.
+ */
+ public static final String REQUEST_WAYBACK_PORT = "waybackport";
+ /**
+ * Request: Wayback Hostname: the string "Host:" HTTP header
+ */
+ public static final String REQUEST_WAYBACK_HOSTNAME = "waybackhostname";
+ /**
+ * Request: Remote Address, string IP address: "127.0.0.1"
+ */
+ public static final String REQUEST_REMOTE_ADDRESS = "remoteaddress";
+ /**
+ * Request: auto resolution (TimeLine mode)
+ */
+ public static final String REQUEST_RESOLUTION_AUTO = "auto";
+ /**
+ * Request: year resolution (TimeLine mode)
+ */
+ public static final String REQUEST_RESOLUTION_YEARS = "years";
+ /**
+ * Request: two-month resolution (TimeLine mode)
+ */
+ public static final String REQUEST_RESOLUTION_TWO_MONTHS = "twomonths";
+ /**
+ * Request: month resolution (TimeLine mode)
+ */
+ public static final String REQUEST_RESOLUTION_MONTHS = "months";
+ /**
+ * Request: day resolution (TimeLine mode)
+ */
+ public static final String REQUEST_RESOLUTION_DAYS = "days";
+ /**
+ * Request: hour resolution (TimeLine mode)
+ */
+ public static final String REQUEST_RESOLUTION_HOURS = "hours";
+ /**
+ * Request: replay actual document or metadata for document: "yes" means
+ * replay metadata only, not the actual document: (TimeLine mode)
+ */
+ public static final String REQUEST_META_MODE = "metamode";
+ /**
+ * Request: resolution of results to be displayed: (TimeLine mode)
+ */
+ public static final String REQUEST_RESOLUTION = "resolution";
+ /**
+ * Request: closest type request
+ */
+ public static final String REQUEST_CLOSEST_QUERY = "urlclosestquery";
+ /**
+ * Request: replay type request
+ */
+ public static final String REQUEST_REPLAY_QUERY = "replay";
+ /**
+ * Request: urlprefixquery type request
+ */
+ public static final String REQUEST_URL_PREFIX_QUERY = "urlprefixquery";
+ /**
+ * Request: urlquery type request
+ */
+ public static final String REQUEST_URL_QUERY = "urlquery";
+ /**
+ * Request: xml data requested
+ */
+ public static final String REQUEST_XML_DATA = "xmldata";
+ /**
+ * Request: defines type - urlquery, urlprefixquery, or replay
+ */
+ public static final String REQUEST_TYPE = "type";
+ /**
+ * Request: URL of referrer, if supplied, or "" if not
+ */
+ public static final String REQUEST_REFERER_URL = "refererurl";
+ /**
+ * Request: Original URL or URL prefix requested.
+ * This version differs from @{link {@link REQUEST_URL} in that its
+ * the URL before it was passed via the UURIFactory cleanup.
+ */
+ public static final String REQUEST_URL_CLEANED = "cleanedurl";
+ /**
+ * Request: URL or URL prefix requested
+ */
+ public static final String REQUEST_URL = "url";
+ /**
+ * Request: (replay) find closest result to this 14-digit timestamp
+ */
+ public static final String REQUEST_EXACT_DATE = "exactdate";
+ /**
+ * Request: filter results after this 14-digit timestamp
+ */
+ public static final String REQUEST_END_DATE = "enddate";
+ /**
+ * Request: filter results before this 14-digit timestamp
+ */
+ public static final String REQUEST_START_DATE = "startdate";
+ /**
+ * Request: (query) filter results to those prefixed with this (possibly
+ * partial) 14-digit timestamp
+ */
+ public static final String REQUEST_DATE = "date";
+ /**
+ * Request: Remote User or "" if the request did not contain auth info.
+ */
+ public static final String REQUEST_REMOTE_USER = "requestremoteuser";
+ /**
+ * Request: Best Guess at users requested locale.
+ */
+ public static final String REQUEST_LOCALE_LANG = "requestlocalelang";
+ /**
+ * Request: Indicates user only wants results that exactly match the
+ * requested hostname -- no canonicalization.
+ */
+ public static final String REQUEST_EXACT_HOST_ONLY = "requestexacthost";
+ /**
+ * Request: indicates positive value for any request boolean flag.
+ */
+ public static final String REQUEST_YES = "yes";
private static String UI_RESOURCE_BUNDLE_NAME = "WaybackUI";
private final static String standardHeaders[] = {
- WaybackConstants.REQUEST_REFERER_URL,
- WaybackConstants.REQUEST_REMOTE_ADDRESS,
- WaybackConstants.REQUEST_WAYBACK_HOSTNAME,
- WaybackConstants.REQUEST_WAYBACK_PORT,
- WaybackConstants.REQUEST_WAYBACK_CONTEXT,
- WaybackConstants.REQUEST_AUTH_TYPE,
- WaybackConstants.REQUEST_REMOTE_USER,
- WaybackConstants.REQUEST_LOCALE_LANG };
+ WaybackRequest.REQUEST_REFERER_URL,
+ WaybackRequest.REQUEST_REMOTE_ADDRESS,
+ WaybackRequest.REQUEST_WAYBACK_HOSTNAME,
+ WaybackRequest.REQUEST_WAYBACK_PORT,
+ WaybackRequest.REQUEST_WAYBACK_CONTEXT,
+ WaybackRequest.REQUEST_AUTH_TYPE,
+ WaybackRequest.REQUEST_REMOTE_USER,
+ WaybackRequest.REQUEST_LOCALE_LANG };
/**
* Constructor, possibly/probably this should BE a Properties, instead of
@@ -91,8 +217,8 @@
* @return true if REQUEST_TYPE is set, and is set to REQUEST_REPLAY_QUERY
*/
public boolean isReplayRequest() {
- String type = get(WaybackConstants.REQUEST_TYPE);
- if(type != null && type.equals(WaybackConstants.REQUEST_REPLAY_QUERY)) {
+ String type = get(WaybackRequest.REQUEST_TYPE);
+ if(type != null && type.equals(WaybackRequest.REQUEST_REPLAY_QUERY)) {
return true;
}
return false;
@@ -191,21 +317,21 @@
*/
private void extractHttpRequestInfo(HttpServletRequest httpRequest) {
// attempt to get the HTTP referer if present..
- put(WaybackConstants.REQUEST_REFERER_URL, emptyIfNull(httpRequest
+ put(WaybackRequest.REQUEST_REFERER_URL, emptyIfNull(httpRequest
.getHeader("REFERER")));
- put(WaybackConstants.REQUEST_REMOTE_ADDRESS, emptyIfNull(httpRequest
+ put(WaybackRequest.REQUEST_REMOTE_ADDRESS, emptyIfNull(httpRequest
.getRemoteAddr()));
- put(WaybackConstants.REQUEST_WAYBACK_HOSTNAME, emptyIfNull(httpRequest
+ put(WaybackRequest.REQUEST_WAYBACK_HOSTNAME, emptyIfNull(httpRequest
.getLocalName()));
- put(WaybackConstants.REQUEST_WAYBACK_PORT, String.valueOf(httpRequest
+ put(WaybackRequest.REQUEST_WAYBACK_PORT, String.valueOf(httpRequest
.getLocalPort()));
- put(WaybackConstants.REQUEST_WAYBACK_CONTEXT, emptyIfNull(httpRequest
+ put(WaybackRequest.REQUEST_WAYBACK_CONTEXT, emptyIfNull(httpRequest
.getContextPath()));
- put(WaybackConstants.REQUEST_AUTH_TYPE, emptyIfNull(httpRequest
+ put(WaybackRequest.REQUEST_AUTH_TYPE, emptyIfNull(httpRequest
.getAuthType()));
- put(WaybackConstants.REQUEST_REMOTE_USER, emptyIfNull(httpRequest
+ put(WaybackRequest.REQUEST_REMOTE_USER, emptyIfNull(httpRequest
.getRemoteUser()));
- put(WaybackConstants.REQUEST_LOCALE_LANG,getUserLocale(httpRequest));
+ put(WaybackRequest.REQUEST_LOCALE_LANG,getUserLocale(httpRequest));
Cookie[] cookies = httpRequest.getCookies();
if(cookies != null) {
@@ -264,32 +390,32 @@
*/
public void fixup(HttpServletRequest httpRequest) {
extractHttpRequestInfo(httpRequest);
- String startDate = get(WaybackConstants.REQUEST_START_DATE);
- String endDate = get(WaybackConstants.REQUEST_END_DATE);
- String exactDate = get(WaybackConstants.REQUEST_EXACT_DATE);
- String partialDate = get(WaybackConstants.REQUEST_DATE);
+ String startDate = get(WaybackRequest.REQUEST_START_DATE);
+ String endDate = get(WaybackRequest.REQUEST_END_DATE);
+ String exactDate = get(WaybackRequest.REQUEST_EXACT_DATE);
+ String partialDate = get(WaybackRequest.REQUEST_DATE);
if (partialDate == null) {
partialDate = "";
}
if (startDate == null || startDate.length() == 0) {
- put(WaybackConstants.REQUEST_START_DATE, Timestamp
+ put(WaybackRequest.REQUEST_START_DATE, Timestamp
.padStartDateStr(partialDate));
} else if (startDate.length() < 14) {
- put(WaybackConstants.REQUEST_START_DATE, Timestamp
+ put(WaybackRequest.REQUEST_START_DATE, Timestamp
.padStartDateStr(startDate));
}
if (endDate == null || endDate.length() == 0) {
- put(WaybackConstants.REQUEST_END_DATE, Timestamp
+ put(WaybackRequest.REQUEST_END_DATE, Timestamp
.padEndDateStr(partialDate));
} else if (endDate.length() < 14) {
- put(WaybackConstants.REQUEST_END_DATE, Timestamp
+ put(WaybackRequest.REQUEST_END_DATE, Timestamp
.padEndDateStr(endDate));
}
if (exactDate == null || exactDate.length() == 0) {
- put(WaybackConstants.REQUEST_EXACT_DATE, Timestamp
+ put(WaybackRequest.REQUEST_EXACT_DATE, Timestamp
.padEndDateStr(partialDate));
} else if (exactDate.length() < 14) {
- put(WaybackConstants.REQUEST_EXACT_DATE, Timestamp
+ put(WaybackRequest.REQUEST_EXACT_DATE, Timestamp
.padEndDateStr(exactDate));
}
}
@@ -359,8 +485,8 @@
}
// If its not http, next line throws exception. TODO: Fix.
UURI requestURI = UURIFactory.getInstance(urlStr);
- put(WaybackConstants.REQUEST_URL_CLEANED, requestURI.toString());
- put(WaybackConstants.REQUEST_URL, urlStr);
+ put(WaybackRequest.REQUEST_URL_CLEANED, requestURI.toString());
+ put(WaybackRequest.REQUEST_URL, urlStr);
}
/**
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixReplayDispatcher.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixReplayDispatcher.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixReplayDispatcher.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -26,7 +26,6 @@
import org.archive.wayback.ReplayDispatcher;
import org.archive.wayback.ReplayRenderer;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.CaptureSearchResult;
import org.archive.wayback.core.Resource;
import org.archive.wayback.core.WaybackRequest;
@@ -60,7 +59,7 @@
// if the result is not for the exact date requested, redirect to the
// exact date. some capture dates are not 14 digits, only compare as
// many digits as are in the result date:
- String reqDateStr = wbRequest.get(WaybackConstants.REQUEST_EXACT_DATE);
+ String reqDateStr = wbRequest.get(WaybackRequest.REQUEST_EXACT_DATE);
String resDateStr = result.getCaptureTimestamp();
if((resDateStr.length() > reqDateStr.length()) ||
!resDateStr.equals(reqDateStr.substring(0, resDateStr.length()))) {
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/domainprefix/DomainPrefixRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -30,7 +30,6 @@
import javax.servlet.http.HttpServletRequest;
import org.apache.commons.httpclient.URIException;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.exception.BadQueryException;
@@ -90,9 +89,9 @@
String requestUrl = getRequestString(host,httpRequest);
- wbRequest.put(WaybackConstants.REQUEST_EXACT_DATE, dateStr);
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_REPLAY_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_EXACT_DATE, dateStr);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_REPLAY_QUERY);
try {
wbRequest.setRequestUrl(requestUrl);
} catch (URIException e) {
@@ -114,10 +113,10 @@
startDate = Timestamp.parseBefore(dateStr).getDateStr();
endDate = Timestamp.parseAfter(dateStr).getDateStr();
}
- wbRequest.put(WaybackConstants.REQUEST_START_DATE,startDate);
- wbRequest.put(WaybackConstants.REQUEST_END_DATE,endDate);
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_URL_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_START_DATE,startDate);
+ wbRequest.put(WaybackRequest.REQUEST_END_DATE,endDate);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_URL_QUERY);
String requestUrl = getRequestString(host,httpRequest);
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/BaseExceptionRenderer.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/BaseExceptionRenderer.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/BaseExceptionRenderer.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -34,7 +34,6 @@
import javax.servlet.http.HttpServletResponse;
import org.archive.wayback.ExceptionRenderer;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.UIResults;
import org.archive.wayback.core.WaybackRequest;
@@ -69,13 +68,13 @@
if (wbRequest == null) {
return false;
}
- String referer = wbRequest.get(WaybackConstants.REQUEST_REFERER_URL);
+ String referer = wbRequest.get(WaybackRequest.REQUEST_REFERER_URL);
return (referer != null && referer.length() > 0);
}
protected boolean requestIsImage(HttpServletRequest httpRequest,
WaybackRequest wbRequest) {
- String requestUrl = wbRequest.get(WaybackConstants.REQUEST_URL);
+ String requestUrl = wbRequest.get(WaybackRequest.REQUEST_URL);
if (requestUrl == null)
return false;
Matcher matcher = IMAGE_REGEX.matcher(requestUrl);
@@ -85,14 +84,14 @@
protected boolean requestIsJavascript(HttpServletRequest httpRequest,
WaybackRequest wbRequest) {
- String requestUrl = wbRequest.get(WaybackConstants.REQUEST_URL);
+ String requestUrl = wbRequest.get(WaybackRequest.REQUEST_URL);
return (requestUrl != null) && requestUrl.endsWith(".js");
}
protected boolean requestIsCSS(HttpServletRequest httpRequest,
WaybackRequest wbRequest) {
- String requestUrl = wbRequest.get(WaybackConstants.REQUEST_URL);
+ String requestUrl = wbRequest.get(WaybackRequest.REQUEST_URL);
return (requestUrl != null) && requestUrl.endsWith(".css");
}
@@ -104,7 +103,7 @@
if(wbRequest.isQueryRequest()) {
- if(wbRequest.containsKey(WaybackConstants.REQUEST_XML_DATA)) {
+ if(wbRequest.containsKey(WaybackRequest.REQUEST_XML_DATA)) {
jspPath = xmlErrorJsp;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -33,14 +33,12 @@
import org.archive.io.arc.ARCLocation;
import org.archive.io.arc.ARCRecord;
import org.archive.wayback.UrlCanonicalizer;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.Resource;
import org.archive.wayback.core.CaptureSearchResult;
import org.archive.wayback.core.CaptureSearchResults;
import org.archive.wayback.core.SearchResults;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
-import org.archive.wayback.exception.AnchorWindowTooSmallException;
import org.archive.wayback.exception.LiveDocumentNotAvailableException;
import org.archive.wayback.exception.ResourceNotInArchiveException;
import org.archive.wayback.exception.WaybackException;
@@ -82,9 +80,9 @@
boolean bUseOlder) throws URIException {
WaybackRequest req = new WaybackRequest();
req.setRequestUrl(url.toString());
- req.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_CLOSEST_QUERY);
- req.put(WaybackConstants.REQUEST_EXACT_DATE,
+ req.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_CLOSEST_QUERY);
+ req.put(WaybackRequest.REQUEST_EXACT_DATE,
Timestamp.currentTimestamp().getDateStr());
Timestamp earliest = null;
if(bUseOlder) {
@@ -93,11 +91,11 @@
Date d = new Date(System.currentTimeMillis() - maxCacheMS);
earliest = new Timestamp(d);
}
- req.put(WaybackConstants.REQUEST_START_DATE,earliest.getDateStr());
+ req.put(WaybackRequest.REQUEST_START_DATE,earliest.getDateStr());
// for now, assume all live web requests are only satisfiable by the
// exact host -- no massaging.
- req.put(WaybackConstants.REQUEST_EXACT_HOST_ONLY,
- WaybackConstants.REQUEST_YES);
+ req.put(WaybackRequest.REQUEST_EXACT_HOST_ONLY,
+ WaybackRequest.REQUEST_YES);
return req;
}
@@ -166,12 +164,7 @@
e.printStackTrace();
throw new IOException(e.getMessage());
}
- CaptureSearchResult result;
- try {
- result = results.getClosest(wbRequest,false);
- } catch (AnchorWindowTooSmallException e) {
- throw new ResourceNotInArchiveException("Not In archive..");
- }
+ CaptureSearchResult result = results.getClosest(wbRequest);
if(result != null) {
if(isForgedFailedSearchResult(result)) {
if(isForgedFailRecentEnough(result)) {
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyReplayRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyReplayRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyReplayRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -30,7 +30,6 @@
import org.apache.commons.httpclient.URIException;
import org.archive.util.InetAddressUtil;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.exception.BadQueryException;
import org.archive.wayback.requestparser.BaseRequestParser;
@@ -92,8 +91,8 @@
e.printStackTrace();
return null;
}
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_REPLAY_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_REPLAY_QUERY);
wbRequest.setResultsPerPage(maxRecords);
return wbRequest;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/proxy/ProxyRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -29,7 +29,6 @@
import javax.servlet.http.HttpServletRequest;
import org.archive.wayback.RequestParser;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.exception.BadQueryException;
@@ -71,7 +70,7 @@
String id = httpRequest.getHeader("Proxy-Id");
if (id == null)
id = httpRequest.getRemoteAddr();
- wbRequest.put(WaybackConstants.REQUEST_EXACT_DATE, Timestamp
+ wbRequest.put(WaybackRequest.REQUEST_EXACT_DATE, Timestamp
.getTimestampForId(httpRequest.getContextPath(), id));
wbRequest.fixup(httpRequest);
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/Renderer.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/Renderer.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/Renderer.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -33,7 +33,6 @@
import org.archive.wayback.QueryRenderer;
import org.archive.wayback.ResultURIConverter;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.CaptureSearchResults;
import org.archive.wayback.core.UrlSearchResults;
import org.archive.wayback.core.WaybackRequest;
@@ -76,7 +75,7 @@
UICaptureQueryResults uiResults = new UICaptureQueryResults(httpRequest, wbRequest,
results, uriConverter);
String jsp = captureJsp;
- if(wbRequest.containsKey(WaybackConstants.REQUEST_XML_DATA)) {
+ if(wbRequest.containsKey(WaybackRequest.REQUEST_XML_DATA)) {
jsp = xmlCaptureJsp;
}
@@ -96,7 +95,7 @@
UIUrlQueryResults uiResults = new UIUrlQueryResults(httpRequest, wbRequest,
results, uriConverter);
String jsp = urlJsp;
- if(wbRequest.containsKey(WaybackConstants.REQUEST_XML_DATA)) {
+ if(wbRequest.containsKey(WaybackRequest.REQUEST_XML_DATA)) {
jsp = xmlUrlJsp;
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/UIQueryResults.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/UIQueryResults.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/UIQueryResults.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -28,7 +28,6 @@
import javax.servlet.http.HttpServletRequest;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.ResultURIConverter;
import org.archive.wayback.core.CaptureSearchResult;
import org.archive.wayback.core.SearchResults;
@@ -80,11 +79,11 @@
WaybackRequest wbRequest, SearchResults results,
ResultURIConverter uriConverter) {
super(wbRequest,uriConverter);
- this.searchUrl = wbRequest.get(WaybackConstants.REQUEST_URL);
+ this.searchUrl = wbRequest.get(WaybackRequest.REQUEST_URL);
this.startTimestamp = Timestamp.parseBefore(results.
- getFilter(WaybackConstants.REQUEST_START_DATE));
+ getFilter(WaybackRequest.REQUEST_START_DATE));
this.endTimestamp = Timestamp.parseAfter(results.getFilter(
- WaybackConstants.REQUEST_END_DATE));
+ WaybackRequest.REQUEST_END_DATE));
this.resultsReturned = results.getReturnedCount();
this.resultsMatching = results.getMatchingCount();
@@ -92,7 +91,7 @@
this.firstResult = results.getFirstReturned() + 1;
this.lastResult = ((firstResult - 1) + resultsReturned);
this.exactRequestedTimestamp = Timestamp.parseAfter(
- wbRequest.get(WaybackConstants.REQUEST_EXACT_DATE));
+ wbRequest.get(WaybackRequest.REQUEST_EXACT_DATE));
// calculate total pages:
numPages = (int) Math.ceil((double)resultsMatching/(double)resultsPerPage);
curPage = (int) Math.floor(((double)(firstResult-1))/(double)resultsPerPage) + 1;
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/resultspartitioner/ResultsPartitionsFactory.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/resultspartitioner/ResultsPartitionsFactory.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/resultspartitioner/ResultsPartitionsFactory.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -29,7 +29,6 @@
import java.util.Date;
import org.archive.util.ArchiveUtils;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.CaptureSearchResults;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
@@ -59,9 +58,9 @@
public static ArrayList<ResultsPartition> get(CaptureSearchResults results,
WaybackRequest wbRequest) {
Timestamp startTS = Timestamp.parseBefore(results.getFilter(
- WaybackConstants.REQUEST_START_DATE));
+ WaybackRequest.REQUEST_START_DATE));
Timestamp endTS = Timestamp.parseAfter(results.getFilter(
- WaybackConstants.REQUEST_END_DATE));
+ WaybackRequest.REQUEST_END_DATE));
String rsd = startTS.getDateStr();
String red = endTS.getDateStr();
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/resultspartitioner/ResultsTimelinePartitionsFactory.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/resultspartitioner/ResultsTimelinePartitionsFactory.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/query/resultspartitioner/ResultsTimelinePartitionsFactory.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -28,7 +28,6 @@
import java.util.Calendar;
import org.archive.util.ArchiveUtils;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.CaptureSearchResults;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
@@ -145,15 +144,15 @@
int last = Timestamp.parseAfter(results.getLastResultTimestamp()).sse();
int diff = last - first;
if(diff < MAX_HOUR_SECONDS) {
- return WaybackConstants.REQUEST_RESOLUTION_HOURS;
+ return WaybackRequest.REQUEST_RESOLUTION_HOURS;
} else if(diff < MAX_DAY_SECONDS) {
- return WaybackConstants.REQUEST_RESOLUTION_DAYS;
+ return WaybackRequest.REQUEST_RESOLUTION_DAYS;
} else if(diff < MAX_MONTH_SECONDS) {
- return WaybackConstants.REQUEST_RESOLUTION_MONTHS;
+ return WaybackRequest.REQUEST_RESOLUTION_MONTHS;
} else if(diff < MAX_TWO_MONTH_SECONDS) {
- return WaybackConstants.REQUEST_RESOLUTION_TWO_MONTHS;
+ return WaybackRequest.REQUEST_RESOLUTION_TWO_MONTHS;
}
- return WaybackConstants.REQUEST_RESOLUTION_YEARS;
+ return WaybackRequest.REQUEST_RESOLUTION_YEARS;
}
private static ArrayList<ResultsPartition> get(ResultsPartitioner
@@ -167,7 +166,7 @@
int totalPartitions = (partitionCount * 2) + 1; // total # of partitions
// first calculate the "center" based on the exact request date:
- String reqDate = results.getFilter(WaybackConstants.REQUEST_EXACT_DATE);
+ String reqDate = results.getFilter(WaybackRequest.REQUEST_EXACT_DATE);
Calendar centerCal = partitioner.dateStrToCalendar(reqDate);
partitioner.alignStart(centerCal);
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/FormRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/FormRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/FormRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -30,7 +30,6 @@
import javax.servlet.http.HttpServletRequest;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.webapp.AccessPoint;
@@ -61,15 +60,15 @@
String base = wbContext.translateRequestPath(httpRequest);
if(base.startsWith(REPLAY_BASE)) {
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_REPLAY_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_REPLAY_QUERY);
} else if(base.startsWith(QUERY_BASE)) {
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_URL_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_URL_QUERY);
} else if(base.startsWith(XQUERY_BASE)){
- wbRequest.put(WaybackConstants.REQUEST_TYPE,
- WaybackConstants.REQUEST_URL_QUERY);
- wbRequest.put(WaybackConstants.REQUEST_XML_DATA,"1");
+ wbRequest.put(WaybackRequest.REQUEST_TYPE,
+ WaybackRequest.REQUEST_URL_QUERY);
+ wbRequest.put(WaybackRequest.REQUEST_XML_DATA,"1");
} else {
return null;
@@ -86,12 +85,12 @@
String val = getMapParam(queryMap,key);
wbRequest.put(key,val);
}
- if(wbRequest.get(WaybackConstants.REQUEST_START_DATE) == null) {
- wbRequest.put(WaybackConstants.REQUEST_START_DATE,
+ if(wbRequest.get(WaybackRequest.REQUEST_START_DATE) == null) {
+ wbRequest.put(WaybackRequest.REQUEST_START_DATE,
getEarliestTimestamp());
}
- if(wbRequest.get(WaybackConstants.REQUEST_END_DATE) == null) {
- wbRequest.put(WaybackConstants.REQUEST_END_DATE,
+ if(wbRequest.get(WaybackRequest.REQUEST_END_DATE) == null) {
+ wbRequest.put(WaybackRequest.REQUEST_END_DATE,
getLatestTimestamp());
}
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/OpenSearchRequestParser.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/OpenSearchRequestParser.java 2008-07-07 21:53:54 UTC (rev 2416)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/requestparser/OpenSearchRequestParser.java 2008-07-07 21:57:30 UTC (rev 2417)
@@ -29,7 +29,6 @@
import javax.servlet.http.HttpServletRequest;
-import org.archive.wayback.WaybackConstants;
import org.archive.wayback.core.WaybackRequest;
import org.archive.wayback.exception.BadQueryException;
import org.archive.wayback.webapp.AccessPoint;
@@ -87,15 +86,15 @@
String base = wbContext.translateRequestPath(httpRequest);
if(base...
[truncated message content] |
|
From: <bra...@us...> - 2008-07-07 21:53:52
|
Revision: 2416
http://archive-access.svn.sourceforge.net/archive-access/?rev=2416&view=rev
Author: bradtofel
Date: 2008-07-07 14:53:54 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
REFACTOR: moved WaybackConstants.REQUEST* to WaybackRequest class.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/query/XMLCaptureResults.jsp
trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/query/XMLUrlResults.jsp
Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/query/XMLCaptureResults.jsp
===================================================================
--- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/query/XMLCaptureResults.jsp 2008-07-07 21:35:06 UTC (rev 2415)
+++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/query/XMLCaptureResults.jsp 2008-07-07 21:53:54 UTC (rev 2416)
@@ -4,9 +4,9 @@
<%@ page import="java.util.ArrayList" %>
<%@ page import="java.util.Map" %>
<%@ page import="java.util.Enumeration" %>
-<%@ page import="org.archive.wayback.WaybackConstants" %>
<%@ page import="org.archive.wayback.core.CaptureSearchResult" %>
<%@ page import="org.archive.wayback.core.CaptureSearchResults" %>
+<%@ page import="org.archive.wayback.core.SearchResults" %>
<%@ page import="org.archive.wayback.core.Timestamp" %>
<%@ page import="org.archive.wayback.core.UIResults" %>
<%@ page import="org.archive.wayback.query.UICaptureQueryResults" %>
@@ -30,7 +30,7 @@
<%
}
%>
- <<%= WaybackConstants.RESULTS_TYPE %>><%= WaybackConstants.RESULTS_TYPE_CAPTURE %></<%= WaybackConstants.RESULTS_TYPE %>>
+ <<%= SearchResults.RESULTS_TYPE %>><%= SearchResults.RESULTS_TYPE_CAPTURE %></<%= SearchResults.RESULTS_TYPE %>>
</request>
<results>
<%
Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/query/XMLUrlResults.jsp
===================================================================
--- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/query/XMLUrlResults.jsp 2008-07-07 21:35:06 UTC (rev 2415)
+++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/query/XMLUrlResults.jsp 2008-07-07 21:53:54 UTC (rev 2416)
@@ -4,11 +4,11 @@
<%@ page import="java.util.ArrayList" %>
<%@ page import="java.util.Map" %>
<%@ page import="java.util.Enumeration" %>
-<%@ page import="org.archive.wayback.WaybackConstants" %>
+<%@ page import="org.archive.wayback.core.SearchResults" %>
+<%@ page import="org.archive.wayback.core.Timestamp" %>
+<%@ page import="org.archive.wayback.core.UIResults" %>
<%@ page import="org.archive.wayback.core.UrlSearchResults" %>
<%@ page import="org.archive.wayback.core.UrlSearchResult" %>
-<%@ page import="org.archive.wayback.core.Timestamp" %>
-<%@ page import="org.archive.wayback.core.UIResults" %>
<%@ page import="org.archive.wayback.query.UIUrlQueryResults" %>
<%
UIUrlQueryResults uiResults = (UIUrlQueryResults) UIResults.getFromRequest(request);
@@ -30,7 +30,7 @@
<%
}
%>
- <<%= WaybackConstants.RESULTS_TYPE %>><%= WaybackConstants.RESULTS_TYPE_URL %></<%= WaybackConstants.RESULTS_TYPE %>>
+ <<%= SearchResults.RESULTS_TYPE %>><%= SearchResults.RESULTS_TYPE_URL %></<%= SearchResults.RESULTS_TYPE %>>
</request>
<results>
<%
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
Revision: 2415
http://archive-access.svn.sourceforge.net/archive-access/?rev=2415&view=rev
Author: bradtofel
Date: 2008-07-07 14:35:06 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
TWEAK: return from all InterruptedExceptions.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexWorker.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexWorker.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexWorker.java 2008-07-07 21:34:45 UTC (rev 2414)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexWorker.java 2008-07-07 21:35:06 UTC (rev 2415)
@@ -161,6 +161,7 @@
}
} catch (InterruptedException e) {
e.printStackTrace();
+ return;
} catch (IOException e) {
e.printStackTrace();
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-07 21:34:37
|
Revision: 2414
http://archive-access.svn.sourceforge.net/archive-access/?rev=2414&view=rev
Author: bradtofel
Date: 2008-07-07 14:34:45 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
INTERAFACE: now call getClosest() with true optional second argument, causing an AnchorWindowTooSmallException if no results are within the requested window.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java 2008-07-07 21:33:24 UTC (rev 2413)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/webapp/AccessPoint.java 2008-07-07 21:34:45 UTC (rev 2414)
@@ -311,7 +311,8 @@
CaptureSearchResults captureResults = (CaptureSearchResults) results;
// TODO: check which versions are actually accessible right now?
- CaptureSearchResult closest = captureResults.getClosest(wbRequest);
+ CaptureSearchResult closest = captureResults.getClosest(wbRequest,
+ true);
resource = collection.getResourceStore().retrieveResource(closest);
ReplayRenderer renderer = replay.getRenderer(wbRequest, closest, resource);
renderer.renderResource(httpRequest, httpResponse, wbRequest,
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
Revision: 2413
http://archive-access.svn.sourceforge.net/archive-access/?rev=2413&view=rev
Author: bradtofel
Date: 2008-07-07 14:33:24 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
BUGFIX: was not throwing exception(BadQuery) when no results are within the requested Window. This is a Bad Query, because the UI shouldn't generate links to result pages beyond those possible, so the user has manually changed the pageNum argument.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/LocalResourceIndex.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/LocalResourceIndex.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/LocalResourceIndex.java 2008-07-07 21:31:49 UTC (rev 2412)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/LocalResourceIndex.java 2008-07-07 21:33:24 UTC (rev 2413)
@@ -476,10 +476,14 @@
public ObjectFilter<T> getFilter() {
return windowFilters;
}
- public void annotateResults(SearchResults results) {
+ public void annotateResults(SearchResults results)
+ throws BadQueryException {
results.setFirstReturned(startResult);
results.setNumRequested(resultsPerPage);
-
+ int numSeen = endFilter.getNumSeen();
+ if(numSeen == 0) {
+ throw new BadQueryException("No results in requested window");
+ }
// how many went by the filters:
results.setMatchingCount(startFilter.getNumSeen());
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-07 21:31:40
|
Revision: 2412
http://archive-access.svn.sourceforge.net/archive-access/?rev=2412&view=rev
Author: bradtofel
Date: 2008-07-07 14:31:49 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
TWEAK: return from all InterruptedExceptions.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/cdx/dynamic/DynamicCDXIndex.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/updater/LocalResourceIndexUpdater.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/AutoIndexThread.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexQueueUpdater.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/ResourceFileLocationDBUpdater.java
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/resourcefile/ResourceFileSourceUpdater.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/cdx/dynamic/DynamicCDXIndex.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/cdx/dynamic/DynamicCDXIndex.java 2008-07-07 21:30:27 UTC (rev 2411)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/cdx/dynamic/DynamicCDXIndex.java 2008-07-07 21:31:49 UTC (rev 2412)
@@ -335,6 +335,7 @@
sleep(sleepInterval);
} catch (InterruptedException e) {
e.printStackTrace();
+ return;
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/updater/LocalResourceIndexUpdater.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/updater/LocalResourceIndexUpdater.java 2008-07-07 21:30:27 UTC (rev 2411)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/updater/LocalResourceIndexUpdater.java 2008-07-07 21:31:49 UTC (rev 2412)
@@ -293,6 +293,7 @@
}
} catch (InterruptedException e) {
e.printStackTrace();
+ return;
}
}
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/AutoIndexThread.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/AutoIndexThread.java 2008-07-07 21:30:27 UTC (rev 2411)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/AutoIndexThread.java 2008-07-07 21:31:49 UTC (rev 2412)
@@ -56,6 +56,7 @@
}
} catch (InterruptedException e) {
e.printStackTrace();
+ return;
}
}
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexQueueUpdater.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexQueueUpdater.java 2008-07-07 21:30:27 UTC (rev 2411)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/indexer/IndexQueueUpdater.java 2008-07-07 21:31:49 UTC (rev 2412)
@@ -151,6 +151,7 @@
sleep(sleepInterval);
} catch (InterruptedException e) {
e.printStackTrace();
+ return;
} catch (IOException e) {
e.printStackTrace();
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/ResourceFileLocationDBUpdater.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/ResourceFileLocationDBUpdater.java 2008-07-07 21:30:27 UTC (rev 2411)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/locationdb/ResourceFileLocationDBUpdater.java 2008-07-07 21:31:49 UTC (rev 2412)
@@ -168,6 +168,7 @@
sleep(sleepInterval);
} catch (InterruptedException e) {
e.printStackTrace();
+ return;
} catch (IOException e) {
e.printStackTrace();
}
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/resourcefile/ResourceFileSourceUpdater.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/resourcefile/ResourceFileSourceUpdater.java 2008-07-07 21:30:27 UTC (rev 2411)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/resourcefile/ResourceFileSourceUpdater.java 2008-07-07 21:31:49 UTC (rev 2412)
@@ -125,6 +125,7 @@
}
} catch (InterruptedException e) {
e.printStackTrace();
+ return;
}
}
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-07 21:30:17
|
Revision: 2411
http://archive-access.svn.sourceforge.net/archive-access/?rev=2411&view=rev
Author: bradtofel
Date: 2008-07-07 14:30:27 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
INTERFACE: now call getClosest with optional 2nd arg to prevent exception.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java 2008-07-07 21:29:53 UTC (rev 2410)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/liveweb/LiveWebCache.java 2008-07-07 21:30:27 UTC (rev 2411)
@@ -40,6 +40,7 @@
import org.archive.wayback.core.SearchResults;
import org.archive.wayback.core.Timestamp;
import org.archive.wayback.core.WaybackRequest;
+import org.archive.wayback.exception.AnchorWindowTooSmallException;
import org.archive.wayback.exception.LiveDocumentNotAvailableException;
import org.archive.wayback.exception.ResourceNotInArchiveException;
import org.archive.wayback.exception.WaybackException;
@@ -165,7 +166,12 @@
e.printStackTrace();
throw new IOException(e.getMessage());
}
- CaptureSearchResult result = results.getClosest(wbRequest);
+ CaptureSearchResult result;
+ try {
+ result = results.getClosest(wbRequest,false);
+ } catch (AnchorWindowTooSmallException e) {
+ throw new ResourceNotInArchiveException("Not In archive..");
+ }
if(result != null) {
if(isForgedFailedSearchResult(result)) {
if(isForgedFailRecentEnough(result)) {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
Revision: 2410
http://archive-access.svn.sourceforge.net/archive-access/?rev=2410&view=rev
Author: bradtofel
Date: 2008-07-07 14:29:53 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
COMMENT: fixed javadoc.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java 2008-07-07 21:28:14 UTC (rev 2409)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java 2008-07-07 21:29:53 UTC (rev 2410)
@@ -100,6 +100,8 @@
}
/**
* @param wbRequest
+ * @param err if true, then check Request Anchor Window and Date, throwing
+ * exception if no Result is within the Window.
* @return The closest CaptureSearchResult to the request.
*/
public CaptureSearchResult getClosest(WaybackRequest wbRequest, boolean err)
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-07 21:28:07
|
Revision: 2409
http://archive-access.svn.sourceforge.net/archive-access/?rev=2409&view=rev
Author: bradtofel
Date: 2008-07-07 14:28:14 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
TWEAK: added static Anchor Window and Date Strings.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-07 21:26:40 UTC (rev 2408)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/WaybackRequest.java 2008-07-07 21:28:14 UTC (rev 2409)
@@ -52,6 +52,9 @@
*/
public class WaybackRequest {
+ public final static String REQUEST_ANCHOR_DATE = "request.anchordate";
+ public final static String REQUEST_ANCHOR_WINDOW = "request.anchorwindow";
+
private int resultsPerPage = 10;
private int pageNum = 1;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
Revision: 2408
http://archive-access.svn.sourceforge.net/archive-access/?rev=2408&view=rev
Author: bradtofel
Date: 2008-07-07 14:26:40 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
FEATURE: added awareness of Anchor Date and Anchor Window to getClosest method, which can optionally throw an exception if no dates are within the range specified.
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java
Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java 2008-07-07 21:25:23 UTC (rev 2407)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/core/CaptureSearchResults.java 2008-07-07 21:26:40 UTC (rev 2408)
@@ -29,6 +29,7 @@
import java.util.Iterator;
import org.archive.wayback.WaybackConstants;
+import org.archive.wayback.exception.AnchorWindowTooSmallException;
/**
*
@@ -78,17 +79,46 @@
return new Timestamp(lastResultTimestamp).getDate();
}
+ public void markClosest(WaybackRequest wbRequest) {
+ CaptureSearchResult closest = getClosest(wbRequest);
+ if(closest != null) {
+ closest.setClosest(true);
+ }
+ }
/**
* @param wbRequest
* @return The closest CaptureSearchResult to the request.
*/
public CaptureSearchResult getClosest(WaybackRequest wbRequest) {
+ try {
+ return getClosest(wbRequest,false);
+ } catch (AnchorWindowTooSmallException e) {
+ // cannot happen with 2nd arg false...
+ e.printStackTrace();
+ }
+ return null;
+ }
+ /**
+ * @param wbRequest
+ * @return The closest CaptureSearchResult to the request.
+ */
+ public CaptureSearchResult getClosest(WaybackRequest wbRequest, boolean err)
+ throws AnchorWindowTooSmallException {
CaptureSearchResult closest = null;
long closestDistance = 0;
CaptureSearchResult cur = null;
+ String anchorDate = wbRequest.get(WaybackRequest.REQUEST_ANCHOR_DATE);
+ long maxWindow = -1;
long wantTime = Timestamp.parseBefore(wbRequest
.get(WaybackConstants.REQUEST_EXACT_DATE)).getDate().getTime();
+ if(anchorDate != null) {
+ wantTime = Timestamp.parseBefore(anchorDate).getDate().getTime();
+ String anchorWindow = wbRequest.get(WaybackRequest.REQUEST_ANCHOR_WINDOW);
+ if(anchorWindow != null) {
+ maxWindow = Long.parseLong(anchorWindow);
+ }
+ }
Iterator<CaptureSearchResult> itr = results.iterator();
while (itr.hasNext()) {
@@ -101,6 +131,13 @@
closestDistance = curDistance;
}
}
+ if(err && (maxWindow != -1)) {
+ if(closestDistance > maxWindow) {
+ throw new AnchorWindowTooSmallException("Closest is " +
+ closestDistance + " seconds away, Window is " +
+ maxWindow);
+ }
+ }
return closest;
}
/**
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
Revision: 2407
http://archive-access.svn.sourceforge.net/archive-access/?rev=2407&view=rev
Author: bradtofel
Date: 2008-07-07 14:25:23 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
INITIAL REV: Exception indicating no resources are within the specified Anchor Window.
Added Paths:
-----------
trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/AnchorWindowTooSmallException.java
Added: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/AnchorWindowTooSmallException.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/AnchorWindowTooSmallException.java (rev 0)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/exception/AnchorWindowTooSmallException.java 2008-07-07 21:25:23 UTC (rev 2407)
@@ -0,0 +1,52 @@
+/* AnchorWindowTooSmallException
+ *
+ * $Id$
+ *
+ * Created on 4:16:45 PM Jul 3, 2008.
+ *
+ * Copyright (C) 2008 Internet Archive.
+ *
+ * This file is part of wayback.
+ *
+ * wayback is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser Public License as published by
+ * the Free Software Foundation; either version 2.1 of the License, or
+ * any later version.
+ *
+ * wayback is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser Public License
+ * along with wayback; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+package org.archive.wayback.exception;
+
+import javax.servlet.http.HttpServletResponse;
+
+/**
+ *
+ *
+ * @author brad
+ * @version $Date$, $Revision$
+ */
+public class AnchorWindowTooSmallException extends WaybackException {
+ private static final long serialVersionUID = 1L;
+ protected static final String ID = "anchorWindowTooSmall";
+ public AnchorWindowTooSmallException(String message) {
+ super(message,"AnchorWindow Too Small");
+ id = ID;
+ }
+ public AnchorWindowTooSmallException(String message,String details) {
+ super(message,"AnchorWindow Too Small",details);
+ id = ID;
+ }
+ /**
+ * @return the HTTP status code appropriate to this exception class.
+ */
+ public int getStatus() {
+ return HttpServletResponse.SC_PRECONDITION_FAILED;
+ }
+}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <bra...@us...> - 2008-07-07 21:24:29
|
Revision: 2406
http://archive-access.svn.sourceforge.net/archive-access/?rev=2406&view=rev
Author: bradtofel
Date: 2008-07-07 14:24:37 -0700 (Mon, 07 Jul 2008)
Log Message:
-----------
TWEAK: added placeholder anchor window error message
Modified Paths:
--------------
trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI.properties
trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI_de.properties
trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI_fr_CA.properties
Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI.properties
===================================================================
--- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI.properties 2008-07-03 22:01:43 UTC (rev 2405)
+++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI.properties 2008-07-07 21:24:37 UTC (rev 2406)
@@ -18,6 +18,8 @@
Exception.resourceNotAvailable.message=The Resource you have requested is temporarily unavailable. Please try again later.
Exception.resourceNotInArchive.title=Resource Not In Archive
Exception.resourceNotInArchive.message=The Resource you requested is not in this archive.
+Exception.anchorWindowTooSmall.title=No Resources within Window Range
+Exception.anchorWindowTooSmall.message=The Window Range specified does not match any Resources. You can try expanding the Anchor Window size and repeating the request.
UIGlobal.pageTitle=Internet Archive Wayback Machine
UIGlobal.helpLink=Help
Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI_de.properties
===================================================================
--- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI_de.properties 2008-07-03 22:01:43 UTC (rev 2405)
+++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI_de.properties 2008-07-07 21:24:37 UTC (rev 2406)
@@ -18,6 +18,8 @@
Exception.resourceNotAvailable.message=Die angeforderte Ressource ist zwischenzeitlich nicht verfügbar. Bitte versuchen Sie es später nocheinmal.
Exception.resourceNotInArchive.title=Ressource ist nicht im Archiv
Exception.resourceNotInArchive.message=Die angeforderte Ressource ist nicht im Archiv.
+Exception.anchorWindowTooSmall.title=No Resources within Window Range
+Exception.anchorWindowTooSmall.message=The Window Range specified does not match any Resources. You can try expanding the Anchor Window size and repeating the request.
UIGlobal.pageTitle=Internet Archive Wayback Machine
UIGlobal.helpLink=Hilfe
Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI_fr_CA.properties
===================================================================
--- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI_fr_CA.properties 2008-07-03 22:01:43 UTC (rev 2405)
+++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/classes/WaybackUI_fr_CA.properties 2008-07-07 21:24:37 UTC (rev 2406)
@@ -14,6 +14,8 @@
Exception.resourceNotAvailable.message=[The Resource you have requested is temporarily unavailable. Please try again later.]
Exception.resourceNotInArchive.title=[Resource Not In Archive]
Exception.resourceNotInArchive.message=[The Resource you requested is not in this archive.]
+Exception.anchorWindowTooSmall.title=No Resources within Window Range
+Exception.anchorWindowTooSmall.message=The Window Range specified does not match any Resources. You can try expanding the Anchor Window size and repeating the request.
UIGlobal.pageTitle=[Internet Archive Wayback Machine]
UIGlobal.helpLink=[Help]
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
