You can subscribe to this list here.
2005 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
(10) |
Sep
(36) |
Oct
(339) |
Nov
(103) |
Dec
(152) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2006 |
Jan
(141) |
Feb
(102) |
Mar
(125) |
Apr
(203) |
May
(57) |
Jun
(30) |
Jul
(139) |
Aug
(46) |
Sep
(64) |
Oct
(105) |
Nov
(34) |
Dec
(162) |
2007 |
Jan
(81) |
Feb
(57) |
Mar
(141) |
Apr
(72) |
May
(9) |
Jun
(1) |
Jul
(144) |
Aug
(88) |
Sep
(40) |
Oct
(43) |
Nov
(34) |
Dec
(20) |
2008 |
Jan
(44) |
Feb
(45) |
Mar
(16) |
Apr
(36) |
May
(8) |
Jun
(77) |
Jul
(177) |
Aug
(66) |
Sep
(8) |
Oct
(33) |
Nov
(13) |
Dec
(37) |
2009 |
Jan
(2) |
Feb
(5) |
Mar
(8) |
Apr
|
May
(36) |
Jun
(19) |
Jul
(46) |
Aug
(8) |
Sep
(1) |
Oct
(66) |
Nov
(61) |
Dec
(10) |
2010 |
Jan
(13) |
Feb
(16) |
Mar
(38) |
Apr
(76) |
May
(47) |
Jun
(32) |
Jul
(35) |
Aug
(45) |
Sep
(20) |
Oct
(61) |
Nov
(24) |
Dec
(16) |
2011 |
Jan
(22) |
Feb
(34) |
Mar
(11) |
Apr
(8) |
May
(24) |
Jun
(23) |
Jul
(11) |
Aug
(42) |
Sep
(81) |
Oct
(48) |
Nov
(21) |
Dec
(20) |
2012 |
Jan
(30) |
Feb
(25) |
Mar
(4) |
Apr
(6) |
May
(1) |
Jun
(5) |
Jul
(5) |
Aug
(8) |
Sep
(6) |
Oct
(6) |
Nov
|
Dec
|
From: <bra...@us...> - 2007-10-15 21:28:29
|
Revision: 2055 http://archive-access.svn.sourceforge.net/archive-access/?rev=2055&view=rev Author: bradtofel Date: 2007-10-15 14:28:33 -0700 (Mon, 15 Oct 2007) Log Message: ----------- BUGFIX: proxy mode requires useServerName=true bean configuration. Modified Paths: -------------- trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml Modified: trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml =================================================================== --- trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml 2007-10-15 21:22:32 UTC (rev 2054) +++ trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml 2007-10-15 21:28:33 UTC (rev 2055) @@ -1014,6 +1014,7 @@ <pre> <bean name="8090" parent="8080:wayback"> + <property name="useServerName" value="true" /> <property name="replay"> <bean class="org.archive.wayback.proxy.ProxyReplayDispatcher" /> </property> Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml 2007-10-15 21:22:32 UTC (rev 2054) +++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml 2007-10-15 21:28:33 UTC (rev 2055) @@ -265,6 +265,7 @@ --> <!-- <bean name="8090" parent="8080:wayback"> + <property name="useServerName" value="true" /> <property name="replay"> <bean class="org.archive.wayback.proxy.ProxyReplayDispatcher" /> </property> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-15 21:22:39
|
Revision: 2054 http://archive-access.svn.sourceforge.net/archive-access/?rev=2054&view=rev Author: bradtofel Date: 2007-10-15 14:22:32 -0700 (Mon, 15 Oct 2007) Log Message: ----------- BUGFIX: uriconverter => uriConverter also WHITESPACE changes Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml 2007-10-14 08:25:37 UTC (rev 2053) +++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml 2007-10-15 21:22:32 UTC (rev 2054) @@ -264,25 +264,25 @@ in your Tomcat's server.xml file. --> <!-- - <bean name="8090" parent="8080:wayback"> - <property name="replay"> - <bean class="org.archive.wayback.proxy.ProxyReplayDispatcher" /> - </property> - <property name="uriconverter"> - <bean class="org.archive.wayback.proxy.RedirectResultURIConverter"> - <property name="redirectURI" value="http://foo.archive.org:8090/jsp/Redirect.jsp" /> - </bean> - </property> - <property name="parser"> - <bean class="org.archive.wayback.proxy.ProxyRequestParser" init-method="init"> - <property name="localhostNames"> - <list> - <value>foo.archive.org</value> - </list> - </property> - <property name="maxRecords" value="1000" /> - </bean> - </property> - </bean> +<bean name="8090" parent="8080:wayback"> + <property name="replay"> + <bean class="org.archive.wayback.proxy.ProxyReplayDispatcher" /> + </property> + <property name="uriConverter"> + <bean class="org.archive.wayback.proxy.RedirectResultURIConverter"> + <property name="redirectURI" value="http://foo.archive.org:8090/jsp/Redirect.jsp" /> + </bean> + </property> + <property name="parser"> + <bean class="org.archive.wayback.proxy.ProxyRequestParser" init-method="init"> + <property name="localhostNames"> + <list> + <value>foo.archive.org</value> + </list> + </property> + <property name="maxRecords" value="1000" /> + </bean> + </property> +</bean> --> </beans> \ No newline at end of file This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <pau...@us...> - 2007-10-14 08:25:34
|
Revision: 2053 http://archive-access.svn.sourceforge.net/archive-access/?rev=2053&view=rev Author: paul_jack Date: 2007-10-14 01:25:37 -0700 (Sun, 14 Oct 2007) Log Message: ----------- Changed all pom.xml to use version 1.0.0, testing the build system (Continuum). Modified Paths: -------------- trunk/archive-access/projects/wayback/dist/pom.xml trunk/archive-access/projects/wayback/pom.xml trunk/archive-access/projects/wayback/wayback-core/pom.xml trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml trunk/archive-access/projects/wayback/wayback-webapp/pom.xml Modified: trunk/archive-access/projects/wayback/dist/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/dist/pom.xml 2007-10-13 01:23:09 UTC (rev 2052) +++ trunk/archive-access/projects/wayback/dist/pom.xml 2007-10-14 08:25:37 UTC (rev 2053) @@ -3,7 +3,7 @@ <parent> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0-SNAPSHOT</version> + <version>1.0.0</version> </parent> <modelVersion>4.0.0</modelVersion> @@ -54,13 +54,13 @@ <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-webapp</artifactId> - <version>1.0.0-SNAPSHOT</version> + <version>1.0.0</version> <type>war</type> </dependency> <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce</artifactId> - <version>1.0.0-SNAPSHOT</version> + <version>1.0.0</version> </dependency> </dependencies> Modified: trunk/archive-access/projects/wayback/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/pom.xml 2007-10-13 01:23:09 UTC (rev 2052) +++ trunk/archive-access/projects/wayback/pom.xml 2007-10-14 08:25:37 UTC (rev 2053) @@ -16,7 +16,7 @@ <modelVersion>4.0.0</modelVersion> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0-SNAPSHOT</version> + <version>1.0.0</version> <packaging>pom</packaging> <name>Wayback</name> Modified: trunk/archive-access/projects/wayback/wayback-core/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-core/pom.xml 2007-10-13 01:23:09 UTC (rev 2052) +++ trunk/archive-access/projects/wayback/wayback-core/pom.xml 2007-10-14 08:25:37 UTC (rev 2053) @@ -17,7 +17,7 @@ <parent> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0-SNAPSHOT</version> + <version>1.0.0</version> </parent> <groupId>org.archive.wayback</groupId> <artifactId>wayback-core</artifactId> Modified: trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml 2007-10-13 01:23:09 UTC (rev 2052) +++ trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml 2007-10-14 08:25:37 UTC (rev 2053) @@ -12,7 +12,7 @@ <parent> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0-SNAPSHOT</version> + <version>1.0.0</version> </parent> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce</artifactId> Modified: trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml 2007-10-13 01:23:09 UTC (rev 2052) +++ trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml 2007-10-14 08:25:37 UTC (rev 2053) @@ -10,7 +10,7 @@ <parent> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0-SNAPSHOT</version> + <version>1.0.0</version> </parent> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce-prereq</artifactId> Modified: trunk/archive-access/projects/wayback/wayback-webapp/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-13 01:23:09 UTC (rev 2052) +++ trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-14 08:25:37 UTC (rev 2053) @@ -3,7 +3,7 @@ <parent> <artifactId>wayback</artifactId> <groupId>org.archive</groupId> - <version>1.0.0-SNAPSHOT</version> + <version>1.0.0</version> </parent> <modelVersion>4.0.0</modelVersion> <groupId>org.archive.wayback</groupId> @@ -27,21 +27,21 @@ <dependencies> <dependency> <groupId>org.archive.wayback</groupId> - <artifactId>wayback-core</artifactId> - <scope>compile</scope> -<!-- - <exclusions> - <exclusion> - <groupId>log4j</groupId> - <artifactId>log4j</artifactId> - </exclusion> - </exclusions> ---> + <artifactId>wayback-core</artifactId> + <scope>compile</scope> +<!-- + <exclusions> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> + </exclusions> +--> </dependency> <dependency> <groupId>tomcat</groupId> <artifactId>servlet-api</artifactId> - <version>5.5.15</version> + <version>5.5.15</version> <scope>provided</scope> </dependency> </dependencies> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-13 01:23:06
|
Revision: 2052 http://archive-access.svn.sourceforge.net/archive-access/?rev=2052&view=rev Author: bradtofel Date: 2007-10-12 18:23:09 -0700 (Fri, 12 Oct 2007) Log Message: ----------- back to prev.. Modified Paths: -------------- trunk/archive-access/projects/wayback/pom.xml Modified: trunk/archive-access/projects/wayback/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/pom.xml 2007-10-13 01:17:12 UTC (rev 2051) +++ trunk/archive-access/projects/wayback/pom.xml 2007-10-13 01:23:09 UTC (rev 2052) @@ -155,13 +155,11 @@ <artifactId>wayback-mapreduce-prereq</artifactId> <version>${project.version}</version> </dependency> - <!-- <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce</artifactId> <version>${project.version}</version> </dependency> - --> <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-webapp</artifactId> @@ -173,9 +171,7 @@ <modules> <module>wayback-core</module> <module>wayback-mapreduce-prereq</module> -<!-- <module>wayback-mapreduce</module> ---> <module>wayback-webapp</module> <module>dist</module> </modules> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-13 01:17:08
|
Revision: 2051 http://archive-access.svn.sourceforge.net/archive-access/?rev=2051&view=rev Author: bradtofel Date: 2007-10-12 18:17:12 -0700 (Fri, 12 Oct 2007) Log Message: ----------- test remove map-red jar Modified Paths: -------------- trunk/archive-access/projects/wayback/pom.xml Modified: trunk/archive-access/projects/wayback/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/pom.xml 2007-10-13 00:51:05 UTC (rev 2050) +++ trunk/archive-access/projects/wayback/pom.xml 2007-10-13 01:17:12 UTC (rev 2051) @@ -155,11 +155,13 @@ <artifactId>wayback-mapreduce-prereq</artifactId> <version>${project.version}</version> </dependency> + <!-- <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce</artifactId> <version>${project.version}</version> </dependency> + --> <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-webapp</artifactId> @@ -171,7 +173,9 @@ <modules> <module>wayback-core</module> <module>wayback-mapreduce-prereq</module> +<!-- <module>wayback-mapreduce</module> +--> <module>wayback-webapp</module> <module>dist</module> </modules> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-13 00:51:03
|
Revision: 2050 http://archive-access.svn.sourceforge.net/archive-access/?rev=2050&view=rev Author: bradtofel Date: 2007-10-12 17:51:05 -0700 (Fri, 12 Oct 2007) Log Message: ----------- test... Modified Paths: -------------- trunk/archive-access/projects/wayback/pom.xml Modified: trunk/archive-access/projects/wayback/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/pom.xml 2007-10-13 00:49:08 UTC (rev 2049) +++ trunk/archive-access/projects/wayback/pom.xml 2007-10-13 00:51:05 UTC (rev 2050) @@ -152,17 +152,17 @@ </dependency> <dependency> <groupId>org.archive.wayback</groupId> - <artifactId>wayback-webapp</artifactId> + <artifactId>wayback-mapreduce-prereq</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>org.archive.wayback</groupId> - <artifactId>wayback-mapreduce-prereq</artifactId> + <artifactId>wayback-mapreduce</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>org.archive.wayback</groupId> - <artifactId>wayback-mapreduce</artifactId> + <artifactId>wayback-webapp</artifactId> <version>${project.version}</version> </dependency> </dependencies> @@ -170,9 +170,9 @@ <modules> <module>wayback-core</module> - <module>wayback-webapp</module> <module>wayback-mapreduce-prereq</module> <module>wayback-mapreduce</module> + <module>wayback-webapp</module> <module>dist</module> </modules> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-13 00:49:05
|
Revision: 2049 http://archive-access.svn.sourceforge.net/archive-access/?rev=2049&view=rev Author: bradtofel Date: 2007-10-12 17:49:08 -0700 (Fri, 12 Oct 2007) Log Message: ----------- test... Modified Paths: -------------- trunk/archive-access/projects/wayback/pom.xml Modified: trunk/archive-access/projects/wayback/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/pom.xml 2007-10-13 00:46:23 UTC (rev 2048) +++ trunk/archive-access/projects/wayback/pom.xml 2007-10-13 00:49:08 UTC (rev 2049) @@ -152,17 +152,17 @@ </dependency> <dependency> <groupId>org.archive.wayback</groupId> - <artifactId>wayback-mapreduce-prereq</artifactId> + <artifactId>wayback-webapp</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>org.archive.wayback</groupId> - <artifactId>wayback-mapreduce</artifactId> + <artifactId>wayback-mapreduce-prereq</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>org.archive.wayback</groupId> - <artifactId>wayback-webapp</artifactId> + <artifactId>wayback-mapreduce</artifactId> <version>${project.version}</version> </dependency> </dependencies> @@ -170,9 +170,9 @@ <modules> <module>wayback-core</module> + <module>wayback-webapp</module> <module>wayback-mapreduce-prereq</module> <module>wayback-mapreduce</module> - <module>wayback-webapp</module> <module>dist</module> </modules> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-13 00:46:20
|
Revision: 2048 http://archive-access.svn.sourceforge.net/archive-access/?rev=2048&view=rev Author: bradtofel Date: 2007-10-12 17:46:23 -0700 (Fri, 12 Oct 2007) Log Message: ----------- rev Modified Paths: -------------- trunk/archive-access/projects/wayback/dist/src/main/assembly/distribution.xml Modified: trunk/archive-access/projects/wayback/dist/src/main/assembly/distribution.xml =================================================================== --- trunk/archive-access/projects/wayback/dist/src/main/assembly/distribution.xml 2007-10-13 00:42:44 UTC (rev 2047) +++ trunk/archive-access/projects/wayback/dist/src/main/assembly/distribution.xml 2007-10-13 00:46:23 UTC (rev 2048) @@ -8,7 +8,7 @@ <dependencySets> <dependencySet> <excludes> - <exclude>wayback-webapp-0.9.0-SNAPSHOT.war</exclude> + <exclude>wayback-webapp-1.0.0-SNAPSHOT.war</exclude> </excludes> <outputDirectory>/lib</outputDirectory> </dependencySet> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-13 00:42:53
|
Revision: 2047 http://archive-access.svn.sourceforge.net/archive-access/?rev=2047&view=rev Author: bradtofel Date: 2007-10-12 17:42:44 -0700 (Fri, 12 Oct 2007) Log Message: ----------- test.. Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-webapp/pom.xml Modified: trunk/archive-access/projects/wayback/wayback-webapp/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-13 00:34:52 UTC (rev 2046) +++ trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-13 00:42:44 UTC (rev 2047) @@ -29,12 +29,14 @@ <groupId>org.archive.wayback</groupId> <artifactId>wayback-core</artifactId> <scope>compile</scope> +<!-- <exclusions> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> +--> </dependency> <dependency> <groupId>tomcat</groupId> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-13 00:34:47
|
Revision: 2046 http://archive-access.svn.sourceforge.net/archive-access/?rev=2046&view=rev Author: bradtofel Date: 2007-10-12 17:34:52 -0700 (Fri, 12 Oct 2007) Log Message: ----------- RELEASE: oops added SNAPSHOT??? Modified Paths: -------------- trunk/archive-access/projects/wayback/dist/pom.xml Modified: trunk/archive-access/projects/wayback/dist/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/dist/pom.xml 2007-10-13 00:20:09 UTC (rev 2045) +++ trunk/archive-access/projects/wayback/dist/pom.xml 2007-10-13 00:34:52 UTC (rev 2046) @@ -3,7 +3,7 @@ <parent> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0</version> + <version>1.0.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> @@ -54,13 +54,13 @@ <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-webapp</artifactId> - <version>${project.version}</version> + <version>1.0.0-SNAPSHOT</version> <type>war</type> </dependency> <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce</artifactId> - <version>${project.version}</version> + <version>1.0.0-SNAPSHOT</version> </dependency> </dependencies> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-13 00:20:07
|
Revision: 2045 http://archive-access.svn.sourceforge.net/archive-access/?rev=2045&view=rev Author: bradtofel Date: 2007-10-12 17:20:09 -0700 (Fri, 12 Oct 2007) Log Message: ----------- RELEASE: oops added SNAPSHOT??? Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-core/pom.xml trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml trunk/archive-access/projects/wayback/wayback-webapp/pom.xml Modified: trunk/archive-access/projects/wayback/wayback-core/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-core/pom.xml 2007-10-13 00:14:24 UTC (rev 2044) +++ trunk/archive-access/projects/wayback/wayback-core/pom.xml 2007-10-13 00:20:09 UTC (rev 2045) @@ -17,7 +17,7 @@ <parent> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0</version> + <version>1.0.0-SNAPSHOT</version> </parent> <groupId>org.archive.wayback</groupId> <artifactId>wayback-core</artifactId> Modified: trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml 2007-10-13 00:14:24 UTC (rev 2044) +++ trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml 2007-10-13 00:20:09 UTC (rev 2045) @@ -12,7 +12,7 @@ <parent> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0</version> + <version>1.0.0-SNAPSHOT</version> </parent> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce</artifactId> Modified: trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml 2007-10-13 00:14:24 UTC (rev 2044) +++ trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml 2007-10-13 00:20:09 UTC (rev 2045) @@ -10,7 +10,7 @@ <parent> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0</version> + <version>1.0.0-SNAPSHOT</version> </parent> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce-prereq</artifactId> Modified: trunk/archive-access/projects/wayback/wayback-webapp/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-13 00:14:24 UTC (rev 2044) +++ trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-13 00:20:09 UTC (rev 2045) @@ -3,7 +3,7 @@ <parent> <artifactId>wayback</artifactId> <groupId>org.archive</groupId> - <version>1.0.0</version> + <version>1.0.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <groupId>org.archive.wayback</groupId> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-13 00:14:20
|
Revision: 2044 http://archive-access.svn.sourceforge.net/archive-access/?rev=2044&view=rev Author: bradtofel Date: 2007-10-12 17:14:24 -0700 (Fri, 12 Oct 2007) Log Message: ----------- Added SNAPSHOT. Modified Paths: -------------- trunk/archive-access/projects/wayback/pom.xml Modified: trunk/archive-access/projects/wayback/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/pom.xml 2007-10-12 23:42:53 UTC (rev 2043) +++ trunk/archive-access/projects/wayback/pom.xml 2007-10-13 00:14:24 UTC (rev 2044) @@ -16,7 +16,7 @@ <modelVersion>4.0.0</modelVersion> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>1.0.0</version> + <version>1.0.0-SNAPSHOT</version> <packaging>pom</packaging> <name>Wayback</name> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-12 23:42:50
|
Revision: 2043 http://archive-access.svn.sourceforge.net/archive-access/?rev=2043&view=rev Author: bradtofel Date: 2007-10-12 16:42:53 -0700 (Fri, 12 Oct 2007) Log Message: ----------- Release 1.0.0 Added Paths: ----------- branches/wayback-1_0_0/wayback/ Copied: branches/wayback-1_0_0/wayback (from rev 2042, trunk/archive-access/projects/wayback) This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-12 23:41:31
|
Revision: 2042 http://archive-access.svn.sourceforge.net/archive-access/?rev=2042&view=rev Author: bradtofel Date: 2007-10-12 16:41:35 -0700 (Fri, 12 Oct 2007) Log Message: ----------- RELEASE: 1.0.0 Modified Paths: -------------- trunk/archive-access/projects/wayback/dist/pom.xml trunk/archive-access/projects/wayback/pom.xml trunk/archive-access/projects/wayback/wayback-core/pom.xml trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml trunk/archive-access/projects/wayback/wayback-webapp/pom.xml Modified: trunk/archive-access/projects/wayback/dist/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/dist/pom.xml 2007-10-12 23:30:00 UTC (rev 2041) +++ trunk/archive-access/projects/wayback/dist/pom.xml 2007-10-12 23:41:35 UTC (rev 2042) @@ -1,9 +1,9 @@ <?xml version="1.0"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <parent> - <groupId>org.archive</groupId> - <artifactId>wayback</artifactId> - <version>0.9.0-SNAPSHOT</version> + <groupId>org.archive</groupId> + <artifactId>wayback</artifactId> + <version>1.0.0</version> </parent> <modelVersion>4.0.0</modelVersion> @@ -54,13 +54,13 @@ <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-webapp</artifactId> - <version>0.9.0-SNAPSHOT</version> + <version>${project.version}</version> <type>war</type> </dependency> <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce</artifactId> - <version>0.9.0-SNAPSHOT</version> + <version>${project.version}</version> </dependency> </dependencies> Modified: trunk/archive-access/projects/wayback/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/pom.xml 2007-10-12 23:30:00 UTC (rev 2041) +++ trunk/archive-access/projects/wayback/pom.xml 2007-10-12 23:41:35 UTC (rev 2042) @@ -16,7 +16,7 @@ <modelVersion>4.0.0</modelVersion> <groupId>org.archive</groupId> <artifactId>wayback</artifactId> - <version>0.9.0-SNAPSHOT</version> + <version>1.0.0</version> <packaging>pom</packaging> <name>Wayback</name> Modified: trunk/archive-access/projects/wayback/wayback-core/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-core/pom.xml 2007-10-12 23:30:00 UTC (rev 2041) +++ trunk/archive-access/projects/wayback/wayback-core/pom.xml 2007-10-12 23:41:35 UTC (rev 2042) @@ -15,9 +15,9 @@ <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> - <groupId>org.archive</groupId> - <artifactId>wayback</artifactId> - <version>0.9.0-SNAPSHOT</version> + <groupId>org.archive</groupId> + <artifactId>wayback</artifactId> + <version>1.0.0</version> </parent> <groupId>org.archive.wayback</groupId> <artifactId>wayback-core</artifactId> Modified: trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml 2007-10-12 23:30:00 UTC (rev 2041) +++ trunk/archive-access/projects/wayback/wayback-mapreduce/pom.xml 2007-10-12 23:41:35 UTC (rev 2042) @@ -10,9 +10,9 @@ <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> - <groupId>org.archive</groupId> - <artifactId>wayback</artifactId> - <version>0.9.0-SNAPSHOT</version> + <groupId>org.archive</groupId> + <artifactId>wayback</artifactId> + <version>1.0.0</version> </parent> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce</artifactId> Modified: trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml 2007-10-12 23:30:00 UTC (rev 2041) +++ trunk/archive-access/projects/wayback/wayback-mapreduce-prereq/pom.xml 2007-10-12 23:41:35 UTC (rev 2042) @@ -8,9 +8,9 @@ <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> - <groupId>org.archive</groupId> - <artifactId>wayback</artifactId> - <version>0.9.0-SNAPSHOT</version> + <groupId>org.archive</groupId> + <artifactId>wayback</artifactId> + <version>1.0.0</version> </parent> <groupId>org.archive.wayback</groupId> <artifactId>wayback-mapreduce-prereq</artifactId> Modified: trunk/archive-access/projects/wayback/wayback-webapp/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-12 23:30:00 UTC (rev 2041) +++ trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-12 23:41:35 UTC (rev 2042) @@ -3,7 +3,7 @@ <parent> <artifactId>wayback</artifactId> <groupId>org.archive</groupId> - <version>0.9.0-SNAPSHOT</version> + <version>1.0.0</version> </parent> <modelVersion>4.0.0</modelVersion> <groupId>org.archive.wayback</groupId> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-12 23:29:59
|
Revision: 2041 http://archive-access.svn.sourceforge.net/archive-access/?rev=2041&view=rev Author: bradtofel Date: 2007-10-12 16:30:00 -0700 (Fri, 12 Oct 2007) Log Message: ----------- 1.0.0 Added Paths: ----------- branches/wayback-1_0_0/ This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-12 23:21:41
|
Revision: 2040 http://archive-access.svn.sourceforge.net/archive-access/?rev=2040&view=rev Author: bradtofel Date: 2007-10-12 16:21:40 -0700 (Fri, 12 Oct 2007) Log Message: ----------- DOCS: update with info for 1.0 release Modified Paths: -------------- trunk/archive-access/projects/wayback/dist/src/site/xdoc/downloads.xml trunk/archive-access/projects/wayback/dist/src/site/xdoc/index.xml Modified: trunk/archive-access/projects/wayback/dist/src/site/xdoc/downloads.xml =================================================================== --- trunk/archive-access/projects/wayback/dist/src/site/xdoc/downloads.xml 2007-10-12 21:55:10 UTC (rev 2039) +++ trunk/archive-access/projects/wayback/dist/src/site/xdoc/downloads.xml 2007-10-12 23:21:40 UTC (rev 2040) @@ -12,16 +12,16 @@ <subsection name="Releases"> <p>All releases are available off the <a href="http://sourceforge.net/project/showfiles.php?group_id=118427">Sourceforge Downloads</a> page. Release notes can be found here, - <a href="articles/releasenotes.html">Wayback Release Notes</a>. </p> </subsection> <subsection name="Continuous build"> <p>Here is a - <a href="http://builds.archive.org:8080/cruisecontrol/buildresults/HEAD-archive-access">pointer</a> to our continuous build box. The latest builds - can be found under the 'Build Artifacts' link. Be aware that - this distribution has been made from HEAD and HEAD builds are - not guaranteed stable. + <a href="http://builds.archive.org:8081/continuum/">pointer</a> to our + continuous build box. The latest builds can be found under + <a href="http://builds.archive.org:8080/maven2/org/archive/wayback/wayback/"> + our maven2 repository</a> Be aware that this distribution has been made + from HEAD and HEAD builds are not guaranteed stable. </p> </subsection> Modified: trunk/archive-access/projects/wayback/dist/src/site/xdoc/index.xml =================================================================== --- trunk/archive-access/projects/wayback/dist/src/site/xdoc/index.xml 2007-10-12 21:55:10 UTC (rev 2039) +++ trunk/archive-access/projects/wayback/dist/src/site/xdoc/index.xml 2007-10-12 23:21:40 UTC (rev 2040) @@ -9,6 +9,54 @@ <body> <section name="News"> + <subsection name="New Release - 1.0.0, 10/12/2007"> + <p> + Release 1.0.0 has several significant changes, most notably a + completely new configuration mechanism using Spring IOC. This new + configuration system introduces some deployment concepts: + <ul> + <li> + <b>WaybackCollections</b> define a set of documents via the + previously existing ResourceStore and ResourceIndex + implementations. + </li> + <li> + <b>AccessPoints</b> define a method by which users can access + and interact with a WaybackCollection. A single + WaybackCollection may be exposed to users through several + AccessPoints simultaneously. Each AccessPoint specifies an + access URL, a Query interface, a Replay interface, and + several optional access restrictions, including limiting who + can connect to the AccessPoint, and which documents in the + WaybackCollection are available through the AccessPoint. + </li> + </ul> + This new configuration frameworks allows hosting of hundreds of + individual collections within a single wayback installation, each + with potentially multiple AccessPoints. + </p> + <p> + This version also includes a major refactoring of the Replay User + Interface framework, simplifying extension and the creation of novel + replay modes. Specifically, one or more external .jsp files can + be used to generate additional HTML content within replayed HTML + pages. The Timeline Replay mode has been completely replaced by one + of these external .jsp files, which inserts the Timeline banner + inside replayed HTML pages. + </p> + <p> + This version includes a very experimental new Replay mode, + <i>domain-prefix</i> replay mode, which performs all markup and + recontextualization of replayed HTML documents on the server-side, + eliminating the need for client-side Javascript execution. Please + ask on the discussion list for assistance in using this Replay mode. + </p> + <p> + Lastly, this version has some internal improvements which should + reduce memory consumption, and the software is now built using + maven2. + </p> + </subsection> <subsection name="New Release - 0.8.0, 01/11/2007"> <p> Release 0.8.0 offers several new features, most notably a CDX @@ -77,7 +125,7 @@ </subsection> <subsection name="First Release - 0.2.0, 12/09/2005"> <p>First public release of the open source wayback. - See below in the <a href="Introduction">Introduction</a> + See below in the <a href="#Introduction">Introduction</a> section for a listing of initial features. </p> </subsection> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-12 21:55:06
|
Revision: 2039 http://archive-access.svn.sourceforge.net/archive-access/?rev=2039&view=rev Author: bradtofel Date: 2007-10-12 14:55:10 -0700 (Fri, 12 Oct 2007) Log Message: ----------- HACKHACK: two command line tools, arc-indexer, and url-client make use of the apache commons http client, via heritrix-commons. There is some class loader issue with the commons-logging in this setup, which causes errors about not being able to find a Log4J implementation class within the http client. This is not a problem when there is a log4j dependency. However, adding this dependency causes Spring configuration errors to be squelched within tomcat in the webapp context.. So, I'm adding the log4j dependency for the command line tools, then excluding the log4j jar from the webapp. This seems to be a valid workaround until we understand the log4j/commons-logging issue. Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-core/pom.xml trunk/archive-access/projects/wayback/wayback-webapp/pom.xml Modified: trunk/archive-access/projects/wayback/wayback-core/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-core/pom.xml 2007-10-12 19:41:52 UTC (rev 2038) +++ trunk/archive-access/projects/wayback/wayback-core/pom.xml 2007-10-12 21:55:10 UTC (rev 2039) @@ -74,5 +74,14 @@ <artifactId>spring-beans</artifactId> <version>2.0.5</version> </dependency> + <!-- + Doh... I'm not sure what package is configuring org.apache.commons-logging + to use log4j, but it's breaking some command line tools. + --> + <dependency> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + <version>1.2.14</version> + </dependency> </dependencies> </project> Modified: trunk/archive-access/projects/wayback/wayback-webapp/pom.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-12 19:41:52 UTC (rev 2038) +++ trunk/archive-access/projects/wayback/wayback-webapp/pom.xml 2007-10-12 21:55:10 UTC (rev 2039) @@ -28,7 +28,13 @@ <dependency> <groupId>org.archive.wayback</groupId> <artifactId>wayback-core</artifactId> - <scope>compile</scope> + <scope>compile</scope> + <exclusions> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> + </exclusions> </dependency> <dependency> <groupId>tomcat</groupId> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-12 19:41:48
|
Revision: 2038 http://archive-access.svn.sourceforge.net/archive-access/?rev=2038&view=rev Author: bradtofel Date: 2007-10-12 12:41:52 -0700 (Fri, 12 Oct 2007) Log Message: ----------- BUGFIX: was not sending results type, which caused problems when rendering on receiving side. Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/jsp/XMLResults.jsp Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/jsp/XMLResults.jsp =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/jsp/XMLResults.jsp 2007-10-12 19:03:38 UTC (rev 2037) +++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/jsp/XMLResults.jsp 2007-10-12 19:41:52 UTC (rev 2038) @@ -26,7 +26,12 @@ <<%= key %>><%= value %></<%= key %>> <% } + String type = WaybackConstants.RESULTS_TYPE_CAPTURE; + if(uiResults.isUrlResults()) { + type = WaybackConstants.RESULTS_TYPE_URL; + } %> + <<%= WaybackConstants.RESULTS_TYPE %>><%= type %></<%= WaybackConstants.RESULTS_TYPE %>> </request> <results> <% This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-12 19:03:38
|
Revision: 2037 http://archive-access.svn.sourceforge.net/archive-access/?rev=2037&view=rev Author: bradtofel Date: 2007-10-12 12:03:38 -0700 (Fri, 12 Oct 2007) Log Message: ----------- BUGFIX: System.err.println >> LOGGER.info|fine Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/staticmap/StaticMapExclusionFilter.java trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/staticmap/StaticMapExclusionFilterFactory.java Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/staticmap/StaticMapExclusionFilter.java =================================================================== --- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/staticmap/StaticMapExclusionFilter.java 2007-10-10 20:47:04 UTC (rev 2036) +++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/staticmap/StaticMapExclusionFilter.java 2007-10-12 19:03:38 UTC (rev 2037) @@ -25,6 +25,7 @@ package org.archive.wayback.accesscontrol.staticmap; import java.util.Map; +import java.util.logging.Logger; import org.apache.commons.httpclient.URIException; import org.archive.wayback.WaybackConstants; @@ -39,6 +40,8 @@ * @version $Date$, $Revision$ */ public class StaticMapExclusionFilter implements ObjectFilter<SearchResult> { + private static final Logger LOGGER = Logger.getLogger( + StaticMapExclusionFilter.class.getName()); private String lastChecked = null; private boolean lastCheckedExcluded = false; @@ -58,9 +61,9 @@ if(nextSearch == null) { break; } - System.err.println("EXCLUSION-MAP:Checking " + nextSearch); + LOGGER.fine("EXCLUSION-MAP:Checking " + nextSearch); if(exclusionMap.containsKey(nextSearch)) { - System.err.println("EXCLUSION-MAP: EXCLUDED: \"" + nextSearch + "\" (" + url +")"); + LOGGER.fine("EXCLUSION-MAP: EXCLUDED: \"" + nextSearch + "\" (" + url +")"); return true; } } Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/staticmap/StaticMapExclusionFilterFactory.java =================================================================== --- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/staticmap/StaticMapExclusionFilterFactory.java 2007-10-10 20:47:04 UTC (rev 2036) +++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/staticmap/StaticMapExclusionFilterFactory.java 2007-10-12 19:03:38 UTC (rev 2037) @@ -96,7 +96,7 @@ } String surt = line.startsWith("(") ? line : SURTTokenizer.prefixKey(line); - System.err.println("EXCLUSION-MAP: adding " + surt); + LOGGER.fine("EXCLUSION-MAP: adding " + surt); newMap.put(surt, null); } itr.close(); This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-10 20:47:07
|
Revision: 2036 http://archive-access.svn.sourceforge.net/archive-access/?rev=2036&view=rev Author: bradtofel Date: 2007-10-10 13:47:04 -0700 (Wed, 10 Oct 2007) Log Message: ----------- CLEANUP: removed tons of unused/refactored code that had been commented out. Modified main() to function as command line arc-indexer. Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/ArcIndexer.java Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/ArcIndexer.java =================================================================== --- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/ArcIndexer.java 2007-10-10 20:45:50 UTC (rev 2035) +++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourcestore/ArcIndexer.java 2007-10-10 20:47:04 UTC (rev 2036) @@ -24,37 +24,21 @@ */ package org.archive.wayback.resourcestore; -//import java.io.BufferedOutputStream; import java.io.File; -//import java.io.FileOutputStream; +import java.io.PrintWriter; import java.io.IOException; -//import java.io.PrintWriter; -//import java.text.ParseException; import java.util.Iterator; -//import java.util.logging.Logger; +import java.util.NoSuchElementException; -//import org.apache.commons.httpclient.Header; -//import org.apache.commons.httpclient.URIException; import org.archive.io.ArchiveRecord; import org.archive.io.arc.ARCReader; import org.archive.io.arc.ARCReaderFactory; import org.archive.io.arc.ARCRecord; -//import org.archive.io.arc.ARCRecordMetaData; -//import org.archive.net.UURI; -//import org.archive.net.UURIFactory; -//import org.archive.wayback.WaybackConstants; -//import org.archive.wayback.bdb.BDBRecord; -//import org.archive.wayback.core.CaptureSearchResults; import org.archive.wayback.core.SearchResult; -//import org.archive.wayback.core.SearchResults; -//import org.archive.wayback.resourceindex.bdb.SearchResultToBDBRecordAdapter; -//import org.archive.wayback.resourceindex.cdx.CDXLineToSearchResultAdapter; -//import org.archive.wayback.resourceindex.cdx.SearchResultToCDXLineAdapter; +import org.archive.wayback.resourceindex.cdx.SearchResultToCDXLineAdapter; import org.archive.wayback.util.AdaptedIterator; import org.archive.wayback.util.Adapter; import org.archive.wayback.util.CloseableIterator; -//import org.archive.wayback.util.UrlCanonicalizer; -//import org.archive.wayback.util.flatfile.FlatFile; /** * Transforms an ARC file into SearchResults, or a serialized SearchResults @@ -70,32 +54,6 @@ */ public final static String CDX_HEADER_MAGIC = " CDX N b h m s k r V g"; -// /** -// * Logger for this class -// */ -// private static final Logger LOGGER = Logger.getLogger(ArcIndexer.class -// .getName()); - -// /** -// * Constant indicating entire CDX line -// */ -// protected final static int TYPE_CDX_LINE = 0; -// -// /** -// * Constant indicating entire url + timestamp only -// */ -// protected final static int TYPE_CDX_KEY = 1; -// -// /** -// * Constant indicating trailing data fields from CDX line following url + -// * timestamp -// */ -// protected final static int TYPE_CDX_VALUE = 2; - -// static UrlCanonicalizer canonicalizer = new UrlCanonicalizer(); - -// private final static int DEFAULT_CAPACITY = 120; - /** * @param arc * @return Iterator of SearchResults for input arc File @@ -103,8 +61,8 @@ */ public CloseableIterator<SearchResult> iterator(File arc) throws IOException { - ARCReader arcReader = ARCReaderFactory.get(arc); - arcReader.setParseHttpHeaders(true); +// ARCReader arcReader = ARCReaderFactory.get(arc); +// arcReader.setParseHttpHeaders(true); Adapter<ArchiveRecord,ARCRecord> adapter1 = new ArchiveRecordToARCRecordAdapter(); @@ -112,7 +70,7 @@ Adapter<ARCRecord,SearchResult> adapter2 = new ARCRecordToSearchResultAdapter(); - Iterator<ArchiveRecord> itr1 = arcReader.iterator(); + Iterator<ArchiveRecord> itr1 = new DurableArchiveRecordIterator(arc); CloseableIterator<ARCRecord> itr2 = new AdaptedIterator<ArchiveRecord,ARCRecord>(itr1,adapter1); @@ -134,299 +92,122 @@ } return rec; } - } -// /** -// * Create a ResourceResults representing the records in ARC file at arcPath. -// * -// * @param arc -// * @return ResourceResults in arcPath. -// * @throws IOException -// */ -// public SearchResults indexArc(File arc) throws IOException { -// CaptureSearchResults results = new CaptureSearchResults(); -// ARCReader arcReader = ARCReaderFactory.get(arc); -// try { -// arcReader.setParseHttpHeaders(true); -// // doh. this does not generate quite the columns we need: -// // arcReader.createCDXIndexFile(arcPath); -// Iterator<ArchiveRecord> itr = arcReader.iterator(); -// while (itr.hasNext()) { -// ARCRecord rec = (ARCRecord) itr.next(); -// SearchResult result; -// try { -// result = arcRecordToSearchResult(rec); -// } catch (NullPointerException e) { -// e.printStackTrace(); -// continue; -// } catch (ParseException e) { -// e.printStackTrace(); -// continue; -// } -// if (result != null) { -// results.addSearchResult(result); -// } -// } -// } finally { -// arcReader.close(); -// } -// return results; -// } + private class DurableArchiveRecordIterator + implements Iterator<ArchiveRecord> { -// /** -// * transform an ARCRecord into a SearchResult -// * -// * @param rec -// * @param arc -// * @return SearchResult for this document -// * @throws IOException -// * @throws ParseException -// */ -// public static SearchResult arcRecordToSearchResult(final ARCRecord rec) -// throws IOException, ParseException { -// rec.close(); -// ARCRecordMetaData meta = rec.getMetaData(); -// -// SearchResult result = new SearchResult(); -// String arcName = meta.getArc(); -// int index = arcName.lastIndexOf(File.separator); -// if (index > 0 && (index + 1) < arcName.length()) { -// arcName = arcName.substring(index + 1); -// } -// result.put(WaybackConstants.RESULT_ARC_FILE, arcName); -// result.put(WaybackConstants.RESULT_OFFSET, String.valueOf(meta -// .getOffset())); -// -// // initialize with default HTTP code... -// result.put(WaybackConstants.RESULT_HTTP_CODE, "-"); -// -// result.put(WaybackConstants.RESULT_MD5_DIGEST, rec.getDigestStr()); -// result.put(WaybackConstants.RESULT_MIME_TYPE, meta.getMimetype()); -// result.put(WaybackConstants.RESULT_CAPTURE_DATE, meta.getDate()); -// -// String uriStr = meta.getUrl(); -// if (uriStr.startsWith(ARCRecord.ARC_MAGIC_NUMBER)) { -// // skip filedesc record altogether... -// return null; -// } -// if (uriStr.startsWith(WaybackConstants.DNS_URL_PREFIX)) { -// // skip URL + HTTP header processing for dns records... -// -// String origHost = uriStr.substring(WaybackConstants.DNS_URL_PREFIX -// .length()); -// result.put(WaybackConstants.RESULT_ORIG_HOST, origHost); -// result.put(WaybackConstants.RESULT_REDIRECT_URL, "-"); -// result.put(WaybackConstants.RESULT_URL, uriStr); -// result.put(WaybackConstants.RESULT_URL_KEY, uriStr); -// -// } else { -// -// UURI uri = UURIFactory.getInstance(uriStr); -// result.put(WaybackConstants.RESULT_URL, uriStr); -// -// String uriHost = uri.getHost(); -// if (uriHost == null) { -// LOGGER.info("No host in " + uriStr + " in " + meta.getArc()); -// } else { -// result.put(WaybackConstants.RESULT_ORIG_HOST, uriHost); -// -// String statusCode = (meta.getStatusCode() == null) ? "-" : meta -// .getStatusCode(); -// result.put(WaybackConstants.RESULT_HTTP_CODE, statusCode); -// -// String redirectUrl = "-"; -// Header[] headers = rec.getHttpHeaders(); -// if (headers != null) { -// -// for (int i = 0; i < headers.length; i++) { -// if (headers[i].getName().equals(LOCATION_HTTP_HEADER)) { -// String locationStr = headers[i].getValue(); -// // TODO: "Location" is supposed to be absolute: -// // (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html) -// // (section 14.30) but Content-Location can be -// // relative. -// // is it correct to resolve a relative Location, as -// // we are? -// // it's also possible to have both in the HTTP -// // headers... -// // should we prefer one over the other? -// // right now, we're ignoring "Content-Location" -// try { -// UURI uriRedirect = UURIFactory.getInstance(uri, -// locationStr); -// redirectUrl = uriRedirect.getEscapedURI(); -// -// } catch (URIException e) { -// LOGGER.info("Bad Location: " + locationStr -// + " for " + uriStr + " in " -// + meta.getArc() + " Skipped"); -// } -// break; -// } -// } -// } -// result.put(WaybackConstants.RESULT_REDIRECT_URL, redirectUrl); -// -// String indexUrl = canonicalizer.urlStringToKey(meta.getUrl()); -// result.put(WaybackConstants.RESULT_URL_KEY, indexUrl); -// } -// -// } -// return result; -// } -// -// /** -// * Write out ResourceResults into CDX file at cdxPath -// * -// * @param results -// * @param target -// * @throws IOException -// */ -// public void serializeResults(final SearchResults results, File target) -// throws IOException { -// -// FileOutputStream os = new FileOutputStream(target); -// BufferedOutputStream bos = new BufferedOutputStream(os); -// PrintWriter pw = new PrintWriter(bos); -// try { -// serializeResults(results, pw); -// } finally { -// pw.close(); -// } -// } -// -// /** -// * @param results -// * @param pw -// * @param addHeader -// * @throws IOException -// */ -// public void serializeResults(final SearchResults results, PrintWriter pw, -// final boolean addHeader) -// throws IOException { -// if(addHeader) { -// pw.println(CDX_HEADER_MAGIC); -// } -// Iterator<SearchResult> itrR = results.iterator(); -// Iterator<String> itrS = new AdaptedIterator<SearchResult,String>(itrR, -// new SearchResultToCDXLineAdapter()); -// while (itrS.hasNext()) { -// pw.println(itrS.next()); -// } -// pw.flush(); -// } -// -// -// /** -// * @param results -// * @param pw -// * @throws IOException -// */ -// public void serializeResults(final SearchResults results, PrintWriter pw) -// throws IOException { -// serializeResults(results,pw,true); -// } + private long lastRestart = 0; + private File arc = null; + Iterator<ArchiveRecord> innerItr = null; + ArchiveRecord cachedNext = null; + + public DurableArchiveRecordIterator(File arc) throws IOException { + this.arc = arc; + restart(0); + } -// /** -// * @param rec -// * @return String in "CDX format" for rec argument -// * @throws IOException -// * @throws ParseException -// */ -// public static String arcRecordToCDXLine(ARCRecord rec) -// throws IOException, ParseException { -// return searchResultToString(arcRecordToSearchResult(rec),TYPE_CDX_LINE); -// } - -// /** -// * Transform a SearchResult into a String representation. -// * -// * @param result -// * @param type -// * @return String value of either line, key or value for the SearchResult -// */ -// protected static String searchResultToString(final SearchResult result, -// int type) { -// -// StringBuilder sb = new StringBuilder(DEFAULT_CAPACITY); -// -// if (type == TYPE_CDX_LINE) { -// -// sb.append(result.get(WaybackConstants.RESULT_URL_KEY)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_CAPTURE_DATE)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_ORIG_HOST)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_MIME_TYPE)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_HTTP_CODE)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_MD5_DIGEST)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_REDIRECT_URL)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_OFFSET)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_ARC_FILE)); -// -// } else if (type == TYPE_CDX_KEY) { -// -// sb.append(result.get(WaybackConstants.RESULT_URL_KEY)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_CAPTURE_DATE)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_OFFSET)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_ARC_FILE)); -// -// } else if (type == TYPE_CDX_VALUE) { -// -// sb.append(result.get(WaybackConstants.RESULT_ORIG_HOST)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_MIME_TYPE)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_HTTP_CODE)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_MD5_DIGEST)); -// sb.append(" "); -// sb.append(result.get(WaybackConstants.RESULT_REDIRECT_URL)); -// -// } else { -// throw new IllegalArgumentException("Unknown transformation type"); -// } -// return sb.toString(); -// } + private void restart(long offset) throws IOException { + ARCReader arcReader = ARCReaderFactory.get(arc,offset); + arcReader.setParseHttpHeaders(true); + innerItr = arcReader.iterator(); + } + + private long parseErrorOffset(String message) { + long found = -1; + int idx = message.indexOf("Offset "); + if(idx >= 0) { + int idx2 = message.indexOf(")"); + if(idx2 > 0) { + String part = message.substring(idx + 7,idx2); + System.err.println("Found(" + part +") from (" + message + ")"); + found = Long.parseLong(part) + 100; + } + } + return found; + } + + public boolean hasNext() { + if(cachedNext != null) { + return true; + } + while(true) { + try { + if(!innerItr.hasNext()) { + return false; + } + cachedNext = innerItr.next(); + } catch (RuntimeException e) { + long offset = parseErrorOffset(e.getMessage()); + if(offset > 0) { + if(lastRestart == offset) { + return false; + } + lastRestart = offset; + try { + restart(offset); + } catch (IOException e1) { + throw new RuntimeException(e1); + } + } else { + throw e; + } + } + if(cachedNext != null) { + break; + } + } + return true; + } -// /** -// * @param cdxFile -// * @return Iterator that will return BDBRecords, one for each line in -// * cdxFile argument -// * @throws IOException -// */ -// public Iterator<BDBRecord> getCDXFileBDBRecordIterator(File cdxFile) throws IOException { -// FlatFile ffile = new FlatFile(cdxFile.getAbsolutePath()); -// AdaptedIterator<String,SearchResult> searchResultItr = -// new AdaptedIterator<String,SearchResult>( -// ffile.getSequentialIterator(), -// new CDXLineToSearchResultAdapter()); -// return new AdaptedIterator<SearchResult,BDBRecord>(searchResultItr, -// new SearchResultToBDBRecordAdapter()); -// } + public ArchiveRecord next() { + if(cachedNext == null) { + throw new NoSuchElementException("next() without hasNext()"); + } + ArchiveRecord tmp = cachedNext; + cachedNext = null; + return tmp; + } -// /** -// * @param args -// */ -// public static void main(String[] args) { -// ArcIndexer indexer = new ArcIndexer(); -// File arc = new File(args[0]); -// File cdx = new File(args[1]); -// try { -// SearchResults results = indexer.indexArc(arc); -// indexer.serializeResults(results, cdx); -// } catch (Exception e) { -// e.printStackTrace(); -// } -// } + public void remove() { + throw new UnsupportedOperationException("remove unimplemented"); + } + } + + private static void USAGE() { + System.err.println("USAGE:"); + System.err.println(""); + System.err.println("arc-indexer ARCFILE"); + System.err.println("arc-indexer ARCFILE CDXFILE"); + System.err.println(""); + System.err.println("Create a CDX format index at CDXFILE or to STDOUT"); + System.exit(1); + } + + /** + * @param args + */ + public static void main(String[] args) { + ArcIndexer indexer = new ArcIndexer(); + File arc = new File(args[0]); + PrintWriter pw = null; + try { + if(args.length == 1) { + // dump to STDOUT: + pw = new PrintWriter(System.out); + } else if(args.length == 2) { + pw = new PrintWriter(args[1]); + } else { + USAGE(); + } + Iterator<SearchResult> res = indexer.iterator(arc); + Iterator<String> lines = SearchResultToCDXLineAdapter.adapt(res); + while(lines.hasNext()) { + pw.println(lines.next()); + } + pw.close(); + } catch (Exception e) { + e.printStackTrace(); + } + } } This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
Revision: 2035 http://archive-access.svn.sourceforge.net/archive-access/?rev=2035&view=rev Author: bradtofel Date: 2007-10-10 13:45:50 -0700 (Wed, 10 Oct 2007) Log Message: ----------- FEATURE: added static method to adapt an Iterator<SearchResult> to Iterator<String> Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/cdx/SearchResultToCDXLineAdapter.java Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/cdx/SearchResultToCDXLineAdapter.java =================================================================== --- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/cdx/SearchResultToCDXLineAdapter.java 2007-10-10 20:44:10 UTC (rev 2034) +++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/resourceindex/cdx/SearchResultToCDXLineAdapter.java 2007-10-10 20:45:50 UTC (rev 2035) @@ -24,8 +24,11 @@ */ package org.archive.wayback.resourceindex.cdx; +import java.util.Iterator; + import org.archive.wayback.WaybackConstants; import org.archive.wayback.core.SearchResult; +import org.archive.wayback.util.AdaptedIterator; import org.archive.wayback.util.Adapter; /** @@ -68,4 +71,8 @@ return sb.toString(); } + public static Iterator<String> adapt(Iterator<SearchResult> input) { + return new AdaptedIterator<SearchResult,String>(input, + new SearchResultToCDXLineAdapter()); + } } This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-10 20:44:09
|
Revision: 2034 http://archive-access.svn.sourceforge.net/archive-access/?rev=2034&view=rev Author: bradtofel Date: 2007-10-10 13:44:10 -0700 (Wed, 10 Oct 2007) Log Message: ----------- TWEAK: changed docs for index-client which is now arc-indexer, and has less functionality. Modified Paths: -------------- trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml Modified: trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml =================================================================== --- trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml 2007-10-10 20:43:24 UTC (rev 2033) +++ trunk/archive-access/projects/wayback/dist/src/site/xdoc/administrator_manual.xml 2007-10-10 20:44:10 UTC (rev 2034) @@ -346,7 +346,7 @@ This implementation is good for larger scale installations, bounded mostly by the size of the index you can (first create, and later) store on a single machine. Using the command line tool - <b>index-client</b>, and the standard UNIX <b>sort</b> tool + <b>arc-indexer</b>, and the standard UNIX <b>sort</b> tool (see note below on LC_ALL), you create a sorted flat text file that is searched on each request. Building these sorted files, and updating the index are manual operations presently. @@ -1294,115 +1294,15 @@ </p> </subsection> - <subsection name="index-client"> + <subsection name="arc-indexer"> <p> - This tool has two usages: - <ol> - <li> - <code> - bin/index-client ARC_PATH - </code> - <p> - Generation of a CDX format index data for a - single ARC file named by ARC_PATH. The CDX - format data is sent to STDOUT, and can be saved - to a file, sorted, etc. This is needed to - generate sorted CDX format indexes. - </p> - </li> - <li> - <code> - bin/index-client TMP_DIR INCOMING_URL LOCATION_URL ARC_DIR ARC_URL_PREFIX - </code> - <p> - where: - <ul> - <li> - <i> - TMP_DIR - </i> - Temporary working directory where - ex. - <b> - /tmp/ - </b> - </li> - <li> - <i> - INCOMING_URL - </i> - HTTP path to the RemoteSubmitFilter - which allows remote submission of index - data in CDX format for automatic merging - with a BDB ResourceIndex. - ex. - <b> - http://wayback-webapp.your-archive.org/wayback/index-incoming/ - </b> - </li> - <li> - <i> - LOCATION_URL - </i> - is the absolute URL where the ArcProxy can be - accessed. ex. - <b> - http://wayback-webapp.your-archive.org:8080/locationdb/locationDB - </b> - </li> - <li> - <i> - ARC_DIR - </i> - is the absolute path to the directory on the local - machine which holds ARC files ex. - <b> - /2/arc-collection-1 - </b> - </li> - <li> - <i> - ARC_URL_PREFIX - </i> - is the absolute URL where the directory ARC_DIR can - be accessed. ex. - <b> - http://arc-storage-node-1.your-archive.org/2/arc-collection-1/ - </b> - </li> - </ul> - </p> - <p> - If you chose the Http11 ResourceStore, and are - using the BDB ResourceIndex implementation then - you will need to run this script with these - arguments once for each directory containing ARC - files (on each machine containing ARC files.) - For each ARC file found, this script will: - <ol> - <li> - generate the plain-text index file for - the ARC file - </li> - <li> - push that plain-text file onto the - machine running the Wayback webapp, - where the ResourceIndex database is - stored. The plain-text index files will - arrive in the IndexPipeline directory - structure so they are merged into the - ResourceIndex. - </li> - <li> - notify the ArcProxy LocationDB of the - URL where the ARC file can be accessed, - for later Replay requests which require - access to documents in the ARC file. - </li> - </ol> - </p> - </li> - </ol> + This tool creates a CDX format index for the ARC file at ARC_PATH, + either on STDOUT, or at the path specified by CDX_PATH. The resulting + file can be sorted and merged with other CDX format index files to + generate CDX format ResourceIndex. + <code> + bin/arc-indexer ARC_PATH [CDX_PATH] + </code> </p> </subsection> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-10 20:43:24
|
Revision: 2033 http://archive-access.svn.sourceforge.net/archive-access/?rev=2033&view=rev Author: bradtofel Date: 2007-10-10 13:43:24 -0700 (Wed, 10 Oct 2007) Log Message: ----------- RENAME: this tool now just generates a CDX for a single ARC Added Paths: ----------- trunk/archive-access/projects/wayback/dist/src/scripts/arc-indexer Removed Paths: ------------- trunk/archive-access/projects/wayback/dist/src/scripts/index-client Copied: trunk/archive-access/projects/wayback/dist/src/scripts/arc-indexer (from rev 1983, trunk/archive-access/projects/wayback/dist/src/scripts/index-client) =================================================================== --- trunk/archive-access/projects/wayback/dist/src/scripts/arc-indexer (rev 0) +++ trunk/archive-access/projects/wayback/dist/src/scripts/arc-indexer 2007-10-10 20:43:24 UTC (rev 2033) @@ -0,0 +1,82 @@ +#!/usr/bin/env sh +## +## This script creates a CDX file for all ARC files in a directory +## PUTs those CDX files into a remote pipeline, and informs a remote +## LocationDB of the locations of all the ARC files. +## +## Optional environment variables +## +## JAVA_HOME Point at a JDK install to use. +## +## WAYBACK_HOME Pointer to your wayback install. If not present, we +## make an educated guess based of position relative to this +## script. +## +## JAVA_OPTS Java runtime options. Default setting is '-Xmx256m'. +## + +# Resolve links - $0 may be a softlink +PRG="$0" +while [ -h "$PRG" ]; do + ls=`ls -ld "$PRG"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '.*/.*' > /dev/null; then + PRG="$link" + else + PRG=`dirname "$PRG"`/"$link" + fi +done +PRGDIR=`dirname "$PRG"` + +# Set WAYBACK_HOME. +if [ -z "$WAYBACK_HOME" ] +then + WAYBACK_HOME=`cd "$PRGDIR/.." ; pwd` +fi + +# Find JAVA_HOME. +if [ -z "$JAVA_HOME" ] +then + JAVA=`which java` + if [ -z "$JAVA" ] + then + echo "Cannot find JAVA. Please set JAVA_HOME or your PATH." + exit 1 + fi + JAVA_BINDIR=`dirname $JAVA` + JAVA_HOME=$JAVA_BINDIR/.. +fi + +if [ -z "$JAVACMD" ] +then + # It may be defined in env - including flags!! + JAVACMD=$JAVA_HOME/bin/java +fi + +# Ignore previous classpath. Build one that contains heritrix jar and content +# of the lib directory into the variable CP. +for jar in `ls $WAYBACK_HOME/lib/*.jar $WAYBACK_HOME/*.jar 2> /dev/null` +do + CP=${CP}:${jar} +done + +# cygwin path translation +if expr `uname` : 'CYGWIN*' > /dev/null; then + CP=`cygpath -p -w "$CP"` + WAYBACK_HOME=`cygpath -p -w "$WAYBACK_HOME"` +fi + +# Make sure of java opts. +if [ -z "$JAVA_OPTS" ] +then + JAVA_OPTS=" -Xmx256m" +fi + +# Main ArcIndexer class. +if [ -z "$CLASS_MAIN" ] +then + CLASS_MAIN='org.archive.wayback.resourcestore.ArcIndexer' +fi + +CLASSPATH=${CP} $JAVACMD ${JAVA_OPTS} $CLASS_MAIN $@ + Deleted: trunk/archive-access/projects/wayback/dist/src/scripts/index-client =================================================================== --- trunk/archive-access/projects/wayback/dist/src/scripts/index-client 2007-10-02 03:35:05 UTC (rev 2032) +++ trunk/archive-access/projects/wayback/dist/src/scripts/index-client 2007-10-10 20:43:24 UTC (rev 2033) @@ -1,82 +0,0 @@ -#!/usr/bin/env sh -## -## This script creates a CDX file for all ARC files in a directory -## PUTs those CDX files into a remote pipeline, and informs a remote -## LocationDB of the locations of all the ARC files. -## -## Optional environment variables -## -## JAVA_HOME Point at a JDK install to use. -## -## WAYBACK_HOME Pointer to your wayback install. If not present, we -## make an educated guess based of position relative to this -## script. -## -## JAVA_OPTS Java runtime options. Default setting is '-Xmx256m'. -## - -# Resolve links - $0 may be a softlink -PRG="$0" -while [ -h "$PRG" ]; do - ls=`ls -ld "$PRG"` - link=`expr "$ls" : '.*-> \(.*\)$'` - if expr "$link" : '.*/.*' > /dev/null; then - PRG="$link" - else - PRG=`dirname "$PRG"`/"$link" - fi -done -PRGDIR=`dirname "$PRG"` - -# Set WAYBACK_HOME. -if [ -z "$WAYBACK_HOME" ] -then - WAYBACK_HOME=`cd "$PRGDIR/.." ; pwd` -fi - -# Find JAVA_HOME. -if [ -z "$JAVA_HOME" ] -then - JAVA=`which java` - if [ -z "$JAVA" ] - then - echo "Cannot find JAVA. Please set JAVA_HOME or your PATH." - exit 1 - fi - JAVA_BINDIR=`dirname $JAVA` - JAVA_HOME=$JAVA_BINDIR/.. -fi - -if [ -z "$JAVACMD" ] -then - # It may be defined in env - including flags!! - JAVACMD=$JAVA_HOME/bin/java -fi - -# Ignore previous classpath. Build one that contains heritrix jar and content -# of the lib directory into the variable CP. -for jar in `ls $WAYBACK_HOME/lib/*.jar $WAYBACK_HOME/*.jar 2> /dev/null` -do - CP=${CP}:${jar} -done - -# cygwin path translation -if expr `uname` : 'CYGWIN*' > /dev/null; then - CP=`cygpath -p -w "$CP"` - WAYBACK_HOME=`cygpath -p -w "$WAYBACK_HOME"` -fi - -# Make sure of java opts. -if [ -z "$JAVA_OPTS" ] -then - JAVA_OPTS=" -Xmx256m" -fi - -# Main ArcIndexer class. -if [ -z "$CLASS_MAIN" ] -then - CLASS_MAIN='org.archive.wayback.resourceindex.indexer.IndexClient' -fi - -CLASSPATH=${CP} $JAVACMD ${JAVA_OPTS} $CLASS_MAIN $@ - This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-02 03:35:03
|
Revision: 2032 http://archive-access.svn.sourceforge.net/archive-access/?rev=2032&view=rev Author: bradtofel Date: 2007-10-01 20:35:05 -0700 (Mon, 01 Oct 2007) Log Message: ----------- UPDATE: major overhaul in prep for 1.0, setting default configuration, and leaving commonly used options present but commented out. Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml 2007-10-02 03:33:49 UTC (rev 2031) +++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/wayback.xml 2007-10-02 03:35:05 UTC (rev 2032) @@ -2,6 +2,12 @@ <!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd"> <beans> +<!-- + The following 3 beans are required when using the ArcProxy for providing + HTTP 1.1 remote access to ARC files distributed across multiple computers + or directories. +--> +<!-- <bean id="filelocationdb" class="org.archive.wayback.resourcestore.http.FileLocationDB" init-method="init"> <property name="bdbPath" value="/tmp/wayback/arc-db" /> @@ -15,133 +21,160 @@ <bean name="8080:locationdb" class="org.archive.wayback.resourcestore.http.FileLocationDBServlet"> <property name="locationDB" ref="filelocationdb" /> </bean> +--> - <bean id="bdbsearchresultsource" - class="org.archive.wayback.resourceindex.bdb.BDBIndex" - init-method="init"> - <property name="bdbName" value="DB1" /> - <property name="bdbPath" value="/tmp/wayback/index/" /> - </bean> - <bean id="cdxsearchresultsource" class="org.archive.wayback.resourceindex.cdx.CDXIndex"> - <property name="path" value="/tmp/wayback/cdx-index/index.cdx" /> - </bean> +<!-- + The following 2 beans are required when using exclusions based on live + robots.txt documents. +--> +<!-- + <bean id="livewebcache" class="org.archive.wayback.liveweb.LiveWebCache"> - <bean id="compositecdxresultsource" class="org.archive.wayback.resourceindex.CompositeSearchResultSource"> - <property name="CDXSources"> - <list> - <value>/tmp/wayback/cdx-index/index.cdx.1</value> - <value>/tmp/wayback/cdx-index/index.cdx.2</value> - </list> - </property> - </bean> + <property name="arcCacheDir"> + <bean class="org.archive.wayback.liveweb.ARCCacheDirectory" + init-method="init"> + + <property name="arcDir" value="/tmp/wayback/liveweb/arcs/" /> + <property name="arcPrefix" value="live" /> + </bean> + </property> - <bean id="localbdbindex" class="org.archive.wayback.resourceindex.LocalResourceIndex"> - <property name="source" ref="bdbsearchresultsource" /> - <property name="maxRecords" value="10000" /> - </bean> - <bean id="localcdxindex" class="org.archive.wayback.resourceindex.LocalResourceIndex"> - <property name="source" ref="cdxsearchresultsource" /> - <property name="maxRecords" value="10000" /> - </bean> + <property name="cacher"> + <bean class="org.archive.wayback.liveweb.URLCacher"> + <property name="tmpDir" value="/tmp/wayback/liveweb/tmp/" /> + </bean> + </property> - <bean id="localstore" class="org.archive.wayback.resourcestore.LocalARCResourceStore"> - <property name="arcDir" value="/tmp/wayback/arcs/" /> - </bean> + <property name="index"> + <bean class="org.archive.wayback.liveweb.LiveWebLocalResourceIndex"> - <bean id="standardquery" class="org.archive.wayback.query.Renderer"> - </bean> - <bean id="calendarquery" class="org.archive.wayback.query.Renderer"> - </bean> - <bean id="xmlquery" class="org.archive.wayback.query.Renderer"> - <property name="captureJsp" value="/jsp/XMLResults.jsp" /> - <property name="urlJsp" value="/jsp/XMLResults.jsp" /> - <property name="errorJsp" value="/jsp/XMLError.jsp" /> - </bean> + <property name="source"> + <bean class="org.archive.wayback.resourceindex.bdb.BDBIndex" + init-method="init"> + <property name="bdbName" value="DB1" /> + <property name="bdbPath" value="/tmp/wayback/liveweb/db/" /> + </bean> + </property> + </bean> + </property> + </bean> + + <bean id="excluder-factory-robot" class="org.archive.wayback.accesscontrol.robotstxt.RobotExclusionFilterFactory"> + <property name="maxCacheMS" value="86400000" /> + <property name="userAgent" value="ia_archiver" /> + <property name="webCache" ref="livewebcache" /> + </bean> +--> - <bean id="archivalurlreplay" class="org.archive.wayback.archivalurl.ArchivalUrlReplayDispatcher"> - <property name="jsInserts"> - <list> - <value>http://localhost:8080/wm.js</value> - </list> - </property> - </bean> - <bean id="archivalurluriconverter" - class="org.archive.wayback.archivalurl.ArchivalUrlResultURIConverter"> - <property name="replayURIPrefix" value="http://localhost:8080/wayback/" /> - </bean> - <bean id="archivalurlparser" class="org.archive.wayback.archivalurl.ArchivalUrlRequestParser" init-method="init"> - <property name="maxRecords" value="1000" /> - </bean> + <bean id="localbdbcollection" class="org.archive.wayback.webapp.WaybackCollection"> + <property name="resourceStore"> + <bean class="org.archive.wayback.resourcestore.LocalARCResourceStore" + init-method="init"> + <property name="arcDir" value="/tmp/wayback/arcs/" /> + <property name="queuedDir" value="/tmp/wayback/arc-indexer/queued" /> + <property name="workDir" value="/tmp/wayback/arc-indexer/work" /> + <property name="runInterval" value="10000" /> + <property name="indexClient"> + <bean class="org.archive.wayback.resourceindex.indexer.IndexClient"> + <property name="tmpDir" value="/tmp/wayback/arc-indexer/tmp" /> + <property name="target" value="/tmp/wayback/index-data/incoming" /> + </bean> + </property> + </bean> + </property> - <bean id="proxyreplay" class="org.archive.wayback.proxy.ProxyReplayDispatcher"> - </bean> - <bean id="proxyuriconverter" class="org.archive.wayback.proxy.RedirectResultURIConverter"> - <property name="redirectURI" value="http://localhost:8090/jsp/QueryUI/Redirect.jsp" /> - </bean> - <bean id="proxyparser" class="org.archive.wayback.proxy.ProxyRequestParser" init-method="init"> - <property name="maxRecords" value="1000" /> - </bean> + <property name="resourceIndex"> + <bean class="org.archive.wayback.resourceindex.LocalResourceIndex"> + <property name="source"> + <bean class="org.archive.wayback.resourceindex.bdb.BDBIndex" + init-method="init"> + <property name="bdbName" value="DB1" /> + <property name="bdbPath" value="/tmp/wayback/index/" /> + <property name="updater"> + <bean class="org.archive.wayback.resourceindex.bdb.BDBIndexUpdater"> + <property name="incoming" value="/tmp/wayback/index-data/incoming/" /> + <property name="failed" value="/tmp/wayback/index-data/failed/" /> + <property name="merged" value="/tmp/wayback/index-data/merged/" /> + <property name="runInterval" value="10000" /> + </bean> + </property> + </bean> + </property> + <property name="maxRecords" value="10000" /> + </bean> + </property> + </bean> +<!-- + The following WaybackCollection bean template is required when using a + manually built local CDX index. +--> +<!-- + <bean id="localcdxcollection" class="org.archive.wayback.webapp.WaybackCollection"> - <bean id="archivalcontext" abstract="true" - class="org.archive.wayback.webapp.WaybackContext"> - <property name="index" ref="localbdbindex" /> - <property name="store" ref="localstore" /> - <property name="query" ref="standardquery" /> - <property name="replay" ref="archivalurlreplay" /> - <property name="parser" ref="archivalurlparser" /> - <property name="uriConverter" ref="archivalurluriconverter" /> - </bean> + <property name="resourceStore"> + <bean class="org.archive.wayback.resourcestore.LocalARCResourceStore" + init-method="init"> + <property name="arcDir" value="/tmp/wayback/arcs/" /> + </bean> + </property> - <bean name="8080:wayback" class="org.archive.wayback.webapp.WaybackContext"> - <property name="configs"> - <props> - <prop key="key1">key1-value</prop> - <prop key="key2">key2-value</prop> - </props> - </property> + <property name="resourceIndex"> + <bean class="org.archive.wayback.resourceindex.LocalResourceIndex"> + <property name="source"> + <bean id="cdxsearchresultsource" class="org.archive.wayback.resourceindex.cdx.CDXIndex"> + <property name="path" value="/tmp/wayback/cdx-index/index.cdx" /> + </bean> + </property> + <property name="maxRecords" value="10000" /> + </bean> + </property> + </bean> +--> + + <!-- - <property name="index" ref="localcdxindex" /> + The following WaybackCollection bean template is required when using a + remote ResourceIndex and ResourceStore implementation. This will also + required setting up an arcproxy and locationdb on the host specified by + the resourceStore:urlPrefix configuration, and an addition AccessPoint + on the host specified by the resourceIndex:searchUrlBase configuration. --> - <property name="index"> - <bean class="org.archive.wayback.resourceindex.LocalResourceIndex"> - <property name="source"> - <bean class="org.archive.wayback.resourceindex.bdb.BDBIndex" - init-method="init"> - <property name="bdbName" value="DB1" /> - <property name="bdbPath" value="/tmp/wayback/index/" /> - <property name="updater"> - <bean class="org.archive.wayback.resourceindex.bdb.BDBIndexUpdater"> - <property name="incoming" value="/tmp/wayback/index-data/incoming/" /> - <property name="failed" value="/tmp/wayback/index-data/failed/" /> - <property name="merged" value="/tmp/wayback/index-data/merged/" /> - <property name="runInterval" value="10000" /> - </bean> - </property> - </bean> - </property> - <property name="maxRecords" value="10000" /> - </bean> - </property> - <property name="store"> - <bean class="org.archive.wayback.resourcestore.LocalARCResourceStore" - init-method="init"> - <property name="arcDir" value="/tmp/wayback/arcs/" /> - <property name="queuedDir" value="/tmp/wayback/arc-indexer/queued" /> - <property name="workDir" value="/tmp/wayback/arc-indexer/work" /> - <property name="runInterval" value="10000" /> - <property name="indexClient"> - <bean class="org.archive.wayback.resourceindex.indexer.IndexClient"> - <property name="tmpDir" value="/tmp/wayback/arc-indexer/tmp" /> - <property name="target" value="/tmp/wayback/index-data/incoming" /> - </bean> - </property> - </bean> - </property> - +<!-- + <bean id="remotecollection" class="org.archive.wayback.webapp.WaybackCollection"> + + <property name="resourceStore"> + <bean class="org.archive.wayback.resourcestore.HttpARCResourceStore"> + <property name="urlPrefix" value="http://localhost:8080/arcproxy/" /> + </bean> + </property> + + <property name="resourceIndex"> + <bean class="org.archive.wayback.resourceindex.RemoteResourceIndex" + init-method="init"> + <property name="searchUrlBase" value="http://indexhost:8080/index/xmlquery" /> + </bean> + </property> + </bean> +--> + +<!-- + This is the only AccessPoint defined by default within this wayback.xml + Spring configuration file, providing an ArchivalURL Replay UI to the + "localbdbcollection" by providing ArchivalURL-specific implementations + of the replay, parser, and uriConverter. + + This AccessPoint currently will provide access only from the machine + running Tomcat. To provide external access, replace "localhost" with your + fully qualified hostname of the computer running Tomcat. +--> + <bean name="8080:wayback" class="org.archive.wayback.webapp.AccessPoint"> + + <property name="collection" ref="localbdbcollection" /> + <property name="query"> <bean class="org.archive.wayback.query.Renderer"> <property name="captureJsp" value="/jsp/HTMLResults.jsp" /> @@ -178,27 +211,78 @@ </property> </bean> - - <bean name="8081" parent="8080:wayback"> - <property name="useServerName" value="true" /> - <property name="replay"> - <bean class="org.archive.wayback.domainprefix.DomainPrefixReplayDispatcher" /> - </property> - - <property name="parser"> - <bean class="org.archive.wayback.domainprefix.DomainPrefixCompositeRequestParser" - init-method="init"> - <property name="hostPort" value="localhost.archive.org:8081" /> - <property name="maxRecords" value="1000" /> - <property name="earliestTimestamp" value="1996" /> +<!-- + The following AccessPoint inherits all configuration from the 8080:wayback + AccessPoint, but only allows access from the specified IP network. +--> +<!-- + <bean name="8080:netsecure" parent="8080:wayback"> + <property name="authentication"> + <bean class="org.archive.wayback.authenticationcontrol.IPMatchesBooleanOperator"> + <property name="allowedRanges"> + <list> + <value>192.168.1.16/24</value> + </list> + </property> </bean> </property> + <property name="uriConverter"> + <bean class="org.archive.wayback.archivalurl.ArchivalUrlResultURIConverter"> + <property name="replayURIPrefix" value="http://192.168.1.16:8080/netsecure/" /> + </bean> + </property> + </bean> +--> - <property name="uriConverter"> - <bean class="org.archive.wayback.domainprefix.DomainPrefixResultURIConverter"> - <property name="hostPort" value="localhost.archive.org:8081" /> - </bean> +<!-- + The following AccessPoint inherits all configuration from the 8080:wayback + AccessPoint, but checks live web robots.txt documents to determine if + archived content should be accessible. + + Note: using this AccessPoint requires enabling the "livewebcache" and + "excluder-factory-robot" beans declared at the top of this file. +--> +<!-- + <bean name="8080:robots" parent="8080:wayback"> + <property name="exclusionFactory" ref="excluder-factory-robot" /> + <property name="uriConverter"> + <bean class="org.archive.wayback.archivalurl.ArchivalUrlResultURIConverter"> + <property name="replayURIPrefix" value="http://localhost:8080/robots/" /> + </bean> + </property> + </bean> +--> + + +<!-- + The following AccessPoint inherits all configuration from the 8080:wayback + AccessPoint, but provides a Proxy Replay UI to the same collection. These + two access points can be used simultaneously on the same Tomcat + installation. + + Note: using this AccessPoint requires adding a "Connector" on port 8090 + in your Tomcat's server.xml file. + --> +<!-- + <bean name="8090" parent="8080:wayback"> + <property name="replay"> + <bean class="org.archive.wayback.proxy.ProxyReplayDispatcher" /> </property> - </bean> - + <property name="uriconverter"> + <bean class="org.archive.wayback.proxy.RedirectResultURIConverter"> + <property name="redirectURI" value="http://foo.archive.org:8090/jsp/Redirect.jsp" /> + </bean> + </property> + <property name="parser"> + <bean class="org.archive.wayback.proxy.ProxyRequestParser" init-method="init"> + <property name="localhostNames"> + <list> + <value>foo.archive.org</value> + </list> + </property> + <property name="maxRecords" value="1000" /> + </bean> + </property> + </bean> +--> </beans> \ No newline at end of file This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |
From: <bra...@us...> - 2007-10-02 03:33:45
|
Revision: 2031 http://archive-access.svn.sourceforge.net/archive-access/?rev=2031&view=rev Author: bradtofel Date: 2007-10-01 20:33:49 -0700 (Mon, 01 Oct 2007) Log Message: ----------- TWEAK: added basic HTTP AUTH example configuration. Modified Paths: -------------- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/web.xml Modified: trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/web.xml =================================================================== --- trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/web.xml 2007-10-02 03:33:00 UTC (rev 2030) +++ trunk/archive-access/projects/wayback/wayback-webapp/src/main/webapp/WEB-INF/web.xml 2007-10-02 03:33:49 UTC (rev 2031) @@ -3,6 +3,21 @@ "http://java.sun.com/dtd/web-app_2_3.dtd"> <web-app> + <security-constraint> + <web-resource-collection> + <web-resource-name>Secured-Wayback</web-resource-name> + <url-pattern>/usersecure/*</url-pattern> + </web-resource-collection> + <auth-constraint> + <role-name>wayback</role-name> + </auth-constraint> + </security-constraint> + + <login-config> + <auth-method>BASIC</auth-method> + <realm-name>Secured-Wayback</realm-name> + </login-config> + <!-- General Installation information --> <context-param> This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |