archive-access-cvs

[Archive-access-cvs] SF.net SVN: archive-access: [1752] trunk/archive-access/projects/wayback/ wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/ RobotExclusionFilter.java

[<bra...@us...>]

Revision: 1752
          http://archive-access.svn.sourceforge.net/archive-access/?rev=1752&view=rev
Author:   bradtofel
Date:     2007-07-09 15:02:05 -0700 (Mon, 09 Jul 2007)

Log Message:
-----------
Comment change

Modified Paths:
--------------
    trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java

Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java	2007-06-04 22:55:13 UTC (rev 1751)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java	2007-07-09 22:02:05 UTC (rev 1752)
@@ -49,6 +49,9 @@
  * This class caches parsed RobotRules that are retrieved, so using the same 
  * instance to filter multiple SearchResults from the same host will be more
  * efficient.
+ * 
+ * Instances are expected to be transient for each request: The internally
+ * cached StringBuilder is not thread safe.
  *
  * @author brad
  * @version $Date$, $Revision$


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Archive-access-cvs] SF.net SVN: archive-access: [1753] trunk/archive-access/projects/wayback/ wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/ RobotExclusionFilter.java

[<bra...@us...>]

Revision: 1753
          http://archive-access.svn.sourceforge.net/archive-access/?rev=1753&view=rev
Author:   bradtofel
Date:     2007-07-09 15:03:58 -0700 (Mon, 09 Jul 2007)

Log Message:
-----------
TWEAK: added String type to List templates

Modified Paths:
--------------
    trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java

Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java	2007-07-09 22:02:05 UTC (rev 1752)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java	2007-07-09 22:03:58 UTC (rev 1753)
@@ -106,7 +106,7 @@
 	 * Otherwise:
 	 *  [originalURL,www.originalURL]
 	 */
-	protected List searchResultToRobotUrlStrings(String resultHost) {
+	protected List<String> searchResultToRobotUrlStrings(String resultHost) {
 		ArrayList<String> list = new ArrayList<String>();
 		list.add(hostToRobotUrlString(resultHost));
 		
@@ -131,8 +131,8 @@
 		RobotRules rules = null;
 		RobotRules tmpRules = null;
 		String host = result.get(WaybackConstants.RESULT_ORIG_HOST);
-		List urlStrings = searchResultToRobotUrlStrings(host);
-		Iterator itr = urlStrings.iterator();
+		List<String> urlStrings = searchResultToRobotUrlStrings(host);
+		Iterator<String> itr = urlStrings.iterator();
 		String firstUrlString = null;
 
 		while(rules == null && itr.hasNext()) {


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Archive-access-cvs] SF.net SVN: archive-access: [2099] trunk/archive-access/projects/wayback/ wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/ RobotExclusionFilter.java

[<bra...@us...>]

Revision: 2099
          http://archive-access.svn.sourceforge.net/archive-access/?rev=2099&view=rev
Author:   bradtofel
Date:     2007-11-29 13:06:55 -0800 (Thu, 29 Nov 2007)

Log Message:
-----------
INTERFACE: no longer call parseHeaders on Resource

Modified Paths:
--------------
    trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java

Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java	2007-11-29 21:05:22 UTC (rev 2098)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java	2007-11-29 21:06:55 UTC (rev 2099)
@@ -148,7 +148,6 @@
 					tmpRules = new RobotRules();
 					Resource resource = webCache.getCachedResource(new URL(urlString),
 							maxCacheMS,true);
-					resource.parseHeaders();
 					tmpRules.parse(resource);
 					rulesCache.put(firstUrlString,tmpRules);
 					rules = tmpRules;


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.