[Archive-access-cvs] SF.net SVN: archive-access:[3247] trunk/archive-access/projects/wayback/ wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/ RobotExclusionFilter.java

[<bra...@us...>]

Revision: 3247
          http://archive-access.svn.sourceforge.net/archive-access/?rev=3247&view=rev
Author:   bradtofel
Date:     2010-09-03 23:19:28 +0000 (Fri, 03 Sep 2010)

Log Message:
-----------
FEATURE: explicitly allowing download of /robots.txt paths, without consulting the robots.txt file.

Modified Paths:
--------------
    trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java

Modified: trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java
===================================================================
--- trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java	2010-09-03 22:32:51 UTC (rev 3246)
+++ trunk/archive-access/projects/wayback/wayback-core/src/main/java/org/archive/wayback/accesscontrol/robotstxt/RobotExclusionFilter.java	2010-09-03 23:19:28 UTC (rev 3247)
@@ -236,7 +236,9 @@
 			URL url;
 			try {
 				url = new URL(ArchiveUtils.addImpliedHttpIfNecessary(resultURL));
-				if(!rules.blocksPathForUA(url.getPath(), userAgent)) {
+				String path = url.getPath();
+				if(path.equals(ROBOT_SUFFIX) || 
+						!rules.blocksPathForUA(path, userAgent)) {
 					if(!notifiedPassed) {
 						if(filterGroup != null) {
 							filterGroup.setPassedRobots();


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.