[Archive-access-cvs] archive-access/projects/nutch/conf nutch-site.xml.nutchwax,1.1,1.2

[Michael S. <sta...@us...>]

Update of /cvsroot/archive-access/archive-access/projects/nutch/conf
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv21415/conf

Modified Files:
	nutch-site.xml.nutchwax 
Log Message:
Implement '[ 1309781 ] Add in skipping certain types if > size' for Dan.
* bin/nutch
    Add new col-dedup command.
* conf/nutch-site.xml.nutchwax 
    Remove dedup collection parameter.  Not used.
* src/java/org/archive/access/nutch/CollectionDeleteDuplicates.java 
    A copy of nutch DeleteDuplicates that adds in hash of collection to url
    and content md5.  Have to make copy rather than subclass because the
    original is not subclassable -- its all private in awkward places.


Index: nutch-site.xml.nutchwax
===================================================================
RCS file: /cvsroot/archive-access/archive-access/projects/nutch/conf/nutch-site.xml.nutchwax,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** nutch-site.xml.nutchwax	21 Oct 2005 00:42:14 -0000	1.1
--- nutch-site.xml.nutchwax	21 Oct 2005 04:16:12 -0000	1.2
***************
*** 142,152 ****
      value is -1 which says don't skip text/html docs.</description>
  </property>
- <property>
-     <name>archive.dedup.count.collection</name>
-     <value>false</value>
-     <description>If true, when deduping, compare collection names
-     as well as URL and content-md5 deduping.
-     </description>
- </property>
- 
  </nutch-conf>
--- 142,144 ----