[htdig-updates] htdig update htdig_4_0

[Anthony A. <aa...@us...>]

htdig/src/spider Spider.cc Spider.h
Update of /cvsroot/htdig/htdig/src/spider
In directory sc8-pr-cvs5.sourceforge.net:/tmp/cvs-serv9246

Modified Files:
      Tag: htdig_4_0
	Spider.cc Spider.h 
Log Message:
Added Sitemap parsing support, *max_docs config verbs and checks, and also updated the addURL calls after parsing to support faceting. Also, updated some debugging statements