[htdig-updates] htdig update htdig_4_0

[Anthony A. <aa...@us...>]

htdig/src/spider Spider.cc
Update of /cvsroot/htdig/htdig/src/spider
In directory sc8-pr-cvs17:/tmp/cvs-serv2065/src/spider

Modified Files:
      Tag: htdig_4_0
	Spider.cc 
Log Message:
Final checkin for 4.0.0 version (and associated tag).
First "gold" version of htdig 4.0 rework.

FEATURES (some may be in 4_0_0_beta tag):
Major multi-year effort to refactor and modernize htdig.
Some highlights:
    UTF-8 support
    BDB word index replaced with CLucene for searching
    CJK language support
    HTTP protocol changes
    Major code changes
    htsearch removed, replaced with php and c++ API
    libhtdig updated to be main interface to package
    index maintenance utilities removed
See htdig40_refactor.pdf for more details (forthcoming)