For indexing an local server it would be handy to have
an option for the case that htdig should ignore the
noindex and nofollow metatags and the robots.txt
I think that the way via User-agent in the robots.txt
is not exact enough in same cases (e.g. an internal
used index where all files should be indexed and an
external index where some files shouldn't be indexed).
Log in to post a comment.