This patch allows to delay loading of "robots.txt" file from the server until first connection really made to the server - the main purpose of this to decrease startup time for htdig - in case of multiple domains, which are located on slow links or in case of some servers are down :)
this gives great startup speed boost for me, YMMV.
please apply patch in htdig directory ...
Logged In: YES
user_id=21420
OK, this sounds useful, but there's no patch attached...
-Geoff
Logged In: NO
what