Delaed retrieval of robots.txt

Brought to you by: angusgb, grdetil, lha, nealr, scherpbier

#5 Delaed retrieval of robots.txt

Status: open

Owner: Geoff Hutchison

Labels: devel (5)

Priority: 5

Updated: 2002-01-04

Created: 2001-07-28

Creator: Gaspar Chilingarov

Private: No

This patch allows to delay loading of "robots.txt" file from the server until first connection really made to the server - the main purpose of this to decrease startup time for htdig - in case of multiple domains, which are located on slow links or in case of some servers are down :)

this gives great startup speed boost for me, YMMV.

please apply patch in htdig directory ...

Discussion

Geoff Hutchison - 2002-01-04

Logged In: YES
user_id=21420

OK, this sounds useful, but there's no patch attached...

-Geoff

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Geoff Hutchison - 2002-01-04

assigned_to: nobody --> ghutchis
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nobody/Anonymous - 2003-03-29

Logged In: NO

what

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Delaed retrieval of robots.txt

Group

Searches

Help

#5 Delaed retrieval of robots.txt

Discussion