Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

#5 Delaed retrieval of robots.txt

open
devel (5)
5
2002-01-04
2001-07-28
No

This patch allows to delay loading of "robots.txt" file from the server until first connection really made to the server - the main purpose of this to decrease startup time for htdig - in case of multiple domains, which are located on slow links or in case of some servers are down :)

this gives great startup speed boost for me, YMMV.

please apply patch in htdig directory ...

Discussion

  • Logged In: YES
    user_id=21420

    OK, this sounds useful, but there's no patch attached...

    -Geoff

     
    • assigned_to: nobody --> ghutchis
     
  • Logged In: NO

    what