#89 RFE - emit 'directory skipped' msg

feature-requests
closed-works-for-me
docs (14)
5
2001-11-28
2001-11-25
R P Herrold
No

RFE - emit 'directory skipped due to robots.txt'
message in the -v messages, when a 'robots.txt' rule
matches, and a directory is accordingly skopped.

I spent a good 6 hours writing and re-writing custom
produced linkfarms, which are auto-generated,
thinking it was a size, or permissions issue -- when
it was just hitting a remnant robots.txt file.

----------------------------------

Also helpful would be a reminder in htdig.conf ,
physically close to the start_url: stanza, reminding
of this.

Discussion

  • Logged In: YES
    user_id=149687

    You need more verbosity in the output.
    See http://www.htdig.org/FAQ.html#5.27

    You could have saved yourself many hours
    by just reading the docs. We can't stuff
    every conceivable configuration tip in
    htdig.conf just for the sake of people who
    don't look at the documentation. This is what
    the FAQ is for!

     
    • assigned_to: nobody --> grdetil
    • status: open --> closed-works-for-me