#89 RFE - emit 'directory skipped' msg

docs (14)
R P Herrold

RFE - emit 'directory skipped due to robots.txt'
message in the -v messages, when a 'robots.txt' rule
matches, and a directory is accordingly skopped.

I spent a good 6 hours writing and re-writing custom
produced linkfarms, which are auto-generated,
thinking it was a size, or permissions issue -- when
it was just hitting a remnant robots.txt file.


Also helpful would be a reminder in htdig.conf ,
physically close to the start_url: stanza, reminding
of this.


  • Gilles Detillieux

    Logged In: YES

    You need more verbosity in the output.
    See http://www.htdig.org/FAQ.html#5.27

    You could have saved yourself many hours
    by just reading the docs. We can't stuff
    every conceivable configuration tip in
    htdig.conf just for the sake of people who
    don't look at the documentation. This is what
    the FAQ is for!

  • Gilles Detillieux

    • assigned_to: nobody --> grdetil
    • status: open --> closed-works-for-me

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks