#217 If a robots.txt file exists, indexing terminates

closed-duplicate
htdig (103)
5
2004-04-25
2004-03-24
Anonymous
No

Email : thomas@stthomasanglican.org.
Package : htdig-3.2.0.b5, self compiled.
OS : FreeBSD 5.1-Release-p16, i386

When I run rundig -vvv the following snipped info comes
up.

Parsing robots.txt file using myname = htdig
Pattern:
pushed
Rejected: forbidden by server robots.txt!

My robots.txt simply has

User-agent: *
Allow: /

However, if I delete the robots.txt, the indexing runs, no
problem.

Thomas

Discussion

  • Lachlan Andrew
    Lachlan Andrew
    2004-04-10

    • assigned_to: nobody --> lha
    • status: open --> open-duplicate
     
  • Lachlan Andrew
    Lachlan Andrew
    2004-04-10

    Logged In: YES
    user_id=663373

    See Bug #897309.

    I think I fixed this on 14 December, 2003. Please download
    a current snapshot and let me know whether or not the
    problem is still there.

    Regards,
    Lachlan

     
  • Logged In: YES
    user_id=1005656

    Took the snapshot dated 11-apr-2004, and it worked. Thank
    you very much.

    Thomas

     
  • Lachlan Andrew
    Lachlan Andrew
    2004-04-25

    • status: open-duplicate --> closed-duplicate