#805 Robots and crawlers showing up in Hosts instead of Robots

6.9
open
nobody
5
2012-10-11
2010-05-25
David Kramer
No

The "Hosts" section of my awstats report contains all the spiders and crawlers. Google, MSN, Yahoo, etc are all showing up there with hostnames like b3091237.crawl.yahoo.net, crawl-66-249-65-232.googlebot.com, and msnbot-207-46-193-80.search.msn.com. I would have expected these to show up in the Robots pane. How does awstats determine whether a request is coming from a robot or not?

Discussion

  • Lengen1971
    Lengen1971
    2012-07-31

    This behaviour come up with ActivePerl 5.14. Downgrade to 5.10 gets the panes to normal, viewing robots, spiders, searchphrases, searchengines under their appopriate pane.