The "Hosts" section of my awstats report contains all the spiders and crawlers. Google, MSN, Yahoo, etc are all showing up there with hostnames like b3091237.crawl.yahoo.net, crawl-66-249-65-232.googlebot.com, and msnbot-207-46-193-80.search.msn.com. I would have expected these to show up in the Robots pane. How does awstats determine whether a request is coming from a robot or not?
Log in to post a comment.