Can't Find Rogue Bot Listed In Awstats

Developers
Chris
2010-10-29
2012-10-11
  • Chris
    Chris
    2010-10-29

    The robot visiting my site the most is "bot" ("bot" with an asterik right
    after the "t"). I looked in the raw access log and found no such robot bot
    .
    Awstats shows the last time bot visited my site and when I look for bots with
    the same time in the raw access log, I find a number of different bots
    including nwtn.ct.charter.com, spider49.yandex.ru, YandexBot/3.0;
    +http://yandex.com/bots), and
    msnbot-207-46-199-44.search.msn.com. What is bot
    ? I assume it's a rogue bot.
    If it is, what bot name do I use to stop it? I tried bot, ^bot$, bot/* and
    they don't work.

    Thanks,

    Chris

     
  • Jean-Luc
    Jean-Luc
    2010-10-29

    Hi,

    "bot*" means any user agent that contains "bot" and that is not in the AWStats
    robots list. This is a way to pick up rare or new robots. Some of them are
    nice, others are rogue.

    Jean-Luc, InternetOfficer AWStats
    Support

     
  • Chris
    Chris
    2010-10-29

    Jean,
    In my htaccess I have
    RewriteCond %{HTTP_USER_AGENT} spider
    RewriteCond %{HTTP_USER_AGENT} crawl

    Without ^ and ? it seems "bot" means all robots containig "bot". I worry that
    there are good and bad bots that contain "spider" and "crawl" in their names.
    Since awstats list counts on "spider" and "crawl" should my code be

    RewriteCond %{HTTP_USER_AGENT} ^spider?
    RewriteCond %{HTTP_USER_AGENT} ^crawl$

    and that's assuming "spider" and "crawl" are bad bots, which may not be true.

    Thanks for the help,

    Chris

     
  • Chris
    Chris
    2010-10-29

    That "?" is meant to be a $.