#862 Please add these spiders to robots.pm

open
nobody
5
2012-10-11
2011-09-26
Nils
No

Hi,

some spiders are not (yet?) recognized by Awstats 7.0 . After adding them to robots.pm, my statistics are clean. Here are the changes I made :

in @RobotsSearchIDOrder_list2 :
'cplanet',
'WikioFeedBot',
'SimplePie',
'Summify',
'Moreoverbot',
'TopBlogsInfo',

in %RobotsHashIDLib :
'cplanet','CPlanet RSS agregator',
'WikioFeedBot','WikioFeedBot',
'SimplePie','SimplePie',
'Summify','Summify',
'Moreoverbot','Moreover bot',
'TopBlogsInfo','TopBlogsInfo (identifies itself as Mozilla compatible',

I added some description in the URL to remind me where they come from.
Here are some Apache logs in case this is needed (I removed the first field, which is the IP adress of the client) :

'cplanet' : - - [26/Sep/2011:08:30:54 +0200] "GET /feed/atom HTTP/1.1" 200 108824 "-" "cplanet/0.6"
'WikioFeedBot' : - - [26/Sep/2011:06:00:53 +0200] "GET /feed/atom HTTP/1.0" 304 - "-" "WikioFeedBot 1.0 (http://www.wikio.com)"
'SimplePie' : - - [26/Sep/2011:08:21:43 +0200] "GET /feed/rss2 HTTP/1.1" 304 - "http://blog.anotherhomepage.org/feed/rss2" "SimplePie/1.2 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20090627192103"
'Summify' : - - [26/Sep/2011:06:40:03 +0200] "GET /feed/atom HTTP/1.1" 200 29635 "-" "Summify (Summify/1.0.1; +http://summify.com)"
'Moreoverbot' : - - [26/Sep/2011:02:28:51 +0200] "GET /feed/atom HTTP/1.0" 200 29635 "-" "Moreoverbot/5.1 (+http://w.moreover.com; webmaster@moreover.com) Mozilla/5.0"
'TopBlogsInfo' : - - [23/Sep/2011:17:24:03 +0200] "GET /post/2011/03/14/configuration-openssh HTTP/1.1" 200 6730 "-" "Mozilla/5.0 (compatible; TopBlogsInfo/2.0; +topblogsinfo@gmail.com)"

I can provide a diff file if needed.

Regards,

Nils

Discussion