Operating Systems and Robots Not Showing

Kim
2011-11-15
2012-10-11
  • Kim

    Kim - 2011-11-15

    I have setup awstats. Operating Systems are all unknown and the only
    robot/spider I'm picking up is Unknown robot (identified by empty user agent
    string)

    System is IIS 7.5
    Log file is:

    Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username

    c-ip cs-version cs(User-Agent) cs(Referer) sc-status sc-substatus sc-
    win32-status sc-bytes time-taken
    2011-11-15 14:37:32 192.168.2.187 POST /mkflash.php - 80 - 71.236.115.36
    HTTP/1.1 Shockwave+Flash - 200 0 0 215 5460
    2011-11-15 14:37:35 192.168.2.187 GET
    /media/facedata/intellitar_1289831827.59894ce145939239f5.21855899.swf - 80 -
    71.236.115.36 HTTP/1.1 Mozilla/5.0+(Windows+NT+5.1)+AppleWebKit/535.2+(KHTML,+
    like+Gecko)+Chrome/15.0.874.106+Safari/535.2 - 200 0 0 1539776 2137
    2011-11-15 14:37:36 192.168.2.187 GET / - 443 - 192.168.2.1 HTTP/1.1
    Mozilla/5.0+(Windows+NT+5.1;+rv:8.0)+Gecko/20100101+Firefox/8.0 - 200 0 0
    10373 109
    2011-11-15 14:37:36 192.168.2.187 GET /css/all.css - 443 - 192.168.2.1
    HTTP/1.1 Mozilla/5.0+(Windows+NT+5.1;+rv:8.0)+Gecko/20100101+Firefox/8.0
    https://www.virtualeternity.com/ 304 0 0
    229 0
    2011-11-15 14:37:36 192.168.2.187 GET /css/form.css - 443 - 192.168.2.1
    HTTP/1.1 Mozilla/5.0+(Windows+NT+5.1;+rv:8.0)+Gecko/20100101+Firefox/8.0
    https://www.virtualeternity.com/ 304 0 0
    229 93
    2011-11-15 14:37:36 192.168.2.187 GET /js/jquery-1.4.2.min.js - 443 -
    192.168.2.1 HTTP/1.1
    Mozilla/5.0+(Windows+NT+5.1;+rv:8.0)+Gecko/20100101+Firefox/8.0
    https://www.virtualeternity.com/ 304 0 0
    229 62
    2011-11-15 14:37:36 192.168.2.187 GET /js/jquery.main.js - 443 - 192.168.2.1
    HTTP/1.1 Mozilla/5.0+(Windows+NT+5.1;+rv:8.0)+Gecko/20100101+Firefox/8.0
    https://www.virtualeternity.com/ 304 0 0
    229 124
    2011-11-15 14:37:38 192.168.2.187 GET /js/gyrolistic/style.css - 443 -
    192.168.2.1 HTTP/1.1
    Mozilla/5.0+(Windows+NT+5.1;+rv:8.0)+Gecko/20100101+Firefox/8.0
    https://www.virtualeternity.com/ 304 0 0
    229 187
    2011-11-15 14:37:38 192.168.2.187 GET /js/form.js - 443 - 192.168.2.1 HTTP/1.1
    Mozilla/5.0+(Windows+NT+5.1;+rv:8.0)+Gecko/20100101+Firefox/8.0
    https://www.virtualeternity.com/ 304 0 0
    229 46

    LogFormat = ="%time2 %other %method %url %query $other %other %host %other %ua
    %referer %code %other %other %bytesd %other"

    LevelForOSDetection=1 (I have also tried 2)
    LevelForRobotsDetection=1 (also tried 2)
    LevelForeSearchEnginesDetection=1 (also tried 2)

    When I look at my unknown operating systems, I see the following:
    Mozilla/5.0_(compatible;Baiduspider/2.0;http://www.baidu.com/search/spider
    .html) 15 Nov 2011 - 22:28
    Mozilla/5.0_(compatible;_Googlebot/2.1;
    http://www.google.com/bot.html) 15 Nov 2011 - 22:22
    Feedfetcher-Google;
    (http://www.google.com/feedfetcher.html;_feed-
    id=5065448767456465765) 15 Nov 2011 - 21:49
    Mozilla/5.0
    (compatible;YodaoBot/1.0;http://www.yodao.com/help/webmaster/sp
    ider/;_)
    15 Nov 2011 -
    21:27
    facebookplatform/1.0_(http://developers.facebook.com) 15 Nov 2011 - 14:07
    facebookexternalhit/1.0
    (http://www.facebook.com/externalhit_uatext.php) 15 Nov 2011 - 12:30
    Mozilla/5.0
    (compatible)Feedfetcher-Google;(http://www.google.com/feedfetc
    her.html) 15 Nov 2011 - 12:08
    OurBrowser/1.0
    (Some_Name)

    Which is very odd, because the logs contain the OS information.

    Any suggestions? Any other information you need to see?

    thanks,
    Kim

     
  • Jean-Luc

    Jean-Luc - 2011-11-15

    Hi,

    I have run a quick test using your LogFormat directive. I pasted the lines
    from your log file in a file here. I got an AWStats report with 8 valid hits:
    - browsers: 6 Firefox + 1 unknonw + 1 Google Chrome
    - operating systems: 7 Windows + 1 unknown
    I am using AWStats 7.0 with the default values for OS and browser detection.

    This is weird. Did you download AWStats from the SourceForge site?

    Jean-Luc, InternetOfficer (AWStats
    Support
    )

     
  • Francesco Mazzi

    Francesco Mazzi - 2011-11-18

    Same problem here. I'm using AWStats 7.0; log format is apache combined;
    browsers are correctly detected, but OS and robots are not; the unique row in
    robots is:

    Unknown robot (identified by hit on 'robots.txt')

    and when I look at my unknown operating systems, int thefull list there are
    589 accesses, if I click on full list/versions I see Unknown at 100% and if I
    come back and click on Unknown (at the right of full list/versions ) I see 21
    unknown robots:

    Mozilla/5.0_(compatible;Googlebot/2.1;http://www.google.com/bot.html) 17 Nov 2011 / 23:58
    facebookexternalhit/1.0_(http://www.facebook.com/externalhit_uatext.php) 17 Nov 2011 / 23:36
    Opera/9.80
    (Android;Opera_Mini/6.24556/26.1153;_U;_it)_Presto/2.8.119_Version
    /10.54 17 Nov 2011 / 21:53
    DoCoMo/2.0_N905i(c100;TB;W24H16)
    (compatible;_Googlebot-Mobile/2.1;
    http://w
    ww.google.com/bot.html) 17 Nov 2011 /
    21:48
    msnbot/2.0b
    (http://search.msn.com/msnbot.htm)._ 17 Nov 2011 / 21:45
    Mozilla/5.0
    (compatible;Yahoo!_Slurp;http://help.yahoo.com/help/us/ysearch/
    slurp)
    17 Nov 2011 / 21:40
    WebNews_Arianna 17 Nov 2011 / 21:34
    Mozilla/5.0_(compatible;bingbot/2.0;http://www.bing.com/bingbot.htm) 17 Nov 2011 / 20:45
    Mozilla/4.0_(compatible;) 17 Nov 2011 / 17:41
    AppEngine-Google;(http://code.google.com/appengine;_appid:_mapthislink) 17 Nov 2011 / 17:25
    Mozilla/5.0_(compatible;_TweetmemeBot/2.11;
    http://tweetmeme.com/) 17 Nov 2011 / 12:47
    UnwindFetchor/1.0
    (http://www.gnip.com/) 17 Nov
    2011 / 12:44
    JS-Kit_URL_Resolver,
    http://js-kit.com/ 17 Nov 2011 /
    12:44
    Mozilla/5.0_(compatible;TweetedTimes_Bot/1.0;http://tweetedtimes.com) 17 Nov 2011 / 10:06
    Mozilla/5.0_(compatible;PaperLiBot/2.1;http://support.paper.li/entries/2002
    3257-what-is-paper-li) 17 Nov 2011 / 09:20
    Showyoubot_(http://showyou.com/support) 17
    Nov 2011 / 08:26
    Mozilla/5.0_(compatible;_YandexBot/3.0;
    http://yandex.com/bots) 17 Nov 2011 / 06:59
    Mozilla/5.0
    (compatible;YandexImages/3.0;__http://yandex.com/bots) 17 Nov 2011 / 06:20
    Content_Crawler 17 Nov 2011 / 01:28
    Mozilla/5.0
    (compatible;MJ12bot/v1.4.0;http://www.majestic12.co.uk/bot.php?
    _)
    17 Nov 2011 / 01:09
    Mozilla/5.0_(en-us)AppleWebKit/534.14(KHTML,_like_Gecko;_Google_Wireless_Tra
    nscoder)_Chrome/9.0.597_Safari/534.14 17 Nov 2011 / 00:32

    I think awstats configuration is correct, I checked it a lot.
    What could be the problem?
    Thanks,
    Francesco

     
  • Francesco Mazzi

    Francesco Mazzi - 2011-11-21

    Update: same problem with 6.95 (build 1.943).
    Tested both on Windows Server 2003 R2 with IIS and Activeperl.

     
  • Francesco Mazzi

    Francesco Mazzi - 2011-11-21

    Another important note: I upload logs from a Apache web server to the Windows
    Server 2003 machine with IIS, where I use awstats.

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks