awstats log parsing

2014-07-18
2014-07-22
  • Yogesh Kumar
    Yogesh Kumar
    2014-07-18

    perl /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=Legacyweb -update
    Create/Update database for config "/etc/awstats/awstats.Legacyweb.conf" by AWStats version 7.3 (build 20140126)
    From data in log file "/www/logs/access_log"...
    Phase 1 : First bypass old records, searching new record...
    Searching new records from beginning of log file...
    Jumped lines in file: 0
    Parsed lines in file: 2
    Found 2 dropped records,
    Found 0 comments,
    Found 0 blank records,
    Found 0 corrupted records,
    Found 0 old records,
    Found 0 new qualified records.

    below are the entries from log file

    [17/Jul/2014:19:59:59 -0400] "75.67.203.213, 184.25.109.95" GET "/cc-common/mlib/1785/06/1785_14029314585.jpg" "" 200 820653 0 "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D201 Safari/9537.53" "http://m.94hjy.com/photos/hjy-vip-photos/hooters-miss-bro-show-bikini-contest-393810/" "content.clearchannel.com" - + 10.9.10.92
    [17/Jul/2014:19:59:59 -0400] "107.77.64.107, 184.26.62.143" GET "/cc-common/local-poc/html/now_playing_tpl_qio.html" "?t=2" 200 363 0 "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; ARM; Touch; WPDesktop)" "http://www.1043myfm.com/onair/dave-styles-53596/lordes-royals-gets-the-weird-al-12573862" "www.1043myfm.com" - + 10.9.10.92

    LOG FORMAT: LogFormat = "%time1 %host_proxy %method %url \"%query\" %code %bytesd %other \"%ua\" \"%referer\" \"%other\" %other %other %other"

     
    • ERROR! The markdown supplied could not be parsed correctly. Did you forget to surround a code snippet with "~~~~"?

      Try
      
      LogFormat = "%time1 \"%host_proxy %other\" %method %url \"%query\" %code
      %bytesd %other \"%ua\" \"%referer\" \"%other\" %other %other %other"
      
      
      2014-07-18 9:53 GMT+02:00 Yogesh Kumar <yogesh2tech@users.sf.net>:
      
      > perl /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=Legacyweb
      > -update
      > Create/Update database for config "/etc/awstats/awstats.Legacyweb.conf" by
      > AWStats version 7.3 (build 20140126)
      > >From data in log file "/www/logs/access_log"...
      > Phase 1 : First bypass old records, searching new record...
      > Searching new records from beginning of log file...
      > Jumped lines in file: 0
      > Parsed lines in file: 2
      >  Found 2 dropped records,
      >  Found 0 comments,
      >  Found 0 blank records,
      >  Found 0 corrupted records,
      >  Found 0 old records,
      >  Found 0 new qualified records.
      >
      >
      > below are the entries from log file
      >
      > [17/Jul/2014:19:59:59 -0400] "75.67.203.213, 184.25.109.95" GET
      > "/cc-common/mlib/1785/06/1785_14029314585.jpg" "" 200 820653 0 "Mozilla/5.0
      > (iPhone; CPU iPhone OS 7_1_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML,
      > like Gecko) Version/7.0 Mobile/11D201 Safari/9537.53" "
      > http://m.94hjy.com/photos/hjy-vip-photos/hooters-miss-bro-show-bikini-contest-393810/"
      > "content.clearchannel.com" - + 10.9.10.92
      > [17/Jul/2014:19:59:59 -0400] "107.77.64.107, 184.26.62.143" GET
      > "/cc-common/local-poc/html/now_playing_tpl_qio.html" "?t=2" 200 363 0
      > "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; ARM;
      > Touch; WPDesktop)" "
      > http://www.1043myfm.com/onair/dave-styles-53596/lordes-royals-gets-the-weird-al-12573862"
      > "www.1043myfm.com" - + 10.9.10.92
      >
      >
      > LOG FORMAT:  LogFormat = "%time1 %host_proxy %method %url \"%query\" %code
      > %bytesd %other \"%ua\" \"%referer\" \"%other\" %other %other %other"
      >
      >
      > ---
      >
      > [awstats log parsing](
      > https://sourceforge.net/p/awstats/discussion/43428/thread/6984e1da/?limit=50#950d
      > )
      >
      >
      > ---
      >
      > Sent from sourceforge.net because you indicated interest in <
      > https://sourceforge.net/p/awstats/discussion/43428/>
      >
      > To unsubscribe from further messages, please visit <
      > https://sourceforge.net/auth/subscriptions/>
      >
      >
      
      
      -- 
      Laurent Destailleur (alias Eldy)
      ------------------------------------------------------------------------------------
      Social networks of my OpenSource projects:
      Dolibarr Google+: https://plus.google.com/+DolibarrOrg/
      Dolibarr Facebook: https://www.facebook.com/dolibarr
      Dolibarr Twitter: http://www.twitter.com/dolibarr
      AWStats Google+: https://plus.google.com/+AWStatsOrgPoject/
      AWStats Facebook: https://www.facebook.com/awstats.org
      AWStats Twitter: http://www.twitter.com/awstats_project
      
       
      • Yogesh Kumar
        Yogesh Kumar
        2014-07-21

        Thank You so much for the response,

        but i am still getting the same issue:

        What I did:

        I use the same LogFormat suggested by you:

        LogFormat = "%time1 \"%host_proxy %other\" %method %url \"%query\" %code
        %bytesd %other \"%ua\" \"%referer\" \"%other\" %other %other %other"

        [root@webanalysis1 logs]# perl /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -co nfig=Legacyweb -update
        Create/Update database for config "/etc/awstats/awstats.Legacyweb.conf" by AWSta ts version 7.3 (build 20140126)
        From data in log file "/www/logs/access_log"...
        Phase 1 : First bypass old records, searching new record...
        Direct access to last remembered record has fallen on another record.
        So searching new records from beginning of log file...
        Jumped lines in file: 0
        Parsed lines in file: 4
        Found 4 dropped records,
        Found 0 comments,
        Found 0 blank records,
        Found 0 corrupted records,
        Found 0 old records,
        Found 0 new qualified records.

        Here are the log entries I tried to parse.

        "[20/Jul/2014:21:36:09 -0400]" "123.125.71.86, 2.20.183.175" GET "/iplaylist/images/albums/200/dru200/u281/u28199m5hqr.jpg" "" 200 11688 0 "Baiduspider-image+(+http://www.baidu.com/search/spider.htm)\nReferer: http://image.baidu.com/i?ct=503316480&z=0&tn=baiduimagedetail" "-" "www.wsrs.com" - + 10.9.10.92
        "[20/Jul/2014:21:36:09 -0400]" "73.181.59.8" GET "/timeline/update/all.html" "?tln=2179502" 200 11050 0 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36" "http://www.khow.com/main.html" "www.khow.com" - + 10.9.10.92
        "[20/Jul/2014:21:36:09 -0400]" "162.90.144.39, 165.254.94.102" GET "/services/now_playing.html" "?streamId=865&limit=10" 200 1833 0 "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" "http://www.v103.com/main.html" "www.v103.com" - + 10.9.10.92
        "[20/Jul/2014:21:36:10 -0400]" "199.87.253.75" GET "/iplaylist/artist/122231/" "" 301 341 0 "Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)" "-" "www.1067litefm.com" - + 10.9.10.92

         
  • Yogesh Kumar
    Yogesh Kumar
    2014-07-22

    i figured out the problem, but still waiting for the solution.

    Apache logs format : LogFormat "\"%t\" \"%{X-Forwarded-For}i\" %m \"%U\" \"%q\" %>s %b %T \"%{User-Agent}i\" \"%{Referer}i\" \"%v\" %u %c %A" awesome

    Awstats Log Format : LogFormat = "%time1 %host_proxy %method %url \"%query\" %code %bytesd %other \"%ua\" \"%referer\" \"%other\" %other %other %other"

    but parsing the entries with single ip address only.

    [22/Jul/2014:02:43:13 -0400] "98.213.47.174" GET "/cc-common/contests/T25contest.css" "" 304 - 0 "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53" "http://news.iheart.clearcontests.com/front/OpenContest.asp?Action=Login&SurveyID=306376&zx=163" "content.clearchannel.com" - + 10.9.10.92

    failed to parse entries like

    "[22/Jul/2014:02:43:13 -0400]" "98.213.47.174, 131.103.136.28" GET "/cc-common/contests/T25contest.css" "" 304 - 0 "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53" "http://news.iheart.clearcontests.com/front/OpenContest.asp?Action=Login&SurveyID=306376&zx=163" "content.clearchannel.com" - + 10.9.10.92

    "[22/Jul/2014:02:43:13 -0400]" "98.213.47.174, 10.74.9.36, 23.74.9.68, 131.103.136.28" GET "/cc-common/contests/T25contest.css" "" 304 - 0 "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53" "http://news.iheart.clearcontests.com/front/OpenContest.asp?Action=Login&SurveyID=306376&zx=163" "content.clearchannel.com" - + 10.9.10.92

    Please help!!