Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

Import of old logs

MPDL
2010-11-24
2012-10-11
  • MPDL
    MPDL
    2010-11-24

    Hi all,
    i will import old logs from apache on Suse.
    Actually the update every day is running normal.
    But now i would like to import the old logs.
    If i will import the old logs i get the following output from awstats if i
    make an update with old logs.

    by AWStats version 6.8 (build 1.910)
    From data in log file "/root/test.txt"...
    Phase 1 : First bypass old records, searching new record...
    Direct access to last remembered record has fallen on another record.
    So searching new records from beginning of log file...
    Jumped lines in file: 0
    Parsed lines in file: 818808
    Found 45 dropped records,
    Found 2998 corrupted records,
    Found 815765 old records,
    Found 0 new qualified records.

    That's normal, awstats add only newer log entrys.

    If i start awstats with -showcorrupt i get this

    Phase 1 : First bypass old records, searching new record...
    Direct access after last parsed record (after line 818808)
    Jumped lines in file: 818808
    Found 818808 already parsed records.
    Parsed lines in file: 0
    Found 0 dropped records,
    Found 0 corrupted records,
    Found 0 old records,
    Found 0 new qualified records.

    I moved the database files from awstats to have any newer logs in awstats.

    if i start the update again, i get this output.

    by AWStats version 6.8 (build 1.910)
    From data in log file "/root/test.txt"...
    Phase 1 : First bypass old records, searching new record...
    Searching new records from beginning of log file...
    Phase 2 : Now process new records (Flush history on disk after 20000 hosts)...
    Jumped lines in file: 0
    Parsed lines in file: 818808
    Found 45 dropped records,
    Found 798245 corrupted records,
    Found 0 old records,
    Found 20518 new qualified records.

    I didn't change the log format, that newer logs will be added without problems
    and adding old logs the most of them are corrupted.

    best regards

    Benny

     
  • Jean-Luc
    Jean-Luc
    2010-11-24

    Hi,

    It looks like the log format has been changed though.

    Can you paste here a line from the old log and your LogFormat directive ?

    Jean-Luc, InternetOfficer AWStats
    Support

     
  • MPDL
    MPDL
    2010-11-24

    Hi Jean-Luc
    i use LogFormat=1.

    65.55.106.156 - - "GET /robots.txt HTTP/1.1" 200 26 "-" "msnbot/2.0b
    (+http://search.msn.com/msnbot.htm)"
    "-"
    192.76.128.12 - - "GET / HTTP/1.1" 200 11242 "-" "OpenNMS HttpMonitor" "-"

     
  • Jean-Luc
    Jean-Luc
    2010-11-25

    Hi,

    LogFormat=1
    

    is the same as

    LogFormat = "%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"
    

    In the two lines from your log file, there is an additional field between
    quotes at the end.

    You can try :

    LogFormat = "%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %otherquot"
    

    Jean-Luc, InternetOfficer AWStats
    Support

     
  • MPDL
    MPDL
    2010-11-25

    Hi,
    i got the same result after i used the LogFormat with and without "%otherquot"

    by AWStats version 6.8 (build 1.910) From data in log file "/root/test.txt"...
    Phase 1 : First bypass old records, searching new record... Searching new
    records from beginning of log file... Phase 2 : Now process new records (Flush
    history on disk after 20000 hosts)... Jumped lines in file: 0 Parsed lines in
    file: 818808 Found 45 dropped records, Found 798245 corrupted records, Found 0
    old records, Found 20518 new qualified records.

     
  • Jean-Luc
    Jean-Luc
    2010-11-25

    A few things to check or try:
    - Aren't there unusual or hidden characters in the log files, maybe at the beginning or at the end of the lines ?
    - What character is used as field separator ? Is it a space or a tab ?

    Jean-Luc, InternetOfficer AWStats
    Support

     
  • MPDL
    MPDL
    2010-11-25

    I didn't found unusual or hidden characters.
    I don't know where i should search of the field seperator. I think it's
    default.

    My old log-files were in seperate files saved for every day.
    I listed all access_logs from apache, cat the content and save the content in
    one file to megrate them in awstats.

    Is it possible that i get so many corrupted items if the log in this file save
    the items not sorted by date?

    For example:

    The compact logfile have 1000 logs from april 2010 than it will be follow with
    500 logs from juli and than follow 500 from february.

    Awstats get the following output after the imput.

    I got many statistics from april, august, september, but any statistic from
    juni - juli and all other month earlier i also couldn't import statistics.

    If it is the problem, who can i import all my logfiles since last year.

    Command which i used to pack all files in one.

    find . -name access_log* -exec cat {} >> log.txt \

     
  • Jean-Luc
    Jean-Luc
    2010-11-25

    You wrote : Is it possible that i get so many corrupted items if the log in
    this file save the items not sorted by date?

    The input file must be chronologically sorted. There is some tolerance, but
    you cannot process day 1 after you already processed a hit from day 2.

    Refer toi How can I process old log
    file
    in the
    FAQ.

    Jean-Luc, InternetOfficer AWStats
    Support

     
  • MPDL
    MPDL
    2010-11-26

    i got a log-file merged all the old logfiles but not chronologically sorted.

    Do you or somebody else know how i can sort this log-file?

     
  • MPDL
    MPDL
    2010-12-10

    Apache logged chronologic the logs and saved daily the log files in seperate
    folder. The problem was that i got a merged file with all logs, but this
    process, where the logs in all the log files merged in one file, but noch
    chronologically.
    And now i must find a solution to import the logs in awstats and if i need a
    sorted file than i need to find a way to sort this file.
    I would like to merge the files with /logresolvemerge.pl. But the first
    problem is, that the merged file with all the logs can't be sorted, if i
    understand it. And the second problem is that the daily saved log files are
    currupted after recompressing.

    Exist a script which sort this log file or must i do a manual sort? :P

     
  • frank
    frank
    2010-12-28

    In the tools dir is Logresolvemerge.pl . check FAQs on how to use it.... It
    works great for me.

    Frank

     
  • MPDL
    MPDL
    2011-01-03

    I don't know what option should i use to sort one file in a new.

    If i run perl logresolvemerge.pl logfile.txt > sorted_logfile i get a 1:1 copy
    of my unsorted file.
    "Diff" is printing no changes and the first entry is from april but older
    files are from february are in this logfile.

    For example:

    3 (Linux/SUSE) (internal dummy connection)" "-"
    ::1 - - "GET / HTTP/1.0" 200 11243 "-" "Apache/2.2.
    3 (Linux/SUSE) (internal dummy connection)" "-"
    ::1 - - "GET / HTTP/1.0" 200 11242 "-" "Apache/2.2.
    3 (Linux/SUSE) (internal dummy connection)" "-"
    ::1 - - "GET / HTTP/1.0" 200 11243 "-" "Apache/2.2.
    3 (Linux/SUSE) (internal dummy connection)" "-"

    Benny

     
  • Jean-Luc
    Jean-Luc
    2011-01-03

    Hi,

    logresolvemerge does not sort files. It merges already sorted files into a
    larger sorted file. Inyour case, the right case would look like:

    perl logresolvemerge.pl access_log* > sorted_logfile
    

    Jean-Luc, InternetOfficer AWStats
    Services

     
  • MPDL
    MPDL
    2011-01-04

    I can't use the old log files. The files are corrupted.
    Now i have only one big file "logfile.txt". This file have all log files
    merged but not sorted.
    If i would like to add this log file in awstats i need a tool where i can sort
    the big file "logfile.txt".

    Do anybody know a tool or a solution to add this unsorted logfile.txt to
    awstats?

    Benny