Chris - 2005-09-09

Hi!

Our websites are hosted on a 2 Server Load Balancing System with logfiles written every day bz2 compressed.

2005_01_01.bz2
2005_01_02.bz2
...

Unfortunately my task is to implement accesslog statistics on a win2003 for about 10 websites; for each one seperate log files are written.

i'm currentlich trying to merge those logfiles with the logresolvemerge.pl and encountered a problem.
The logresolvemerge.pl didn't seem to support use of multiple bz2 compressed files when calling it with something like

logresolvemerge.pl ../logs/machine1/. ../logs/machine2/. > testoutput.log

so i've added some lines to the script
line 431:
foreach my $i (0..$#filearray) {
if ("$filearray[$i]" =~ /^$ParamFile[$key]$/ && "$filearray[$i]" ne "." && "$filearray[$i]" ne "..") {
if ($Debug) { debug("Log file $filearray[$i] is added to LogFileToDo with number $cpt."); }
if ("$filearray[$i]" =~ /$bzcat_file/) {
if ($Debug) { debug("BZ2 compression detected for Log file $DirFile $filearray[$i]."); }
# Modify the name to include the bzcat command
$filearray[$i] = $bzcat . ' ' . $DirFile .''.$filearray[$i] . ' |';
}
$LogFileToDo{$cpt}="$filearray[$i]";
#$LogFileToDo{$cpt}="$DirFile/$filearray[$i]";
$cpt++;
}

which seems to work fine as far as the file search is concerned ($Debug=3)
DEBUG 1 - Fri Sep 9 08:46:18 2005 : Logresolvemerge - 1.2 (build 1.31) - Perl C:\Perl\bin\perl.exe 5.008007
DEBUG 1 - Fri Sep 9 08:46:18 2005 : DNSLookup=0
DEBUG 1 - Fri Sep 9 08:46:18 2005 : DNSCache=
DEBUG 1 - Fri Sep 9 08:46:18 2005 : MaxNbOfThread=0
DEBUG 1 - Fri Sep 9 08:46:18 2005 : Search for file "..." into "../logs/201/bz2/"
DEBUG 1 - Fri Sep 9 08:46:18 2005 : Log file xxx_access_log.2005_06_01#36.bz2 is added to LogFileToDo with number 1.
DEBUG 1 - Fri Sep 9 08:46:18 2005 : BZ2 compression detected for Log file ../logs/201/bz2/
xxx_access_log.2005_06_01#36.bz2.
....
DEBUG 1 - Fri Sep 9 08:46:18 2005 : Search for file "..." into "../logs/202/bz2/"
DEBUG 1 - Fri Sep 9 08:46:18 2005 : Log file xxx_access_log.2005_06_01#36.bz2 is added to LogFileToDo with number 62.
DEBUG 1 - Fri Sep 9 08:46:18 2005 : BZ2 compression detected for Log file ../logs/202/bz2/ xxx_access_log.2005_06_01#36.bz2.
...

then opening of the files

DEBUG 1 - Fri Sep 9 08:46:18 2005 : Open log file number 33: "bzcat ../logs/201/bz2/xxx_access_log.2005_07_03#36.bz2 |"
DEBUG 1 - Fri Sep 9 08:46:18 2005 : Open log file number 32: "bzcat ../logs/201/bz2/xxx_access_log.2005_07_02#36.bz2 |"
DEBUG 1 - Fri Sep 9 08:46:18 2005 : Open log file number 90: "bzcat ../logs/202/bz2/xxx_access_log.2005_06_29#36.bz2 |"
...

after opening some files

Error: Couldn't open log file "bzcat ../logs/202/bz2/P_ARGEALP_access_log.2005_06_15#36.bz2 |" : Bad file descriptor.

also in my command line the bzcat prog reports various errors like "bad argument", "broken pipe", and "I/O Error". The Logs are surely in the right place so i dont understand why some files seem to open correctly and then i get this "bad file descriptor".

I'd like to know if anyone encountered the same problem or can tell me how/if it's possible to realize somthing like that on a win2003 system.

thx in advance

chris