[Expectperl-discuss] Logfile-watching program - design questions

[Chap H. <cl...@po...>]

First of all, this may be considered purely an Expect question, but I do
intend to use Expect.pm, and I can't seem to find a viable Expect forum
anywhere else.

My job entails running a GUI-driven school district data conversion program. 
This involves (a) selecting a school, (b) clicking "convert school", and (c)
moving the output files - of which there are about 15 - to a safe location
when the conversion is finished, which can take between 3 minutes and 1.5
hours.  Then I start again at (a), selecting the next school in the
district.  

The GUI also has a button labelled "convert entire district".  You'd think
it would alleviate the need to babysit on a school-by-school basis, but it
doesn't: it OVERWRITES the output files for each new school!  Unfortunately,
we don't own the source and can't fix this obviously broken behavior, so we
have had to live with converting on a school-by-school basis.

However, the conversion program *does* write log files - a separate one for
each of the 15 output files it creates.  And happily, it appends to - rather
than overwriting - each one!

I want to work around the flaw in the "convert entire district"
functionality with an Expect program, as follows: 

- spawn the Unix 'tail -f' command to watch each log file, 
- look for the messages indicating that the conversion program has finished
writing a file, 
- rename the file with a unique qualifier so that it won't be overwritten
when the conversion circles around to the next school.

NOTE that once conversion finishes one file (e.g. Students), it starts a
different one (e.g. Schedules) with a different file name.  As I mentioned,
converting a school produces around 15 different files, and it takes a
minimum of about 3 minutes per school.  So, while my Expect script would be
"racing" to rename the output file before the conversion program circles
around to start the next school, there is a multi-minute window of time in
which to do so.

Here's a simplified example of one log file:

Students.log
- - - - -
initialization preamble
blah blah blah
THIS MESSAGE INDICATES BEGINNING OF A CONVERSION
blah blah blah
more blah blah blah
THIS IS THE ID OF THE SCHOOL WE'RE CONVERTING: 003
blah blah blah
lots more blah blah blah - hundreds of lines
 :   :   :
THIS MESSAGE INDICATES END OF A FILE CONVERSION
blah blah blah
yadda yadda yadda
THIS MESSAGE INDICATES BEGINNING OF A FILE CONVERSION
blah blah blah
more blah blah blah
THIS IS THE ID OF THE SCHOOL WE'RE CONVERTING: 003
blah blah blah
lots more blah blah blah - hundreds of lines
 :   :   :
THIS MESSAGE INDICATES END OF A FILE CONVERSION
blah blah blah
yadda yadda yadda
THIS MESSAGE INDICATES BEGINNING OF A FILE CONVERSION
blah blah blah
etc.
- - - - -

Here are the simplified guts of my program (in TCL):

# Spawn a 'tail' process for each log file
foreach file $logfiles {
    spawn -noecho /usr/bin/tail -f $file
    lappend spawn_id_list $spawn_id
}

# Watch for significant output
while {1} {
   expect {
      -i $spawn_id_list
   
      -exact {THIS MESSAGE INDICATES BEGINNING OF A CONVERSION} {
      }

      -re {THIS IS THE ID OF THE SCHOOL WE'RE CONVERTING: (\d\d\d)} {
      }

      -exact {THIS MESSAGE INDICATES END OF A FILE CONVERSION} {
         # Give the output file a unique name by appending school id
      }
   }
}

I've obviously left out a lot here.  One problem I've been having is that
sometimes Expect "misses" the end-of-conversion message, and I think that's
because the end-of-conversion message and the next beginning-of-conversion
message happen so closely to one another that sometimes they appear together
in the same buffer Expect examines.  Since "BEGINNING..." will be searched
(and found) before "END...", this could be my problem.

I'm wondering how to get around this problem, and wondering, too, if this is
the best way to approach the whole thing.

Incidentally, I want to use Perl rather than TCL because I encountered bugs
with TCL in CygWin, whereas Perl has always been reliable on that platform.
-- 
View this message in context: http://old.nabble.com/Logfile-watching-program---design-questions-tp27313339p27313339.html
Sent from the Perl - Expectperl-Discuss mailing list archive at Nabble.com.