Re: [Expectperl-discuss] Logfile-watching program - design questions

[Roland G. <rgi...@cp...>]

Hi Chap,

no need to use Expect for that, I'd recommend using File::Tail instead. 
It can tail multiple files at once and you can handle each line 
separately, which is easier than emulating that with Expect.

Hope this helps,

Roland

Chap Harrison schrieb:
> First of all, this may be considered purely an Expect question, but I do
> intend to use Expect.pm, and I can't seem to find a viable Expect forum
> anywhere else.
>
> My job entails running a GUI-driven school district data conversion program. 
> This involves (a) selecting a school, (b) clicking "convert school", and (c)
> moving the output files - of which there are about 15 - to a safe location
> when the conversion is finished, which can take between 3 minutes and 1.5
> hours.  Then I start again at (a), selecting the next school in the
> district.  
>
> The GUI also has a button labelled "convert entire district".  You'd think
> it would alleviate the need to babysit on a school-by-school basis, but it
> doesn't: it OVERWRITES the output files for each new school!  Unfortunately,
> we don't own the source and can't fix this obviously broken behavior, so we
> have had to live with converting on a school-by-school basis.
>
> However, the conversion program *does* write log files - a separate one for
> each of the 15 output files it creates.  And happily, it appends to - rather
> than overwriting - each one!
>
> I want to work around the flaw in the "convert entire district"
> functionality with an Expect program, as follows: 
>
> - spawn the Unix 'tail -f' command to watch each log file, 
> - look for the messages indicating that the conversion program has finished
> writing a file, 
> - rename the file with a unique qualifier so that it won't be overwritten
> when the conversion circles around to the next school.
>
> NOTE that once conversion finishes one file (e.g. Students), it starts a
> different one (e.g. Schedules) with a different file name.  As I mentioned,
> converting a school produces around 15 different files, and it takes a
> minimum of about 3 minutes per school.  So, while my Expect script would be
> "racing" to rename the output file before the conversion program circles
> around to start the next school, there is a multi-minute window of time in
> which to do so.
>
> Here's a simplified example of one log file:
>
> Students.log
> - - - - -
> initialization preamble
> blah blah blah
> THIS MESSAGE INDICATES BEGINNING OF A CONVERSION
> blah blah blah
> more blah blah blah
> THIS IS THE ID OF THE SCHOOL WE'RE CONVERTING: 003
> blah blah blah
> lots more blah blah blah - hundreds of lines
>  :   :   :
> THIS MESSAGE INDICATES END OF A FILE CONVERSION
> blah blah blah
> yadda yadda yadda
> THIS MESSAGE INDICATES BEGINNING OF A FILE CONVERSION
> blah blah blah
> more blah blah blah
> THIS IS THE ID OF THE SCHOOL WE'RE CONVERTING: 003
> blah blah blah
> lots more blah blah blah - hundreds of lines
>  :   :   :
> THIS MESSAGE INDICATES END OF A FILE CONVERSION
> blah blah blah
> yadda yadda yadda
> THIS MESSAGE INDICATES BEGINNING OF A FILE CONVERSION
> blah blah blah
> etc.
> - - - - -
>
> Here are the simplified guts of my program (in TCL):
>
> # Spawn a 'tail' process for each log file
> foreach file $logfiles {
>     spawn -noecho /usr/bin/tail -f $file
>     lappend spawn_id_list $spawn_id
> }
>
> # Watch for significant output
> while {1} {
>    expect {
>       -i $spawn_id_list
>    
>       -exact {THIS MESSAGE INDICATES BEGINNING OF A CONVERSION} {
>       }
>
>       -re {THIS IS THE ID OF THE SCHOOL WE'RE CONVERTING: (\d\d\d)} {
>       }
>
>       -exact {THIS MESSAGE INDICATES END OF A FILE CONVERSION} {
>          # Give the output file a unique name by appending school id
>       }
>    }
> }
>
> I've obviously left out a lot here.  One problem I've been having is that
> sometimes Expect "misses" the end-of-conversion message, and I think that's
> because the end-of-conversion message and the next beginning-of-conversion
> message happen so closely to one another that sometimes they appear together
> in the same buffer Expect examines.  Since "BEGINNING..." will be searched
> (and found) before "END...", this could be my problem.
>
> I'm wondering how to get around this problem, and wondering, too, if this is
> the best way to approach the whole thing.
>
> Incidentally, I want to use Perl rather than TCL because I encountered bugs
> with TCL in CygWin, whereas Perl has always been reliable on that platform.
>