[Expectperl-discuss] Beginner Questions: Reference & How to Handle Large Output

[Rick_Jasper/Delphion%<DEL...@de...>]

Where's the best how-to/tutorial reference for Expect under Perl for
beginners?  I have the Exploring Expect and Perl books, but the Expect book
is difficult to digest and the Perl books don't have much on Expect.  I'm
looking for good, detailed examples of using the Perl/Expect module, not
the Tcl/Expect, aimed at the novice.  The pod is sparse and presumes more
Perl and Expect experience than I have.

I've got Expect 1.12 on an AIX 4.3.3 system, trying to figure out how best
to use it to ssh to some other system overseas, run a command on that
oversea system that produces a lot of output (1.5 million lines, 136 MB
total), collect and process that output on my system.

When I ssh to the remote system and run the command manually, the 1.5
million lines scrolls past my screen in 46 minutes (that's 32,000
lines/minute).  But when I use Expect to login, run the command and capture
the output, it takes forever!  It actually starts out at 10,000
lines/minute, which I can live with I suppose (2.5 hours total), but then
it slows down progressively until it's crawling at  700 lines/minute.  At
this rate, it will take 32 hours to finish.

Why does capturing output with Expect, slow things down so dramatically?
I've tried tweaking my Expect loop, removing regex matching (that helped),
but most of the things I've tried - removing all patterns, setting
restart_timeout_upon_receive=1, increasing match_max - don't make a
difference.

What is the best way to debug this?  Due to the amount of data, it's
impractical to turn on normal debugging (Debug, Exp_Internal, Log_Stdout).
I'm tracking the progress by setting log_file.  I don't know how much this
slows things down.  Possibly a lot, but when I didn't do that, the program
would die with no indication why.

Any guidance would be greatly appreciated.

Rick Jasper