Re: [Expectperl-discuss] Another threading question

[Bryan B. <br...@bu...>]

>
> Hey folks.  I've really appreciated the posts in this forum, as they have
> gotten me past the initial roadblocks to getting Expect to work in a
> threaded PERL application.  Unfortunately, I've hit a bit of a snag and I
> thought I'd post the problem here to see if anyone's experienced the same
> type of issue.  Following what's been suggested here, I'm creating my
> initial Expect objects in the main thread and establishing ssh connections
> to the load balancers I need.  I'm passing these objects to a handful of
> boss threads, which first loop through a set of servers to remove them
> from
> the load balancer via Expect.  Each boss thread then creates a pool of
> child
> threads that issue commands against this same set of servers and doesn't
> use
> Expect (or the connection object).  The code works great on the first set
> of
> servers, but after I join the first group of threads and they finish, the
> second set is returning undefined for all the expect(...) calls.  What
> makes
> this so strange is that if I turn on debugging for Expect, it works fine
> (although it floods STDOUT).  I've tried a number of things to try and
> solve
> the problem (flushing STDOUT, calling clear_accum(), turning on warnings
> and
> diagnostics).  The only hint of what's going on came through diagnostics,
> which throws the following error when the problems start to occur:
>
> print() on closed filehandle GEN0 at
>         /usr/lib/perl5/5.8.0/i386-linux-thread-multi/IO/Handle.pm line
> 395,
> <STDIN> line 3 (#1)
>     (W closed) The filehandle you're printing on got itself closed
> sometime
>     before now.  Check your control flow.
>
> Here's a simplified version of the code from the boss thread that appears
> to
> be causing problems:
>
> # First remove all the servers from the load balancer pool
> foreach my $serv (@server_stack) {
>     $result = &run_command($expect_connection,"disable $serv");
> }
> # Run some commands against this set of servers now that they're out of
> the
> pool
> my @threads = ();
> while (@server_stack) {
>         my $server = pop(@server_stack);
>         push @threads, threads->new(\&shutdown_server,$server);
> }
> foreach my $thr (@threads) {
>     @return_vals = (@return_vals, $thr->join());
> }
> foreach my $thr (@threads) {
>     undef $thr;
> }
>
> Since I'm using PERL 5.8.0, I'm undef'ing the threads once they are
> finished, since there were some issues with this in 5.8.0.  This code
> block
> runs fine the first time, but as soon as I send it a new @server_stack,
> the
> calls to &run_command no longer work.  Here's what run_command looks like:
>
> sub run_command {
>     my($proc, $command) = @_;
>     my $timeout = 6;
>     $proc->raw_pty(1);
>     $proc->log_stdout(0);
>     $proc->log_file("temp.log");
>     $proc->debug(0);
>     $proc->send("$command\r");
>     my $i = $proc->expect($timeout, "-re", " # ");
>     if (!(defined $i)) {
>         return "ERROR: no response to command!\n";
>     }
>     return $proc->before();
> }
>
> However, if I set $proc->debug(4), it works fine, albeit a bit ugly to
> watch.  It almost seems like something is closing the filehandle that the
> Expect connection object is using after the first thread group finishes,
> even though the thread group isn't even operating on the connection
> object.
> I apologize for only posting snippets, but there is too much code to post
> otherwise.  I'm getting this system upgraded to PERL 5.8.8, but it's not
> going to be anytime soon, so I'm trying to make this work with what I've
> got.
>
> If anyone has experienced something similar or has an idea what might fix
> this, it would be much appreciated.  I've been pulling my hair out on this
> one for a while now.  Thanks for listening.
>
> -Phil
> --

I've had the same problem, and my fix was the undef piece you already
have.  However, I'm not sure what state the expect objects are in when you
start your new group of threads.

What has worked for me is this: I setup expect objects on a group of
servers first, then exec, then close the expect objects, and finally join
them.  Next I move on to the next group of servers and do the same.

This is my basic routine:


my $max_threads = 4;
while (@servers || @server_stack) {
  my $host = shift @servers;

  my $exp = function_to_setup_expect($host) || next;
  push(@server_stack, $host);

  if (@server_stack == $max_threads || !@servers) {
    foreach my $server (@server_stack) {
      push @threads, threads->new(\&shutdown_server,$server);
    }
    foreach my $thr (@threads) {
      @return_vals = (@return_vals, $thr->join());
      undef $thr;
    }
    @server_stack = ();
    @threads = ();
  }
}

Not saying there isnt something else wrong here, but like i said, this
method works for me in all 5.8.0 versions.

Hope that helps.


Bryan
http://sourceforge.net/projects/rover