Menu
▾
▴
Re: [Expectperl-discuss] Another threading question
|
From: Philip M. v. R. <po...@po...> - 2007-10-04 17:03:05
|
Hi Bryan. Thanks so much for the quick reply. Due to the threading model/=
hierarchy I'm using (main_thread->boss_threads->worker_threads), I'm establ=
ishing the connection to the load balancers in the main thread to try and a=
void any issues with thread-safety. The boss threads don't return until al=
l of it's worker thread groups is complete, so I'm not closing the Expect o=
bjects after the first group is complete. That may be part of the problem,=
but it seems like they shouldn't interfere since the worker threads aren't=
using the Expect objects at all. I'll give your suggestion a try. In the=
meantime, I've used Net::Telnet and a slightly modified version of the exa=
mple in the docs to establish the ssh connection (I also needed to fix a bu=
g in the example). I just threw it in a simple module:=0A=0A=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=0Apackage MySSH;=0A=0Arequire Exporter;=0A=0Ause strict;=0Ause=
IO::Pty;=0Ause Net::Telnet;=0A=0Aour @ISA =3D ("Exporter");=0Aour @EXPORT =
=3D qw(get_connection);=0A=0A=0Asub get_connection {=0A my ($pty, $s=
sh, @lines);=0A my $host =3D shift;=0A my $user =3D shift;=0A=
my $password =3D shift;=0A my $prompt =3D '/ # /';=0A=0A =
## Start ssh program.=0A $pty =3D &spawn("ssh", "-l", $user, $ho=
st); # spawn() defined below=0A if ($pty =3D~ /ERROR/i) {=0A =
return $pty;=0A }=0A=0A ## Create a Net::Telnet obje=
ct to perform I/O on ssh's tty.=0A use Net::Telnet;=0A $ssh =
=3D new Net::Telnet (-fhopen =3D> $pty,=0A -=
prompt =3D> $prompt,=0A -telnetmode =3D> 0,=
=0A -cmd_remove_mode =3D> 1,=0A =
-output_record_separator =3D> "\r");=0A=0A ## Log=
in to remote host.=0A $ssh->waitfor(-match =3D> '/Password: ?$/i',=
=0A -errmode =3D> "return")=0A or return "E=
RROR: problem connecting to host: ", $ssh->lastline;=0A $ssh->print(=
$password);=0A $ssh->waitfor(-match =3D> $ssh->prompt,=0A =
-errmode =3D> "return")=0A or return "ERROR: login fa=
iled: ", $ssh->lastline;=0A=0A return $ssh;=0A}=0A=0Asub spawn {=0A =
my(@cmd) =3D @_;=0A my($pid, $pty, $tty, $tty_fd);=0A=0A =
## Create a new pseudo terminal.=0A use IO::Pty ();=0A $pt=
y =3D new IO::Pty=0A or return "ERROR: $!";=0A=0A ## Exec=
ute the program in another process.=0A unless ($pid =3D fork) { # c=
hild process=0A return "ERROR: problem spawning program: $!\n" u=
nless defined $pid;=0A=0A ## Disassociate process from existing =
controlling terminal.=0A use POSIX ();=0A POSIX::sets=
id=0A or return "ERROR: setsid failed: $!";=0A=0A =
## Associate process with a new controlling terminal.=0A $tty =
=3D $pty->slave;=0A ## This line is missing in the Net::Telnet=
docs - it won't work otherwise=0A $pty->make_slave_controlling_=
terminal();=0A $tty_fd =3D $tty->fileno;=0A close $pt=
y;=0A=0A ## Make stdio use the new controlling terminal.=0A =
open STDIN, "<&$tty_fd" or die $!;=0A open STDOUT, ">&$tt=
y_fd" or die $!;=0A open STDERR, ">&STDOUT" or die $!;=0A =
close $tty;=0A=0A ## Execute requested program.=0A =
exec @cmd=0A or return "ERROR: problem executing $cmd[0]\=
n";=0A } # end child process=0A=0A $pty;=0A} # end sub spawn=
=0A=0A1;=0A=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=0A=0AThen I can just call get_connecti=
on($server,$user,$password) in the main thread and pass the returned Net::T=
elnet object around without closing the connection after each worker thread=
group is complete. This has been working so far. =0A=0AOne thing that ha=
s not been resolved in any solution I've tried is the "Scalars leaked: 2" w=
arnings. I've read that these are relatively harmless and there is even so=
me mention that this is a bug in Perl 5.8.x. Is there any way to suppress =
these? I've commented out warnings and diagnostics, but they still show up=
.=0A=0AThanks again for the help.=0A=0A-Phil=0A=0A----- Original Message --=
--=0AFrom: Bryan Bueter <br...@bu...>=0ATo: POMvR <pomvr@pobox.c=
om>=0ACc: exp...@li...=0ASent: Thursday, Octobe=
r 4, 2007 6:57:14 AM=0ASubject: Re: [Expectperl-discuss] Another threading =
question=0A=0A=0A>=0A> Hey folks. I've really appreciated the posts in thi=
s forum, as they have=0A> gotten me past the initial roadblocks to getting =
Expect to work in a=0A> threaded PERL application. Unfortunately, I've hit=
a bit of a snag and I=0A> thought I'd post the problem here to see if anyo=
ne's experienced the same=0A> type of issue. Following what's been suggest=
ed here, I'm creating my=0A> initial Expect objects in the main thread and =
establishing ssh connections=0A> to the load balancers I need. I'm passing=
these objects to a handful of=0A> boss threads, which first loop through a=
set of servers to remove them=0A> from=0A> the load balancer via Expect. =
Each boss thread then creates a pool of=0A> child=0A> threads that issue co=
mmands against this same set of servers and doesn't=0A> use=0A> Expect (or =
the connection object). The code works great on the first set=0A> of=0A> s=
ervers, but after I join the first group of threads and they finish, the=0A=
> second set is returning undefined for all the expect(...) calls. What=0A=
> makes=0A> this so strange is that if I turn on debugging for Expect, it w=
orks fine=0A> (although it floods STDOUT). I've tried a number of things t=
o try and=0A> solve=0A> the problem (flushing STDOUT, calling clear_accum()=
, turning on warnings=0A> and=0A> diagnostics). The only hint of what's go=
ing on came through diagnostics,=0A> which throws the following error when =
the problems start to occur:=0A>=0A> print() on closed filehandle GEN0 at=
=0A> /usr/lib/perl5/5.8.0/i386-linux-thread-multi/IO/Handle.pm line=
=0A> 395,=0A> <STDIN> line 3 (#1)=0A> (W closed) The filehandle you're =
printing on got itself closed=0A> sometime=0A> before now. Check your =
control flow.=0A>=0A> Here's a simplified version of the code from the boss=
thread that appears=0A> to=0A> be causing problems:=0A>=0A> # First remove=
all the servers from the load balancer pool=0A> foreach my $serv (@server_=
stack) {=0A> $result =3D &run_command($expect_connection,"disable $serv=
");=0A> }=0A> # Run some commands against this set of servers now that they=
're out of=0A> the=0A> pool=0A> my @threads =3D ();=0A> while (@server_stac=
k) {=0A> my $server =3D pop(@server_stack);=0A> push @threa=
ds, threads->new(\&shutdown_server,$server);=0A> }=0A> foreach my $thr (@th=
reads) {=0A> @return_vals =3D (@return_vals, $thr->join());=0A> }=0A> f=
oreach my $thr (@threads) {=0A> undef $thr;=0A> }=0A>=0A> Since I'm usi=
ng PERL 5.8.0, I'm undef'ing the threads once they are=0A> finished, since =
there were some issues with this in 5.8.0. This code=0A> block=0A> runs fi=
ne the first time, but as soon as I send it a new @server_stack,=0A> the=0A=
> calls to &run_command no longer work. Here's what run_command looks like=
:=0A>=0A> sub run_command {=0A> my($proc, $command) =3D @_;=0A> my =
$timeout =3D 6;=0A> $proc->raw_pty(1);=0A> $proc->log_stdout(0);=0A=
> $proc->log_file("temp.log");=0A> $proc->debug(0);=0A> $proc->=
send("$command\r");=0A> my $i =3D $proc->expect($timeout, "-re", " # ")=
;=0A> if (!(defined $i)) {=0A> return "ERROR: no response to co=
mmand!\n";=0A> }=0A> return $proc->before();=0A> }=0A>=0A> However,=
if I set $proc->debug(4), it works fine, albeit a bit ugly to=0A> watch. =
It almost seems like something is closing the filehandle that the=0A> Expec=
t connection object is using after the first thread group finishes,=0A> eve=
n though the thread group isn't even operating on the connection=0A> object=
.=0A> I apologize for only posting snippets, but there is too much code to =
post=0A> otherwise. I'm getting this system upgraded to PERL 5.8.8, but it=
's not=0A> going to be anytime soon, so I'm trying to make this work with w=
hat I've=0A> got.=0A>=0A> If anyone has experienced something similar or ha=
s an idea what might fix=0A> this, it would be much appreciated. I've been=
pulling my hair out on this=0A> one for a while now. Thanks for listening=
.=0A>=0A> -Phil=0A> --=0A=0AI've had the same problem, and my fix was the u=
ndef piece you already=0Ahave. However, I'm not sure what state the expect=
objects are in when you=0Astart your new group of threads.=0A=0AWhat has w=
orked for me is this: I setup expect objects on a group of=0Aservers first,=
then exec, then close the expect objects, and finally join=0Athem. Next I=
move on to the next group of servers and do the same.=0A=0AThis is my basi=
c routine:=0A=0A=0Amy $max_threads =3D 4;=0Awhile (@servers || @server_stac=
k) {=0A my $host =3D shift @servers;=0A=0A my $exp =3D function_to_setup_=
expect($host) || next;=0A push(@server_stack, $host);=0A=0A if (@server_s=
tack =3D=3D $max_threads || !@servers) {=0A foreach my $server (@server_=
stack) {=0A push @threads, threads->new(\&shutdown_server,$server);=0A=
}=0A foreach my $thr (@threads) {=0A @return_vals =3D (@return_=
vals, $thr->join());=0A undef $thr;=0A }=0A @server_stack =3D ()=
;=0A @threads =3D ();=0A }=0A}=0A=0ANot saying there isnt something els=
e wrong here, but like i said, this=0Amethod works for me in all 5.8.0 vers=
ions.=0A=0AHope that helps.=0A=0A=0ABryan=0Ahttp://sourceforge.net/projects=
/rover=0A=0A=0A=0A=0A=0A=0A=0A=0A =0A________________________________=
____________________________________________________=0AMoody friends. Drama=
queens. Your life? Nope! - their life, your story. Play Sims Stories at Ya=
hoo! Games.=0Ahttp://sims.yahoo.com/ |
