Menu
▾
▴
expectperl-discuss
|
From: Bryan B. <br...@bu...> - 2007-07-12 17:45:32
|
Austin/Roland,
Ok, there have been a few threads lately about threading so I thought i
would re-visit the issue. I wrote a script that seems to work, but I
would like your opinions on if it really should be working, and/or any
snafu's i may run into if I continue doing it this way.
My script (which I will add to this e-mail) does the following steps in
order:
1. Initialize and store 1 expect object for each host, spawning an SSH
command
2. Get to a command prompt on each expect object, so we have some place to
start
3. Start one thread for each host, sending a list of commands, expecting
the prompt in between each command
4. Close the expect object and return, joining thread
I know it works with at least 5 hosts because i keep separate logs for
each expect session. I'm using Expect v1.20, IO::Tty v1.07, and i've
tried this on two different hosts:
SunOS 5.10 Generic_125101-09 i86pc
Redhat Linux AS4 2.6.9-5.EL
And finally, here is the script:
--- start --->
#!/usr/bin/perl -w
use strict;
use threads;
use Expect;
$Expect::Log_Stdout = 0;
our $prompt = '[>#\$] $';
my @hosts = ("ravager", "jupiter", "europa", "firebee", "invader");
my %hosts_exp;
foreach my $host (@hosts) {
print "Getting expect object for $host\n";
$hosts_exp{$host} = new Expect;
$hosts_exp{$host}->log_file("$host.log");
$hosts_exp{$host}->spawn("ssh -x $host");
$hosts_exp{$host}->expect(5, '-re', $prompt);
}
my $thread_id = 0;
my @thread_ids = ();
my @commands = ("uptime", "who", "who am i", "pwd");
foreach my $host ( keys %hosts_exp ) {
$thread_ids[$thread_id] = threads->new("run_threaded",
$hosts_exp{$host}, $host, @commands);
$thread_id++;
}
for (my $t=0; $t<$thread_id; $t++) {
my $result = $thread_ids[$t]->join();
if ( ! $result ) { print "Thread $t returned bad result\n"; }
}
sub run_threaded {
my $exp = shift;
my $hostname = shift;
my @commands = @_;
my $result = 0;
foreach my $command (@commands) {
print "Executing on $hostname: $command\n";
$exp->send("$command \n");
$result = $exp->expect(5, '-re', $prompt);
last if ! $result;
}
if ($result) {
$exp->send("exit;\n exit;\n exit;\n");
$exp->soft_close();
} else {
$exp->hard_close();
}
return($result);
}
<--- end ---
Thanks,
Bryan Bueter
http://sourceforge.net/projects/rover
|
|
From: Bryan B. <br...@bu...> - 2007-07-15 22:29:49
|
> Austin/Roland,
>
> Ok, there have been a few threads lately about threading so I thought i
> would re-visit the issue. I wrote a script that seems to work, but I
> would like your opinions on if it really should be working, and/or any
> snafu's i may run into if I continue doing it this way.
>
Ok, responding to my own thread... I've been playing with this because
even though the login portion is single threaded, it still outperforms
forking and running in parallel. However, I think i ran into one bug that
I am consistently able to re-produce.
My previous script works fine and is pretty quick to spawn and join the
threads. However, it seems that if I exit and soft_close()/hard_close()
the expect object after the thread is joined, it hangs for a couple of
seconds before joining the first thread.
Here is the script that causes the join() delay. You can compare this to
my previous script to demonstrate what I'm talking about.
--- start --->
#!/usr/bin/perl -w
use strict;
use threads;
use Expect;
$Expect::Log_Stdout = 0;
our $prompt = '[>#\$] $';
my @hosts = ("mythtv", "valhalla", "eldorado" );
my %hosts_exp;
foreach my $host (@hosts) {
print "Getting expect object for $host\n";
$hosts_exp{$host} = new Expect;
$hosts_exp{$host}->log_file("$host.log");
$hosts_exp{$host}->spawn("ssh -x $host");
$hosts_exp{$host}->expect(5, '-re', $prompt);
}
my $thread_id = 0;
my @thread_ids = ();
my @thread_hosts;
my @commands = ("uptime", "who", "who am i", "pwd");
foreach my $host ( keys %hosts_exp ) {
$thread_ids[$thread_id] = threads->new("run_threaded",
$hosts_exp{$host}, $host, @commands);
$thread_hosts[$thread_id] = $host;
$thread_id++;
}
for (my $t=0; $t<$thread_id; $t++) {
print "Joining thread $t (". $thread_hosts[$t] .")\n";
my $result = $thread_ids[$t]->join();
print "Completed joining thread $t (". $thread_hosts[$t] .")\n";
if ($result) {
$hosts_exp{$thread_hosts[$t]}->send("exit;\n exit;\n exit;\n");
$hosts_exp{$thread_hosts[$t]}->soft_close();
} else {
print "Thread $t returned bad result\n";
$hosts_exp{$thread_hosts[$t]}->hard_close();
}
}
sub run_threaded {
my $exp = shift;
my $hostname = shift;
my @commands = @_;
my $result = 0;
foreach my $command (@commands) {
print "Executing on $hostname: $command\n";
$exp->send("$command \n");
$result = $exp->expect(5, '-re', $prompt);
last if ! $result;
}
return($result);
}
<--- end ---
Thanks, please let me know what you think.
Bryan Bueter
http://sourceforge.net/projects/rover
|
|
From: Roland G. <rgi...@cp...> - 2007-07-16 10:38:28
|
Hmm, could be the wait delay in soft/hard_close which is trying to
send signals to the spawned process via "kill". Maybe that also
doesn't work in a thread (like fork()). Could you try closing the
Expect objects in the main thread? Maybe that gets rid of the delay...
Seems a sensible rule to me: "Create and destroy Expect objects only
in the main thread."
BTW, anybody tried to use Net::SSH2 instead of spawning ssh? Net::SSH2
objects are tied filehandles, so they should be usable via
exp_init()...
Cheers!
Roland
On 7/16/07, Bryan Bueter <br...@bu...> wrote:
> > Austin/Roland,
> >
> > Ok, there have been a few threads lately about threading so I thought i
> > would re-visit the issue. I wrote a script that seems to work, but I
> > would like your opinions on if it really should be working, and/or any
> > snafu's i may run into if I continue doing it this way.
> >
>
> Ok, responding to my own thread... I've been playing with this because
> even though the login portion is single threaded, it still outperforms
> forking and running in parallel. However, I think i ran into one bug that
> I am consistently able to re-produce.
>
> My previous script works fine and is pretty quick to spawn and join the
> threads. However, it seems that if I exit and soft_close()/hard_close()
> the expect object after the thread is joined, it hangs for a couple of
> seconds before joining the first thread.
>
> Here is the script that causes the join() delay. You can compare this to
> my previous script to demonstrate what I'm talking about.
>
> --- start --->
> #!/usr/bin/perl -w
>
> use strict;
> use threads;
>
> use Expect;
> $Expect::Log_Stdout = 0;
>
> our $prompt = '[>#\$] $';
> my @hosts = ("mythtv", "valhalla", "eldorado" );
> my %hosts_exp;
>
> foreach my $host (@hosts) {
> print "Getting expect object for $host\n";
> $hosts_exp{$host} = new Expect;
>
> $hosts_exp{$host}->log_file("$host.log");
> $hosts_exp{$host}->spawn("ssh -x $host");
> $hosts_exp{$host}->expect(5, '-re', $prompt);
> }
>
> my $thread_id = 0;
> my @thread_ids = ();
> my @thread_hosts;
> my @commands = ("uptime", "who", "who am i", "pwd");
> foreach my $host ( keys %hosts_exp ) {
> $thread_ids[$thread_id] = threads->new("run_threaded",
> $hosts_exp{$host}, $host, @commands);
> $thread_hosts[$thread_id] = $host;
>
> $thread_id++;
> }
>
> for (my $t=0; $t<$thread_id; $t++) {
> print "Joining thread $t (". $thread_hosts[$t] .")\n";
> my $result = $thread_ids[$t]->join();
> print "Completed joining thread $t (". $thread_hosts[$t] .")\n";
>
> if ($result) {
> $hosts_exp{$thread_hosts[$t]}->send("exit;\n exit;\n exit;\n");
> $hosts_exp{$thread_hosts[$t]}->soft_close();
> } else {
> print "Thread $t returned bad result\n";
> $hosts_exp{$thread_hosts[$t]}->hard_close();
> }
>
> }
>
> sub run_threaded {
> my $exp = shift;
> my $hostname = shift;
> my @commands = @_;
>
> my $result = 0;
> foreach my $command (@commands) {
> print "Executing on $hostname: $command\n";
> $exp->send("$command \n");
> $result = $exp->expect(5, '-re', $prompt);
> last if ! $result;
> }
>
> return($result);
> }
> <--- end ---
>
> Thanks, please let me know what you think.
>
>
> Bryan Bueter
> http://sourceforge.net/projects/rover
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Expectperl-discuss mailing list
> Exp...@li...
> https://lists.sourceforge.net/lists/listinfo/expectperl-discuss
>
|
|
From: Austin S. <te...@of...> - 2007-07-16 17:08:54
|
On Mon, Jul 16, 2007 at 12:38:24PM +0200, Roland Giersig wrote: > Hmm, could be the wait delay in soft/hard_close which is trying to > send signals to the spawned process via "kill". Maybe that also > doesn't work in a thread (like fork()). Could you try closing the > Expect objects in the main thread? Maybe that gets rid of the delay... > soft_close (e.g.) will read from the process until it returns EOF, then wait on the process until it dies. This is typically considered responsible behavior because it reduces the possibility of leaving zombies around. My question would be "what is ssh doing instead of exiting?" Maybe truss/strace would answer that question. Austin |
|
From: Bryan B. <br...@bu...> - 2007-07-16 11:40:11
|
> Hmm, could be the wait delay in soft/hard_close which is trying to > send signals to the spawned process via "kill". Maybe that also > doesn't work in a thread (like fork()). Could you try closing the > Expect objects in the main thread? Maybe that gets rid of the delay... > > Seems a sensible rule to me: "Create and destroy Expect objects only > in the main thread." > Actually its the other way around. If i soft/hard close the object in the thread it takes a reasonable amount of time. If I soft/hard close it in the main thread it takes longer. The actual delay is in the join function, then closing the objects works normally. I re-wrote the last script to output the number of seconds elapsed and here is the output: Getting expect object for mythtv: 0 seconds Getting expect object for valhalla: 0 seconds Getting expect object for eldorado: 0 seconds Spawning thread: 1 seconds Spawning thread: 1 seconds Spawning thread: 1 seconds Joining thread 0 (mythtv): 1 seconds Completed joining thread 0 (mythtv): 16 seconds Closing expect object: 16 seconds Joining thread 1 (eldorado): 16 seconds Completed joining thread 1 (eldorado): 16 seconds Closing expect object: 16 seconds Joining thread 2 (valhalla): 16 seconds Completed joining thread 2 (valhalla): 16 seconds Closing expect object: 16 seconds You can see it takes about 15 seconds to join the first thread and then things progress normally. > > BTW, anybody tried to use Net::SSH2 instead of spawning ssh? Net::SSH2 > objects are tied filehandles, so they should be usable via > exp_init()... > > Cheers! > Roland > I persued Net::SSH2 over a year ago for just this purpose. You can read the thread I had with the author here: http://sourceforge.net/mailarchive/message.php?msg_id=200601192208.43932.dbrobins%40cpan.org Basically David Robins stated this: Ah, then there's the problem. Fileno tries to get a real Unix filehandle out of the object; since it isn't a real Unix file, you won't get one. I could not get it to work, and David seemed to think that it should not work. In that thread there is very simple code that I wrote and even that doesnt work. However, all that aside, even without using Expect, Net::SSH2 was not thread safe. I sent emails regarding that as well, to no avail. At this point in time, I really think Expect on top of a spawned ssh command is the best way to do things (from a perl perspective). Thanks Bryan Bueter http://sourceforge.net/projects/rover |
|
From: Roland G. <rgi...@cp...> - 2007-07-18 12:19:02
|
On 7/16/07, Bryan Bueter <br...@bu...> wrote: > I persued Net::SSH2 over a year ago for just this purpose. You can read > the thread I had with the author here: > > http://sourceforge.net/mailarchive/message.php?msg_id=200601192208.43932.dbrobins%40cpan.org > > Basically David Robins stated this: > > Ah, then there's the problem. Fileno tries to get a real Unix filehandle out > of the object; since it isn't a real Unix file, you won't get one. > > I could not get it to work, and David seemed to think that it should not > work. In that thread there is very simple code that I wrote and even that > doesnt work. I think all that would be needed is that Net::SSH2::Channel returns the fileno of the socket connection. Expect needs a real fileno to do a select() on it, but everything else is done via the object. Ideally, Net::SSH2 should be rewritten as an I/O filter ala PerlIO::gzip. Well, one may dream... :-) > However, all that aside, even without using Expect, Net::SSH2 was not > thread safe. I sent emails regarding that as well, to no avail. At this > point in time, I really think Expect on top of a spawned ssh command is > the best way to do things (from a perl perspective). I browsed through the source and didn't notice anything obvious that could hinder thread-usage. Can you remember what it was that was not thread-safe? Yeah, let's stick to ssh for now, nothing wrong with that. But please, people, remember to not use password-authentication... Best regards, Roland |
|
From: Bryan B. <br...@bu...> - 2007-07-18 14:31:28
|
> On 7/16/07, Bryan Bueter <br...@bu...> wrote: >> I persued Net::SSH2 over a year ago for just this purpose. You can read >> the thread I had with the author here: >> >> http://sourceforge.net/mailarchive/message.php?msg_id=200601192208.43932.dbrobins%40cpan.org >> >> Basically David Robins stated this: >> >> Ah, then there's the problem. Fileno tries to get a real Unix filehandle >> out >> of the object; since it isn't a real Unix file, you won't get one. >> >> I could not get it to work, and David seemed to think that it should not >> work. In that thread there is very simple code that I wrote and even >> that >> doesnt work. > > I think all that would be needed is that Net::SSH2::Channel returns > the fileno of the socket connection. Expect needs a real fileno to do > a select() on it, but everything else is done via the object. > > Ideally, Net::SSH2 should be rewritten as an I/O filter ala > PerlIO::gzip. Well, one may dream... :-) > >> However, all that aside, even without using Expect, Net::SSH2 was not >> thread safe. I sent emails regarding that as well, to no avail. At >> this >> point in time, I really think Expect on top of a spawned ssh command is >> the best way to do things (from a perl perspective). > > I browsed through the source and didn't notice anything obvious that > could hinder thread-usage. Can you remember what it was that was not > thread-safe? > > Yeah, let's stick to ssh for now, nothing wrong with that. But please, > people, remember to not use password-authentication... > > Best regards, Roland > If I remember correctly, when i tried threading with Net::SSH2 in the past (v0.06), it would connect to multiple sessions and then hang when reading from those channels simultaneously. However, now I cant even get the latest version (v0.10) to work in single threaded mode. So for the moment i'm not going to waste any more time on it. Thanks, Bryan Bueter http://sourceforge.net/projects/rover |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X