ssh-sftp-perl-users

[Ssh-sftp-perl-users] Problem getting output using Net::SSH2

[Steve P. <st...@fo...>]

Hey There,

I'm busy pulling my hair out over this one. I have a small script 
similar to the below.

my $ssh2 = Net::SSH2->new();
$ssh2->debug(1);
$ssh2->connect($server) or die "Unable to connect Host $@ \n";
$ssh2->auth_publickey($userName,$publicKey,$privateKey);

my $chan = $ssh2->channel();
$chan->blocking(0);
$chan->shell();

This seems to work quite happily and a connection is established, 
however - when running small commands (ls -l, whoami etc) everything 
works as expected

print $chan "ls -la\n";
while (<$chan>) { print "LINE: " . $_ }

But, when I run a command that takes a while to return, I simply get a 
blank output.

print $chan "vip/listmse /WideSpan/config/vipclient.conf 1 -c 'RADIUS 
Server'\n";
while (<$chan>) { print "LINE: " . $_ }

The only thing I can think of is that the second command seems to take a 
few miliseconds longer to produce output.

I've tried to play around with "$ssh2->poll(0)" which I read somewhere 
sets polling to infinite, but still no results.

When debug is enabled I get thousands of lines similar to

Net::SSH2::Channel::read(size = 1, ext = 0)
- read 1 bytes
- read 1 total
Net::SSH2::poll: timeout = 250, array[1]
- [0] = channel
- [0] events 1
- libssh2_poll returned 1
- [0] revents 1

scrolling up the screen, which I would assume means that I have exceeded 
a timeout of 250ms ?

I've tried setting blocking to 0 and 1 with no joy either. (I've found 
that setting blocking to 0 when performing shell commands can lead to 
interesting output tho, so I assume that interactive shell commands 
should leave blocking to 1 ? setting to 0 seems to have interesting 
results when trying to run commands like ls -la)

If I manually ssh into the remote box and run the command it returns 
values as expected, and if I do something similar using Net::SSH::Perl 
it works as well, however, Net::SSH::Perl seems to take upward of 30 
seconds to create a connection which isn't an option in this environment 
(and yeah - I've tried all the math module stuff which didn't seem to 
make a difference :-( )

Any pointers as to why small/fast commands would work but commands that 
take around 1/2 a second to run would drop out would be appreciated.

-- 
Steve.

Re: [Ssh-sftp-perl-users] Problem getting output using Net::SSH2

[Fitzpatrick, R. M (Rob) <rob...@ve...>]

 First, can you confirm that it is a simple timing issue, and not
something about the input or output of your command?  What if you just
run a "sleep 1", does that cause your issue?  Sleep 10?

That will also give the rest of us a replicatable test situation if we
can't resolve this quickly.

Best of luck,
  Rob

-----Original Message-----
From: Steve Phillips [mailto:st...@fo...] 
Sent: Wednesday, May 04, 2011 8:54 PM
To: ssh...@li...
Subject: [Ssh-sftp-perl-users] Problem getting output using Net::SSH2

Hey There,

I'm busy pulling my hair out over this one. I have a small script
similar to the below.

my $ssh2 = Net::SSH2->new();
$ssh2->debug(1);
$ssh2->connect($server) or die "Unable to connect Host $@ \n";
$ssh2->auth_publickey($userName,$publicKey,$privateKey);

my $chan = $ssh2->channel();
$chan->blocking(0);
$chan->shell();

This seems to work quite happily and a connection is established,
however - when running small commands (ls -l, whoami etc) everything
works as expected

print $chan "ls -la\n";
while (<$chan>) { print "LINE: " . $_ }

But, when I run a command that takes a while to return, I simply get a
blank output.

print $chan "vip/listmse /WideSpan/config/vipclient.conf 1 -c 'RADIUS
Server'\n"; while (<$chan>) { print "LINE: " . $_ }

The only thing I can think of is that the second command seems to take a
few miliseconds longer to produce output.

I've tried to play around with "$ssh2->poll(0)" which I read somewhere
sets polling to infinite, but still no results.

When debug is enabled I get thousands of lines similar to

Net::SSH2::Channel::read(size = 1, ext = 0)
- read 1 bytes
- read 1 total
Net::SSH2::poll: timeout = 250, array[1]
- [0] = channel
- [0] events 1
- libssh2_poll returned 1
- [0] revents 1

scrolling up the screen, which I would assume means that I have exceeded
a timeout of 250ms ?

I've tried setting blocking to 0 and 1 with no joy either. (I've found
that setting blocking to 0 when performing shell commands can lead to
interesting output tho, so I assume that interactive shell commands
should leave blocking to 1 ? setting to 0 seems to have interesting
results when trying to run commands like ls -la)

If I manually ssh into the remote box and run the command it returns
values as expected, and if I do something similar using Net::SSH::Perl
it works as well, however, Net::SSH::Perl seems to take upward of 30
seconds to create a connection which isn't an option in this environment
(and yeah - I've tried all the math module stuff which didn't seem to
make a difference :-( )

Any pointers as to why small/fast commands would work but commands that
take around 1/2 a second to run would drop out would be appreciated.

--
Steve.

Re: [Ssh-sftp-perl-users] Problem getting output using Net::SSH2

[Steve P. <st...@fo...>]

Hmm, the sleep got it to work.

Does this mean that 'blocking' being set to 1 doesn't actually mean that 
the command executed by the shell blocks until it returns ? or is 
blocking set to 1 or 0 something else entirely ?

Which I guess means that the next question is 'how do you get it to wait 
until output returns' for a command that could return in an indefinite 
amount of time ? (without the obvious of lowest common denominator and 
getting all commands to wait for 20 seconds or so)

Also, thanks for the speedy response :-)

-- 
Steve.

On 5/05/2011 3:30 PM, Fitzpatrick, Robert M (Rob) wrote:
>
>   First, can you confirm that it is a simple timing issue, and not
> something about the input or output of your command?  What if you just
> run a "sleep 1", does that cause your issue?  Sleep 10?
>
> That will also give the rest of us a replicatable test situation if we
> can't resolve this quickly.
>
> Best of luck,
>    Rob
>
> -----Original Message-----
> From: Steve Phillips [mailto:st...@fo...]
> Sent: Wednesday, May 04, 2011 8:54 PM
> To: ssh...@li...
> Subject: [Ssh-sftp-perl-users] Problem getting output using Net::SSH2
>
> Hey There,
>
> I'm busy pulling my hair out over this one. I have a small script
> similar to the below.
>
> my $ssh2 = Net::SSH2->new();
> $ssh2->debug(1);
> $ssh2->connect($server) or die "Unable to connect Host $@ \n";
> $ssh2->auth_publickey($userName,$publicKey,$privateKey);
>
> my $chan = $ssh2->channel();
> $chan->blocking(0);
> $chan->shell();
>
> This seems to work quite happily and a connection is established,
> however - when running small commands (ls -l, whoami etc) everything
> works as expected
>
> print $chan "ls -la\n";
> while (<$chan>) { print "LINE: " . $_ }
>
> But, when I run a command that takes a while to return, I simply get a
> blank output.
>
> print $chan "vip/listmse /WideSpan/config/vipclient.conf 1 -c 'RADIUS
> Server'\n"; while (<$chan>) { print "LINE: " . $_ }
>
> The only thing I can think of is that the second command seems to take a
> few miliseconds longer to produce output.
>
> I've tried to play around with "$ssh2->poll(0)" which I read somewhere
> sets polling to infinite, but still no results.
>
> When debug is enabled I get thousands of lines similar to
>
> Net::SSH2::Channel::read(size = 1, ext = 0)
> - read 1 bytes
> - read 1 total
> Net::SSH2::poll: timeout = 250, array[1]
> - [0] = channel
> - [0] events 1
> - libssh2_poll returned 1
> - [0] revents 1
>
> scrolling up the screen, which I would assume means that I have exceeded
> a timeout of 250ms ?
>
> I've tried setting blocking to 0 and 1 with no joy either. (I've found
> that setting blocking to 0 when performing shell commands can lead to
> interesting output tho, so I assume that interactive shell commands
> should leave blocking to 1 ? setting to 0 seems to have interesting
> results when trying to run commands like ls -la)
>
> If I manually ssh into the remote box and run the command it returns
> values as expected, and if I do something similar using Net::SSH::Perl
> it works as well, however, Net::SSH::Perl seems to take upward of 30
> seconds to create a connection which isn't an option in this environment
> (and yeah - I've tried all the math module stuff which didn't seem to
> make a difference :-( )
>
> Any pointers as to why small/fast commands would work but commands that
> take around 1/2 a second to run would drop out would be appreciated.
>
> --
> Steve.
>

Re: [Ssh-sftp-perl-users] Problem getting output using Net::SSH2

[Thierry C. <thi...@gm...>]

Le 05/05/2011 07:30, Fitzpatrick, Robert M (Rob) a écrit :
> If I manually ssh into the remote box and run the command it returns
> values as expected, and if I do something similar using Net::SSH::Perl
> it works as well, however, Net::SSH::Perl seems to take upward of 30
> seconds to create a connection which isn't an option in this environment
> (and yeah - I've tried all the math module stuff which didn't seem to
> make a difference:-(  )
Hello,

There is a very good alternative to Net::SSH::Perl. It is Net::OpenSSH. 
It is using openssh, but it is a very good implantation.  It is working 
well, it implement scp, sftp, channel.
But I am quite sure that connecting in 30 second  is not a normal 
behavior. The dependancies of  Net::SSH::Perl are painfull. I never 
remember if it is Math::BigInt that accelerate the whole thing, or a 
GMP::*. I see that you think that you have installed all this stuff, but 
I think you should check this an other time.

Thierry

Re: [Ssh-sftp-perl-users] Problem getting output using Net::SSH2

[David R. <Dav...@mi...>]

The problem with waiting for responses from a command shell channel is that it's no different from any other channel and one cannot intrinsically tell the difference between no response due to a command being busy or slow and no response due to the output being finished. The 250ms timeout is in Net::SSH2::Channel::GETC (READLINE depends on GETC, and READLINE is what is implementing <>). You can get more control over reading from the channel with the read method - but less convenience. Similarly you can use the Net::SSH2::poll method on your own to check for pending events (and give it a timeout), again, at a cost of convenience.

To avoid requiring a 1 second sleep before getting each line, you could poll the channel with a 1000ms timeout and then read a line if it indicates data being ready; this means you only wait a whole second in the end of output case (assuming 1000 ms is long enough for your command). To eliminate that last case you could either exec a single command (see Net::SSH2::Channel::exec) which should terminate the poll invocation when the command exits, or send something like "cmd; exit" to the shell channel to have it exit after the command runs.

________________________________________
From: Steve Phillips [st...@fo...]
Sent: Thursday, May 05, 2011 6:24 PM
Cc: ssh...@li...
Subject: Re: [Ssh-sftp-perl-users] Problem getting output using Net::SSH2

On 5/05/2011 5:43 PM, Thierry Chich wrote:
> Le 05/05/2011 07:30, Fitzpatrick, Robert M (Rob) a écrit :
>> If I manually ssh into the remote box and run the command it returns
>> values as expected, and if I do something similar using Net::SSH::Perl
>> it works as well, however, Net::SSH::Perl seems to take upward of 30
>> seconds to create a connection which isn't an option in this environment
>> (and yeah - I've tried all the math module stuff which didn't seem to
>> make a difference:-(  )
> Hello,
>
> There is a very good alternative to Net::SSH::Perl. It is Net::OpenSSH.
> It is using openssh, but it is a very good implantation.  It is working
> well, it implement scp, sftp, channel.
> But I am quite sure that connecting in 30 second  is not a normal
> behavior. The dependancies of  Net::SSH::Perl are painfull. I never
> remember if it is Math::BigInt that accelerate the whole thing, or a
> GMP::*. I see that you think that you have installed all this stuff, but
> I think you should check this an other time.

Yeah, I spent quite a bit of time trying to figure out why this took so
long. I had the Math::BigInt::GMP module installed - which people
advised would speed things up and for some reason it took even longer (I
gave up waiting after around 1:30 per connection) and also had Math::GMP
installed with no discernible improvements.

I had decided that it was possibly a version problem (this was running
on a Redhat 9 box, don't ask) and so put a new, more powerful system in
(CentOS 5, with an i7 CPU) and re-installed Net::SSH::Perl along with
most dependencies (I think I was missing one of the Crypt modules that
caused it to burp at private key auth using DSA) but the connection side
of things still took over 30 seconds. (this is the initial key exchange
sequence that seems to take this long)

All this was using pre compiled packages as well, as compiling isn't
really an option due to the system it's going to end up on (well, it
COULD be but would require a lot more effort than simply switching to
something that works out of the box :-) )

At that point I gave up and found Net::SSH2 which using the compiled
library is stupidly fast in comparison, but now have this output problem
that is semi solved (thanks Rob) and just have to figure out how to tell
when there is data waiting to be read off the filehandle so I don't have
to use random sleep timers.

At this point I'm contemplating a loop using usleep but this seems like
a bit of a hack - when spawning a shell to run your commands, is there
really no way to block until the command has exited or do you have to
find other ways of figuring out the exit state of the last passed
command (like running $command and then a 'whoami' and read input until
you come across the username on a single line - seems a little much)

--
Steve.

Re: [Ssh-sftp-perl-users] Problem getting output using Net::SSH2

[Steve P. <st...@fo...>]

On 6/05/2011 12:55 PM, David Robins wrote:
> The problem with waiting for responses from a command shell channel is that it's no different from any other channel and one cannot intrinsically tell the difference between no response due to a command being busy or slow and no response due to the output being finished. The 250ms timeout is in Net::SSH2::Channel::GETC (READLINE depends on GETC, and READLINE is what is implementing<>). You can get more control over reading from the channel with the read method - but less convenience. Similarly you can use the Net::SSH2::poll method on your own to check for pending events (and give it a timeout), again, at a cost of convenience.
>
> To avoid requiring a 1 second sleep before getting each line, you could poll the channel with a 1000ms timeout and then read a line if it indicates data being ready; this means you only wait a whole second in the end of output case (assuming 1000 ms is long enough for your command). To eliminate that last case you could either exec a single command (see Net::SSH2::Channel::exec) which should terminate the poll invocation when the command exits, or send something like "cmd; exit" to the shell channel to have it exit after the command runs.

Thanks for the replies,

I've since implemented a while loop that waits for the first line of 
output then continues for the rest, this also lets me implement a 
timeout where the command takes an inordinate amount of time I can exit 
the script with a timeout error as well.

I'll look more into the Net::SSH2::Channel:Exec stuff as well as that 
looks quite promising.

Thanks,

-- 
Steve.

Re: [Ssh-sftp-perl-users] Problem getting output using Net::SSH2

[Steve P. <st...@fo...>]

On 5/05/2011 5:43 PM, Thierry Chich wrote:
> Le 05/05/2011 07:30, Fitzpatrick, Robert M (Rob) a écrit :
>> If I manually ssh into the remote box and run the command it returns
>> values as expected, and if I do something similar using Net::SSH::Perl
>> it works as well, however, Net::SSH::Perl seems to take upward of 30
>> seconds to create a connection which isn't an option in this environment
>> (and yeah - I've tried all the math module stuff which didn't seem to
>> make a difference:-(  )
> Hello,
>
> There is a very good alternative to Net::SSH::Perl. It is Net::OpenSSH.
> It is using openssh, but it is a very good implantation.  It is working
> well, it implement scp, sftp, channel.
> But I am quite sure that connecting in 30 second  is not a normal
> behavior. The dependancies of  Net::SSH::Perl are painfull. I never
> remember if it is Math::BigInt that accelerate the whole thing, or a
> GMP::*. I see that you think that you have installed all this stuff, but
> I think you should check this an other time.

Yeah, I spent quite a bit of time trying to figure out why this took so 
long. I had the Math::BigInt::GMP module installed - which people 
advised would speed things up and for some reason it took even longer (I 
gave up waiting after around 1:30 per connection) and also had Math::GMP 
installed with no discernible improvements.

I had decided that it was possibly a version problem (this was running 
on a Redhat 9 box, don't ask) and so put a new, more powerful system in 
(CentOS 5, with an i7 CPU) and re-installed Net::SSH::Perl along with 
most dependencies (I think I was missing one of the Crypt modules that 
caused it to burp at private key auth using DSA) but the connection side 
of things still took over 30 seconds. (this is the initial key exchange 
sequence that seems to take this long)

All this was using pre compiled packages as well, as compiling isn't 
really an option due to the system it's going to end up on (well, it 
COULD be but would require a lot more effort than simply switching to 
something that works out of the box :-) )

At that point I gave up and found Net::SSH2 which using the compiled 
library is stupidly fast in comparison, but now have this output problem 
that is semi solved (thanks Rob) and just have to figure out how to tell 
when there is data waiting to be read off the filehandle so I don't have 
to use random sleep timers.

At this point I'm contemplating a loop using usleep but this seems like 
a bit of a hack - when spawning a shell to run your commands, is there 
really no way to block until the command has exited or do you have to 
find other ways of figuring out the exit state of the last passed 
command (like running $command and then a 'whoami' and read input until 
you come across the username on a single line - seems a little much)

-- 
Steve.