Thread: [JSch-users] SCP to SFTP file streaming
Status: Alpha
Brought to you by:
ymnk
Menu
▾
▴
jsch-users
|
From: Sensen, A. <a.s...@th...> - 2019-04-12 06:59:49
|
Hi all,
first i'd like to thank JCraft for the JSch library. It has been really helpful so far!
But during my development i ran into a problem: trying to copy a file with SCP to an SFTP-Server
using the Streams provided by JSch. The error is easily reproducable in Version 0.1.55 by doing
the following steps:
* Retrieve an InputStream (MyPipedInputStream) from a Channel
* Try to upload this file with an ChannelSftp:
--------
JSch jsch = new JSch();
Session session = jsch.getSession(username, host, port);
ChannelExec channel = (ChannelExec) session.openChannel("exec");
...
InputStream sshInputStream = channel.getInputStream();
JSch jsch = new JSch();
Session session = jsch.getSession(username, host, port);
ChannelSftp channel = (ChannelSftp) session.openChannel("sftp");
channel.put(sshInputStream , file);
--------
The problem lies in ChannelSftp's _put method (lines 635 to 643):
--------
do{
nread=src.read(data, s, datalen);
if(nread>0){
s+=nread;
datalen-=nread;
count+=nread;
}
}
while(datalen>0 && nread>0);
--------
As far as i understand it, the while-loop tries to read as many bytes as possible into the data
array. But since a (My)PipedInputStream doesn't return -1 if it's empty, but rather waits for more
data to come, this ends in an endlessly blocking read() at the end of the file transfer.
The only way i could solve this is by removing the do-while-loop around the code and adding an
SftpProgressMonitor implementation, that returns false on count(long byte) if the whole file has
been transferred. But that involves manipulating JSch source code, which i'd prefer not to do...
Is there any better solution to this particular problem?
Any help would be greatly appreciated!
Thanks
Andreas
|
|
From: Lothar K. <jo...@ki...> - 2019-04-12 10:30:35
|
Am 12.04.2019 um 08:42 schrieb Sensen, Andreas:
> ChannelExec channel = (ChannelExec) session.openChannel("exec");
>
> …
that part would be interesting as well in order to give an answer.
> InputStream sshInputStream = channel.getInputStream();
Aftersending a "scp -f filename" the scp-command starts with an information
about the lenght of the file to be transfered. You should read that and keep
that information for the next step.
Here you should wrap the returned inputstream into an own InputStream
that uses the above size-information to return -1 as soon as the announced
number of bytes has been reached.
Cheers, Lothar
|
|
From: Jung, V. <vol...@so...> - 2019-04-12 12:21:18
Attachments:
smime.p7s
|
Hi Andreas,
as far as I understand, you are trying to stream data received by the SCP 'protocol' directly to a target host via SFTP.
The streams of the EXEC-channel are open as long as the EXEC channel is not closed. However, your implementation seems to assume that the InputStream is closed as soon as the data of the file is fully received via SCP. This is not the case and looks to me like a misunderstanding on your side.
I think you have to implement the SCP specific details yourself, meaning that you have to provide a new InputStream for SFTP-upload which is closed upon receiving the end-transmission-token via SCP.
Sincerely yours
Volker
Von: Sensen, Andreas <a.s...@th...>
Gesendet: Freitag, 12. April 2019 08:42
An: 'jsc...@li...' <jsc...@li...>
Betreff: [JSch-users] SCP to SFTP file streaming
Hi all,
first i'd like to thank JCraft for the JSch library. It has been really helpful so far!
But during my development i ran into a problem: trying to copy a file with SCP to an SFTP-Server
using the Streams provided by JSch. The error is easily reproducable in Version 0.1.55 by doing
the following steps:
* Retrieve an InputStream (MyPipedInputStream) from a Channel
* Try to upload this file with an ChannelSftp:
--------
JSch jsch = new JSch();
Session session = jsch.getSession(username, host, port);
ChannelExec channel = (ChannelExec) session.openChannel("exec");
...
InputStream sshInputStream = channel.getInputStream();
JSch jsch = new JSch();
Session session = jsch.getSession(username, host, port);
ChannelSftp channel = (ChannelSftp) session.openChannel("sftp");
channel.put(sshInputStream , file);
--------
The problem lies in ChannelSftp's _put method (lines 635 to 643):
--------
do{
nread=src.read(data, s, datalen);
if(nread>0){
s+=nread;
datalen-=nread;
count+=nread;
}
}
while(datalen>0 && nread>0);
--------
As far as i understand it, the while-loop tries to read as many bytes as possible into the data
array. But since a (My)PipedInputStream doesn't return -1 if it's empty, but rather waits for more
data to come, this ends in an endlessly blocking read() at the end of the file transfer.
The only way i could solve this is by removing the do-while-loop around the code and adding an
SftpProgressMonitor implementation, that returns false on count(long byte) if the whole file has
been transferred. But that involves manipulating JSch source code, which i'd prefer not to do...
Is there any better solution to this particular problem?
Any help would be greatly appreciated!
Thanks
Andreas
|
|
From: Sensen, A. <a.s...@th...> - 2019-04-12 17:50:59
|
Dear Lothar and Volker,
thanks for your quick reply. I indeed misunderstood how the InputStream should be handled.
I implemented a wrapper that counts the transferred bytes and returns -1 when the file has
been transferred completely, just as Lothar suggested.
Thank you very much for your help!
Andreas
________________________________
Von: Jung, Volker [vol...@so...]
Gesendet: Freitag, 12. April 2019 13:58
An: 'jsc...@li...'
Betreff: Re: [JSch-users] SCP to SFTP file streaming
Hi Andreas,
as far as I understand, you are trying to stream data received by the SCP ‘protocol’ directly to a target host via SFTP.
The streams of the EXEC-channel are open as long as the EXEC channel is not closed. However, your implementation seems to assume that the InputStream is closed as soon as the data of the file is fully received via SCP. This is not the case and looks to me like a misunderstanding on your side.
I think you have to implement the SCP specific details yourself, meaning that you have to provide a new InputStream for SFTP-upload which is closed upon receiving the end-transmission-token via SCP.
Sincerely yours
Volker
Von: Sensen, Andreas <a.s...@th...>
Gesendet: Freitag, 12. April 2019 08:42
An: 'jsc...@li...' <jsc...@li...>
Betreff: [JSch-users] SCP to SFTP file streaming
Hi all,
first i’d like to thank JCraft for the JSch library. It has been really helpful so far!
But during my development i ran into a problem: trying to copy a file with SCP to an SFTP-Server
using the Streams provided by JSch. The error is easily reproducable in Version 0.1.55 by doing
the following steps:
* Retrieve an InputStream (MyPipedInputStream) from a Channel
* Try to upload this file with an ChannelSftp:
--------
JSch jsch = new JSch();
Session session = jsch.getSession(username, host, port);
ChannelExec channel = (ChannelExec) session.openChannel("exec");
…
InputStream sshInputStream = channel.getInputStream();
JSch jsch = new JSch();
Session session = jsch.getSession(username, host, port);
ChannelSftp channel = (ChannelSftp) session.openChannel("sftp");
channel.put(sshInputStream , file);
--------
The problem lies in ChannelSftp's _put method (lines 635 to 643):
--------
do{
nread=src.read(data, s, datalen);
if(nread>0){
s+=nread;
datalen-=nread;
count+=nread;
}
}
while(datalen>0 && nread>0);
--------
As far as i understand it, the while-loop tries to read as many bytes as possible into the data
array. But since a (My)PipedInputStream doesn't return -1 if it's empty, but rather waits for more
data to come, this ends in an endlessly blocking read() at the end of the file transfer.
The only way i could solve this is by removing the do-while-loop around the code and adding an
SftpProgressMonitor implementation, that returns false on count(long byte) if the whole file has
been transferred. But that involves manipulating JSch source code, which i'd prefer not to do...
Is there any better solution to this particular problem?
Any help would be greatly appreciated!
Thanks
Andreas
|
Thanks for helping keep SourceForge clean.
X