Thread: [JSch-users] Transfering large number of small files
Status: Alpha
Brought to you by:
ymnk
From: Luís L. <lf...@gm...> - 2016-07-13 21:59:34
Attachments:
signature.asc
|
Hi all. I'm trying to transfer a large number of small files to a remote server (with reasonable latency) using JSch. The code I'm using creates a Session and a ChannelSftp and then iterates over the local files (some times thousands) to transfer them to the server, with something like (I omitted exception handling for simplicity): Session session=...; ChannelSftp sftp=(ChannelSftp)session.openChannel("sftp"); for(Path path:...){ sftp.put(path.toString(), String.format("/destination/%s",path.getFileName().toString())); } sftp.disconnect(); session.disconnect(); The problem is this is very slow (sometimes it takes hours) compared with the something like: rsync -avu /local/<some glob> server:/destination/ # takes seconds Is there any faster way to transfer large number of small files with JSch? Regards, LL |
From: Leonardo K. S. <sh...@gm...> - 2016-07-14 02:07:57
|
you can't compare sftp with rsync rsync calculates the incremental changes it must send over the wire, sftp will always send the whole file, no matter it's overwriting the same file on the other side you can compare jsch with native sftp, but I believe sftp will still be faster for obvious reasons [] Leo On Wed, Jul 13, 2016 at 6:59 PM, Luís Lobo <lf...@gm...> wrote: > Hi all. > > I'm trying to transfer a large number of small files to a remote server > (with reasonable latency) using JSch. > > The code I'm using creates a Session and a ChannelSftp and then iterates > over the local files (some times thousands) to transfer them to the server, > with something like (I omitted exception handling for simplicity): > > Session session=...; > ChannelSftp sftp=(ChannelSftp)session.openChannel("sftp"); > for(Path path:...){ > sftp.put(path.toString(), String.format("/destination/%s",path.getFileName().toString())); > } > sftp.disconnect(); > session.disconnect(); > > > The problem is this is very slow (sometimes *it takes hours*) compared > with the something like: > > rsync -avu /local/<some glob> server:/destination/ # takes seconds > > Is there any faster way to transfer large number of small files with JSch? > > Regards, > LL > > > ------------------------------------------------------------------------------ > What NetFlow Analyzer can do for you? Monitors network bandwidth and > traffic > patterns at an interface-level. Reveals which users, apps, and protocols > are > consuming the most bandwidth. Provides multi-vendor support for NetFlow, > J-Flow, sFlow and other flows. Make informed decisions using capacity > planning > reports.http://sdm.link/zohodev2dev > _______________________________________________ > JSch-users mailing list > JSc...@li... > https://lists.sourceforge.net/lists/listinfo/jsch-users > > |