Re: [JSch-users] SFTP put and large files

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi,

I finally got around to running the Sftp example.  I had to take the  
Swing stuff out since I ssh into a box with a faster connection to do  
the file transfer.  I ended up hard-coding the username, host, and  
password in Sftp.java.  I also modified the monitor class to display  
the progress with a println.  The rest of Sftp.java was not  
modified.  When I tried to transfer a 4GB file, the Sftp example had  
the same OutOfMemoryError.  Here's the output from it ...

sch-0.1.29.jar:. Sftp
sftp> put aFileName
aFileName: Uploaded 32673 bytes (0%) out of 4294967296.
aFileName: Uploaded 42964995 bytes (1%) out of 4294967296.
aFileName: Uploaded 85929990 bytes (2%) out of 4294967296.
aFileName: Uploaded 128862312 bytes (3%) out of 4294967296.
aFileName: Uploaded 171827307 bytes (4%) out of 4294967296.
aFileName: Uploaded 214759629 bytes (5%) out of 4294967296.
aFileName: Uploaded 257724624 bytes (6%) out of 4294967296.
aFileName: Uploaded 300656946 bytes (7%) out of 4294967296.
aFileName: Uploaded 343621941 bytes (8%) out of 4294967296.
aFileName: Uploaded 386554263 bytes (9%) out of 4294967296.
aFileName: Uploaded 429519258 bytes (10%) out of 4294967296.
aFileName: Uploaded 472451580 bytes (11%) out of 4294967296.
aFileName: Uploaded 515416575 bytes (12%) out of 4294967296.
aFileName: Uploaded 558348897 bytes (13%) out of 4294967296.
aFileName: Uploaded 601313892 bytes (14%) out of 4294967296.
aFileName: Uploaded 644246214 bytes (15%) out of 4294967296.
aFileName: Uploaded 687211209 bytes (16%) out of 4294967296.
aFileName: Uploaded 730176204 bytes (17%) out of 4294967296.
aFileName: Uploaded 773108526 bytes (18%) out of 4294967296.
aFileName: Uploaded 816073521 bytes (19%) out of 4294967296.
aFileName: Uploaded 859005843 bytes (20%) out of 4294967296.
aFileName: Uploaded 901970838 bytes (21%) out of 4294967296.
aFileName: Uploaded 944903160 bytes (22%) out of 4294967296.
aFileName: Uploaded 987868155 bytes (23%) out of 4294967296.
aFileName: Uploaded 1030800477 bytes (24%) out of 4294967296.
aFileName: Uploaded 1073765472 bytes (25%) out of 4294967296.
aFileName: Uploaded 1116697794 bytes (26%) out of 4294967296.
aFileName: Uploaded 1159662789 bytes (27%) out of 4294967296.
aFileName: Uploaded 1202595111 bytes (28%) out of 4294967296.
aFileName: Uploaded 1245560106 bytes (29%) out of 4294967296.
aFileName: Uploaded 1288492428 bytes (30%) out of 4294967296.
aFileName: Uploaded 1331457423 bytes (31%) out of 4294967296.
aFileName: Uploaded 1374389745 bytes (32%) out of 4294967296.
aFileName: Uploaded 1417354740 bytes (33%) out of 4294967296.
aFileName: Uploaded 1460319735 bytes (34%) out of 4294967296.
aFileName: Uploaded 1503252057 bytes (35%) out of 4294967296.
aFileName: Uploaded 1546217052 bytes (36%) out of 4294967296.
aFileName: Uploaded 1589149374 bytes (37%) out of 4294967296.
aFileName: Uploaded 1632114369 bytes (38%) out of 4294967296.
aFileName: Uploaded 1675046691 bytes (39%) out of 4294967296.
aFileName: Uploaded 1718011686 bytes (40%) out of 4294967296.
aFileName: Uploaded 1760944008 bytes (41%) out of 4294967296.
aFileName: Uploaded 1803909003 bytes (42%) out of 4294967296.
aFileName: Uploaded 1846841325 bytes (43%) out of 4294967296.
aFileName: Uploaded 1889806320 bytes (44%) out of 4294967296.
aFileName: Uploaded 1932738642 bytes (45%) out of 4294967296.
aFileName: Uploaded 1975703637 bytes (46%) out of 4294967296.
aFileName: Uploaded 2018635959 bytes (47%) out of 4294967296.
aFileName: Uploaded 2061600954 bytes (48%) out of 4294967296.
aFileName: Uploaded 2104565949 bytes (49%) out of 4294967296.
aFileName: Uploaded 2147498271 bytes (50%) out of 4294967296.
aFileName: Uploaded 2190463266 bytes (51%) out of 4294967296.
aFileName: Uploaded 2233395588 bytes (52%) out of 4294967296.
aFileName: Uploaded 2276360583 bytes (53%) out of 4294967296.
aFileName: Uploaded 2319292905 bytes (54%) out of 4294967296.
aFileName: Uploaded 2362257900 bytes (55%) out of 4294967296.
aFileName: Uploaded 2405190222 bytes (56%) out of 4294967296.
aFileName: Uploaded 2448155217 bytes (57%) out of 4294967296.
aFileName: Uploaded 2491087539 bytes (58%) out of 4294967296.
aFileName: Uploaded 2534052534 bytes (59%) out of 4294967296.
aFileName: Uploaded 2576984856 bytes (60%) out of 4294967296.
aFileName: Uploaded 2619949851 bytes (61%) out of 4294967296.
aFileName: Uploaded 2662882173 bytes (62%) out of 4294967296.
aFileName: Uploaded 2705847168 bytes (63%) out of 4294967296.
aFileName: Uploaded 2748779490 bytes (64%) out of 4294967296.
aFileName: Uploaded 2791744485 bytes (65%) out of 4294967296.
aFileName: Uploaded 2834709480 bytes (66%) out of 4294967296.
aFileName: Uploaded 2877641802 bytes (67%) out of 4294967296.
aFileName: Uploaded 2920606797 bytes (68%) out of 4294967296.
aFileName: Uploaded 2963539119 bytes (69%) out of 4294967296.
aFileName: Uploaded 3006504114 bytes (70%) out of 4294967296.
aFileName: Uploaded 3049436436 bytes (71%) out of 4294967296.
aFileName: Uploaded 3092401431 bytes (72%) out of 4294967296.
aFileName: Uploaded 3135333753 bytes (73%) out of 4294967296.
aFileName: Uploaded 3178298748 bytes (74%) out of 4294967296.
aFileName: Uploaded 3221231070 bytes (75%) out of 4294967296.
aFileName: Uploaded 3264196065 bytes (76%) out of 4294967296.
aFileName: Uploaded 3307128387 bytes (77%) out of 4294967296.
aFileName: Uploaded 3350093382 bytes (78%) out of 4294967296.
aFileName: Uploaded 3393025704 bytes (79%) out of 4294967296.
java(5363,0x1804600) malloc: *** vm_allocate(size=32768) failed  
(error code=3)
java(5363,0x1804600) malloc: *** error: can't allocate region
java(5363,0x1804600) malloc: *** set a breakpoint in szone_error to  
debug
java(5363,0x1804600) malloc: *** vm_allocate(size=32768) failed  
(error code=3)
java(5363,0x1804600) malloc: *** error: can't allocate region
java(5363,0x1804600) malloc: *** set a breakpoint in szone_error to  
debug
Exception in thread "main" java.lang.OutOfMemoryError
         at java.io.FileInputStream.readBytes(Native Method)
         at java.io.FileInputStream.read(FileInputStream.java:194)
         at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:536)
         at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:403)
         at Sftp.main(Sftp.java:219)

The example was run on Mac OS X with java v1.5.  Any idea what is  
going wrong?

Thanks,
Tiff

On Oct 24, 2006, at 6:33 PM, Atsuhiko Yamanaka wrote:

> Hi,
>
>    +-From: Tiffany Dodge <ti...@ub...> --
>    |_Date: Tue, 24 Oct 2006 16:35:43 -0700 ____
>    |
>    ...
>    |In putByte, begin = 0, index = 27, length = 4
>    |Runtime.getRuntime().freeMemory() = 27104200
>    |datalen = 32673
>    |DEBUG: java.lang.OutOfMemoryError
>    |         at java.io.FileInputStream.readBytes(Native Method)
>    |         at java.io.FileInputStream.read(FileInputStream.java:194)
>    |         at com.jcraft.jsch.ChannelSftp._put(Unknown Source)
>
>    |This time I also  noticed this in the shell, which might be  
> useful ...
>    |java(4425,0x1804600) malloc: *** vm_allocate(size=32768) failed
>    |(error code=3)
>    |java(4425,0x1804600) malloc: *** error: can't allocate region
>    |java(4425,0x1804600) malloc: *** set a breakpoint in  
> szone_error to
>    |debug
>    |java(4425,0x1804600) malloc: *** vm_allocate(size=32768) failed
>    |(error code=3)
>    |java(4425,0x1804600) malloc: *** error: can't allocate region
>    |java(4425,0x1804600) malloc: *** set a breakpoint in  
> szone_error to
>    |debug
>
>    |It looks to me like the amount to be allocated is not greater than
>    |the free memory, so it does look like it would be explained by the
>    |bug until I read the work-around ... "Attempting to read the  
> file in
>    |multiple smaller chunks seem to work,." ... and then I wasn't  
> sure.
>    |It seems to me that ChannelSftp.java is reading the file in chunks
>    |(actually 32673 byte chunks).  Would you agree?  FYI, I think  
> the vm
>    |arg -Xmx64m is used so I will specify -Xmx32m instead since the  
> bug
>    |seems to indicate that a smaller heap size will not cause the
>    |problem.  I'll let you know if this helps.
>    |Any more advice or suggestions would be greatly appreciated!
>
> Can you try 'examples/Sftp.java' to upload that huge file?
> IMHO, I guess your application has really used too much memory and
> the memory allocation(malloc) in the native method has been just  
> failed.
>
>
> Sincerely,
> --
> Atsuhiko Yamanaka
> JCraft,Inc.
> 1-14-20 HONCHO AOBA-KU,
> SENDAI, MIYAGI 980-0014 Japan.
> Tel +81-22-723-2150
>     +1-415-578-3454
> Fax +81-22-224-8773
> Skype callto://jcraft/
>