uploading large files eats memory and crahes

denis dag
2008-04-06
2013-05-02
  • denis dag

    denis dag - 2008-04-06

    Hi,
    I installed curlftpfs on my Fedora8 (it's in the main repository) because I was searching for a fast method using fuse to transfer files on a file-server.
    curlftpfs was the perfect solution.
    But today I couldn't transfer a large archive file to my server because curlftpfs is trying to load it entirely into memory and then crashes.
    So for me, it is finally unusable. I filed a bug on Fedora bugzilla. I didn't check the forums on this site so I thought this project was unmaintained anymore. I'm so sorry for duplicating that bug.
    I only checked the bug-report (dated 2006-06-03 15:55) and the comments associated with it.

    So my questions are: is there a fix planned ? and if there is one planned, when will it be available ?
    If it can't be fixed, can anyone help me find something that can replace curlftpfs (no sshfs, it's too slow) ?

    Regards,
    Denis

     
    • Steven Shiau

      Steven Shiau - 2008-04-11

      We encountered the same problem,
      http://sourceforge.net/forum/forum.php?thread_id=1785376&forum_id=542750
      Robson Braga Araujo replied me about 9 months ago that this will be fixed soon, but it is not fixed so far. I think this project is not maintained anymore. Maybe someone can fork another project to do this.
      So far I still can not find a replacement about this. lftpfs is good, but it's read-only.

       
      • n.f.

        n.f. - 2008-04-11

        Actually the code in the google svn fixes this problem:

        http://code.google.com/p/curlftpfs/source/checkout

        regards,
        Norbert

         
        • denis dag

          denis dag - 2008-04-15

          Hi
          I tried the google svn version and it works like charm.
          My question is why not make a new release then with this critical bug corrected ?

          Then with a brand new release, linux distributions would be able to update their packages with the new corrected version.
          So how can do it and to whom should I address this kind of requests ?

          thanks
          Bye for now

           
          • Robson Braga Araujo

            I'm the original author of curlftpfs. Norbert volunteered to solve the problem with file uploading. He took a patch from Miklos Szeredi, applied and fixed some bugs. I did not make a new release because I felt that this is not a satisfactory solution. It seems to work, but it really doesn't when you're doing reading and writing from the same handle or if you're writing to arbitrary positions. I felt it will fail in many obscure ways and leave the user wondering what happened.

            If a lot of you have tested it extensively and feel that it's a good solution I can make a release.

             
            • Steven Shiau

              Steven Shiau - 2008-04-15

              Robson,
              Nice to hear you.
              curlftpfs is such a nice project, thanks.
              I am wondering maybe you are very busy so you haven't release a new one for more than 8 months. Anyways, thanks for all your effort.
              The codes I got from codes.google.com, although it works for uploading large file, but it seems to have some problems when reading the file immediately after it is created, as I mentioned in https://sourceforge.net/forum/message.php?msg_id=4901916.
              Before I made such a change, there was no such problem.
              Maybe you can solve it ?
              Thanks in advance.

               
            • Steven Shiau

              Steven Shiau - 2008-04-25

              Robson,
              Any plan to fix the problem which norbertf and I encountered:
              https://sourceforge.net/forum/message.php?msg_id=4901916.

              Thanks in advance.

               
    • Steven Shiau

      Steven Shiau - 2008-04-13

      Norbert,
      Great! Thanks for that. However, when I svn checkout the source code, and tried to compile on Debian etch, it failed in:
      cc -DHAVE_CONFIG_H -I. -I. -I.  -DFUSE_USE_VERSION=25   -g -Wall -O2 -Wall -W -D_REENTRANT -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include   -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse    -c ftpfs.c
      ftpfs.c: In function ‘ftpfs_write_thread’:
      ftpfs.c:494: error: ‘CURLOPT_APPEND’ undeclared (first use in this function)
      ftpfs.c:494: error: (Each undeclared identifier is reported only once
      ftpfs.c:494: error: for each function it appears in.)
      ftpfs.c:473: warning: unused variable ‘range’
      make[3]: *** [ftpfs.o] Error 1
      make[3]: Leaving directory `/home/steven/tmp/curlftpfs/curlftpfs-0.9.1'
      make[2]: *** [all-recursive] Error 1
      make[2]: Leaving directory `/home/steven/tmp/curlftpfs/curlftpfs-0.9.1'
      make[1]: *** [all] Error 2
      make[1]: Leaving directory `/home/steven/tmp/curlftpfs/curlftpfs-0.9.1'
      make: *** [debian/stamp-makefile-build] Error 2

      Any idea ?

       
      • n.f.

        n.f. - 2008-04-13

        Hmm - libcurl too old? I don't know when they introduced CURLOPT_APPEND...

         
      • Steven Shiau

        Steven Shiau - 2008-04-13

        OK, I switched to Ubuntu 8.04 beta, the libcurl3 is 7.18.0-1ubuntu, and this time it failed in the different place:
        cc -DHAVE_CONFIG_H -I. -I. -I.  -DFUSE_USE_VERSION=25   -g -O2 -g -Wall -O2 -Wall -W -D_REENTRANT -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include   -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse    -c ftpfs.c
        ftpfs.c: In function 'ftpfs_write_thread':
        ftpfs.c:473: warning: unused variable 'range'
        ftpfs.c: At top level:
        ftpfs.c:1674: error: static declaration of 'fuse_opt_insert_arg' follows non-static declaration
        /usr/include/fuse/fuse_opt.h:236: error: previous declaration of 'fuse_opt_insert_arg' was here
        make[3]: *** [ftpfs.o] Error 1
        make[3]: Leaving directory `/home/steven/tmp/curlftpfs/curlftpfs-0.9.1'
        make[2]: *** [all-recursive] Error 1
        make[2]: Leaving directory `/home/steven/tmp/curlftpfs/curlftpfs-0.9.1'
        make[1]: *** [all] Error 2
        make[1]: Leaving directory `/home/steven/tmp/curlftpfs/curlftpfs-0.9.1'
        make: *** [debian/stamp-makefile-build] Error 2 

        Could you please tell me which GNU/Linux and version are you using ?
        Thanks.

         
        • n.f.

          n.f. - 2008-04-13

          This seems to be a FUSE version problem. Which version do you use? I wonder if it's really 2.5, as specified by -DFUSE_USE_VERSION=25... I'm using FUSE 2.6 for instance.

           
          • Steven Shiau

            Steven Shiau - 2008-04-13

            Great. By modifying DFUSE_USE_VERSION=27 in Makefile.in, I can successfully compiled it on Ubuntu 8.04 beta.
            Looks like more efforts are necessary on Debian etch. Newer libcurl and maybe more about its dependence on etch.

            Thanks!

             
            • n.f.

              n.f. - 2008-04-13

              Cool - can you test if it works, too? (uploading big files).

              Cheers,
              Norbert

               
    • Steven Shiau

      Steven Shiau - 2008-04-13

      Norbert,
      Yes, I confirm that the codes from http://code.google.com/p/curlftpfs/source/checkout
      solve this problem. Now I can dump a 700 MB file to the ftpfs. The memory in my testing virtual machine (ftpfs client) is 300 MB.
      However, I have another issue. It looks like I can not access the the file immediately after it was created:
      ----------------
      root@hardy:~# df
      Filesystem           1K-blocks      Used Available Use% Mounted on
      aufs                    115640     72444     43196  63% /
      varrun                  115640        40    115600   1% /var/run
      varlock                 115640         0    115640   0% /var/lock
      udev                    115640        40    115600   1% /dev
      devshm                  115640         0    115640   0% /dev/shm
      tmpfs                   115640         0    115640   0% /tmp
      curlftpfs#ftp://192.168.120.254/tmp/img/
                           7999999992         0 7999999992   0% /mnt
      root@hardy:~# cp /etc/fstab /mnt
      root@hardy:~# cat /mnt/fstab
      cat: /mnt/fstab: Permission denied
      root@hardy:~# cat /mnt/fstab
      # /etc/fstab: static file system information.
      #
      # <file system> <mount point>   <type>  <options>       <dump>  <pass>

      aufs / aufs rw 0 0
      tmpfs /tmp tmpfs nosuid,nodev 0 0
      /dev/sda2 swap swap defaults 0 0
      ----------------

      It seems I have to access or wait for a while, otherwise I will get the error message
      "Permission denied". Not sure it is the problem with vsftpd server or curlftps.
      Some info about my environment in cureftpfs client:
      -------------
      kernel 2.6.24-16-generic
      curl           7.18.0-1ubuntu
      libcurl3       7.18.0-1ubuntu
      -------------

      Do you know any reason why ?

      Thank you very much.

       
      • n.f.

        n.f. - 2008-04-24

        I can reproduce this bug. Could be a problem with cache.c in curlftpfs. With the option "-o cache=no" it seems to go away.

        Although this produces a lot of warnings like this one:

        (process:30512): GLib-CRITICAL **: g_hash_table_lookup: assertion `hash_table != NULL' failed

        Norbert

         
        • n.f.

          n.f. - 2008-04-25

          Hmm - trying again the problem also occurs with "-o cache=no". So the problem seems to be somewhere else.

           
    • n.f.

      n.f. - 2008-04-25

      I did some more testing: This problem only occurs with newer versions of libcurl:

      libcurl/7.18.1

      but not with version

      libcurl/7.17.0

      Might be a libcurl bug.

       

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks