s3cmd-speedup

Simon Law
2009-08-19
2013-02-19
  • Simon Law

    Simon Law - 2009-08-19

    What's the progress on the speedup branch?  I have a lot of files to upload and download and would really like to see downloads become pipelined and threaded.

     
    • Michal Ludvig

      Michal Ludvig - 2009-08-19

      Hi, you can safely use s3cmd-speedup. Very little has changed between 0.9.9-rc3 and 0.9.9, so s3cmd-0.9.9-rc3-speedup should be good enough for everyday use.

      It does pipelining but not threading. I've done some experiments with threaded upload and realised that the speed difference was only marginal, not worth the added complexity.

       
    • Simon Law

      Simon Law - 2009-08-25

      I get one of these messages for each file s3cmd-speedup is trying to download:

      s3://akoha-hg/akoha/lib/python-flup/.hg/hgrc -> akoha/lib/python-flup/.hg/hgrc  [284 of 340]
      0 of 0     0% in    0s     0.00 B/s  failed
      WARNING: Retrying failed request: /akoha/lib/python-flup/.hg/hgrc ()
      WARNING: Waiting 3 sec...

      After doing some hunting, it appears the exception is here:

      Traceback (most recent call last):
        File "/home/akoha/hg/tools/lib/python/S3/S3.py", line 542, in recv_file
          http_response = conn.c.getresponse()
        File "/usr/lib/python2.5/httplib.py", line 918, in getresponse
          raise ResponseNotReady()
      ResponseNotReady

       

Log in to post a comment.