Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

s3cmd-speedup

Simon Law
2009-08-19
2013-02-19
  • Simon Law
    Simon Law
    2009-08-19

    What's the progress on the speedup branch?  I have a lot of files to upload and download and would really like to see downloads become pipelined and threaded.

     
    • Michal Ludvig
      Michal Ludvig
      2009-08-19

      Hi, you can safely use s3cmd-speedup. Very little has changed between 0.9.9-rc3 and 0.9.9, so s3cmd-0.9.9-rc3-speedup should be good enough for everyday use.

      It does pipelining but not threading. I've done some experiments with threaded upload and realised that the speed difference was only marginal, not worth the added complexity.

       
    • Simon Law
      Simon Law
      2009-08-25

      I get one of these messages for each file s3cmd-speedup is trying to download:

      s3://akoha-hg/akoha/lib/python-flup/.hg/hgrc -> akoha/lib/python-flup/.hg/hgrc  [284 of 340]
      0 of 0     0% in    0s     0.00 B/s  failed
      WARNING: Retrying failed request: /akoha/lib/python-flup/.hg/hgrc ()
      WARNING: Waiting 3 sec...

      After doing some hunting, it appears the exception is here:

      Traceback (most recent call last):
        File "/home/akoha/hg/tools/lib/python/S3/S3.py", line 542, in recv_file
          http_response = conn.c.getresponse()
        File "/usr/lib/python2.5/httplib.py", line 918, in getresponse
          raise ResponseNotReady()
      ResponseNotReady