Hi s3cmd users!
Another week passed, another release arrived - s3cmd
The main feature of this release is Persistent reusable HTTP
connections - an improvement that brings massive speedup for
pretty much all Amazon S3 operations.
Up until now s3cmd used to open a new connection for each file
upload, download, delete, etc. That resulted in quite a big
overhead, especially for short requests like remote file copy or
delete. Even worse when HTTPS was turned on - setting up the SSL
channel for each and every request was a very resource intensive and
time consuming exercise.
From s3cmd 1.5.0-alpha3 on we support a much faster approach - we
open the connection once and use it again and again and again
without closing and re-opening for each subsequent request. That
achieves a significant speedup in all operations.
From my office on a remote Pacific island with slow ADSL and huge
latency to S3 I get:
- Testsuite run without SSL from 3 min 54 sec down to 2' 54" -
that's 26% speedup
- Testsuite run with SSL (i.e. HTTPS) from 5' 59" down to 3' 11" =
- Uploading 10000 files from 145' 43" down to 57' 43" = 61% speedup
- Deleting 10000 files from 121' 41" down to 33' 6" - that's 72%
Obviously if you are on a fat pipe with a low latency close to an
Amazon datacentre you may not experience such an extreme improvement
- I'm keen to hear how much (if at all) the new connection handling
helps in your case!
With regards to other changes in alpha3
- new switch --quiet / -q to suppress all "normal" output from
s3cmd. (Siddarth Prakash)
- s3cmd get now honours umask again - for some time the downloaded
files were created with 0600 permissions - that's now corrected.
- Matt Domsch did a great job fixing small annoying bugs and closing
resolved tickets - thanks!
Please download, test and report any problems
The most up to date sourcecode is available on GitHub:
Also check out our website for further updates:
Have a nice day!