S3 tools

Hi all s3cmd users!

I have uploaded s3cmd 0.9.9-pre2 to SourceForge just a few minutes ago.
This pre-release contains many new features you may find handy:

* Progress meter
  - The old useless info about of every block sent/received is now gone.
    Instead I have implemented user friendly progress meter for both
    upload and download operations:
    big_document.pdf     126976 of 2304411     5% in 9s  12.51 kB/s
    This is on by default on interactive terminals (TTYs) and off when
    running non-interactively, for instance as a cron job. However on /
    off can be forced with --progress / --no-progress parameters.

* Continue getting partially downloaded files
  - Should a download of a large file from Amazon S3 bucket fatally
    fail, for instance a laptop went offline, it is now possible to
    restart 'get' from where it stopped using --continue parameter.
    The other alternative is to use --force and start download again
    from the beginning. That's good for small files but in case of, say,
    ISO images you may prefer to continue the download.

* Improved resistance to errors
  - Although s3cmd 0.9.8.4 improved resistance to some upload problems
    there were still places where s3cmd would fail, especially during
    the initial (connect) phase. Furthermore errors in download, that is
    'get' command and 'sync' from S3 to local filesystem, were not
    handled at all.
  - All these code paths were much improved and every failed request,
    be it file upload, file download or a generic command like getting a
    bucket listing, is now reissued up to 5 times. I have spent a lot of
    time with 'tcpkill' tool and hopefully there should be no more
    "Connection reset by peer", "Name or service not known", etc errors
    breaking out.

* Removing of non-empty buckets
  - "How do I remove non-empty bucket?" FAQ now has a simple answer:
    Use "s3cmd rb --force"

* Recursively remove objects from buckets
  - Related task, i.e. removing all objects with a given prefix, is now
    possible with --recursive (-r) parameter.

* Copying, renaming and moving objects
  - Amazon S3 now supports Copy operation on existing objects.
  - That allows s3cmd to have 'cp' command for making a copy of an
    object and 'mv' command for renaming / moving objects. Destination
    can be the same bucket or a different one. It's even possible to
    copy / move objects from US buckets to EU buckets and back, however
    note that Amazon will charge you a data transfer rate for such
    a move. Thanks Andrew Ryan for initial patch.
  - Note that you can change ACLs on copy / move with --acl-public and
    --acl-private options.

Although s3cmd 0.9.9-pre2 is a "works for me" release, all the above
features deserve a good testing. I may have missed some corner cases.

I'm pretty confident this is a stable enough release and the major
reason why it isn't called "final" is a list of features that I'd like
to implement for 0.9.9. Here it goes:

TODO list for final 0.9.9 (in the order of importance):
- Support --recursive for most commands. As you know 's3cmd ls' of a
  large bucket is useless. Also 'put', 'get', 'cp' and 'mv' are good
  candidates for --recursive support. They should also process --exclude
  lists.
- Add 'setacl' command for changing access rights for existing objects.
  At least for making private objects public and vice versa.
- Add --include/--include-from/--rinclude*, similar to
  what --exclude does. This in fact would be a good exercise for a
  wannabe python developer who'd like to contribute to s3cmd project ;-)
- Update documentation and manual page ;-)

That's it. Give s3cmd 0.9.9-pre2 some bashing if you can. Download the
source either from 'testing' package on SourceForge:
https://sourceforge.net/project/showfiles.php?group_id=178907
or stay up to date with a SVN checkout of s3cmd "trunk":
http://s3tools.logix.cz/download

I'm ready to hear what problems you experience. Emails of "it works
great, no problems at all" nature are welcome even more ;-)

Enjoy

Michal