You can subscribe to this list here.
2008 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(2) |
Dec
(1) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2009 |
Jan
(2) |
Feb
(2) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
2011 |
Jan
(3) |
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2012 |
Jan
(2) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2013 |
Jan
|
Feb
(1) |
Mar
(2) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(1) |
2014 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
2015 |
Jan
|
Feb
(2) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
2016 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2017 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
2018 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2020 |
Jan
|
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
2021 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
2022 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
2023 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(1) |
From: Florent V. (Sodria) <fl...@so...> - 2023-12-12 10:28:41
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.4.0 has been released on Github, Sourceforge and PyPi. This release should not contain any breaking change except a few output message changes. With this release, we reach the awesome milestone of exactly 100 contributors to s3cmd. Thanks again to all the contributors, the testers and the issue reporters. s3cmd-2.4.0 - 2023-12-12 =============== * Added "setversioning" command for versioning configuration (Kuan-Chun Wang) * Added "settagging", "gettagging", and "deltagging" commands for bucket/object tagging (Kuan-Chun Wang) * Added "setobjectretention" and "setobjectlegalhold" commands (Etienne Adam/Withings SAS) * Added "setownership" and "setblockpublicaccess" commands * Added "cfinval" command to request Cloudfront to invalidate paths (#1256) * Added "--keep-dirs" option to have the folder structure preserved on remote side * Added --skip-destination-validation option for "setnotification" command (Kuan-Chun Wang) * Added "--max-retries" flag and "max_retries" config option (#914) * Added FIPS support (Michael Roth) * Added "object ownership" and block public access" values to "info" command output for buckets * Added to "ls" command a "DIROBJ" tag for directory objects in S3 remote * Added server profiles to run-tests.py to skip tests depending on the server type * Fixed "TypeError: sequence item 1: expected str instance, bytes found" error with Python 3.12 (#1343) * Fixed a missing return for "object_batch_delete" of S3.py (James Hewitt) * Fixed "object is not callable" error because of md5 FIPS test (#1005) * Fixed "compute_content_md5 is not defined" error for "setversioning" (#1312) (Gavin John) * Fixed list objects to use NextMarker when only prefixes are returned (Albin Parou) * Fixed upload to not retry when an S3 compatible server is full * Fixed recursive delete of objects named with whitespace (#976) * Fixed the mime type when uploading directories to be "application/x-directory" * Fixed "string indices must be integers" error for sync when in dry-run mode (#1313) (Shohei Tanaka) * Fixed SignatureDoesNotMatch error when modifying an object on Cloudflare R2 (#1332) (Philip C Nilsson) * Fixed Cloudfront invalidation issue for paths with wildcard or special characters * Fixed Cloudfront crash because of error reporting for retries * Fixed Cloudfront "unable to parse URL" error (#1292) * Improved the handling of "empty" files on the remote side to sync with local folders * Improved "abortmp" command by requiring an object to avoid bad accidents when using Ceph (Joshua Haas) * Improved file download by retrying when encountering SlowDown or TooManyRequests errors (Robin Geiger) * Improved error messages in case of connection error or host unreachable * Improved error messages to be more explicit for upload errors after retries * Improved remote2local attributes setting code * Improved remote2local with more explicit error messages when setting attributes (#1288) * Improved remote2local output messages by using the "mkdir" prefix instead of "make dir" * Improved the SortedDict class * Improved run-test.py by using "--include" when calling Curl instead of "-include" (Matthew James Kraai) * Improved GitHub CI by enabling pip cache in actions/setup-python (Anton Yakutovich) * Improved GitHub CI by adding a "codespell" check on push and PRs (Yaroslav Halchenko) * Updated GitHub CI tests to use more recent versions of Minio and Python * Upgraded GitHub actions (Anton Yakutovich) * Cleanup and update of copyright headers, docs, comments and setup.py * Cleanup to fix "invalid escape sequence" syntax warnings * Many other bug fixes and cleanups Shortlog below. Thanks, Florent Albin Parou (1): fix: Use NextMarker if only prefixes are returned Anton Yakutovich (4): Bump GitHub actions Enable pip cache in actions/setup-python Extract deps from `setup.py` to `requirements.txt` Return additional pip install from `setup.py` Etienne Adam (1): Add setobjectretention and setobjectlegalhold commands Florent Viard (54): Change version to development 2.3.0-dev Merge pull request #1233 from maroth96/maroth96-add-fips-support Fixes #1005 - Fixes object is not callable error because of md5 FIPS test Merge pull request #1277 from jeffwang0516/notification-option Merge pull request #1278 from Fumesover/master Better error message in case of connection error or host unreachable Retry a file download in case of SlowDown or TooManyRequests error Fixes #914 - Add "--max-retries" and "max_retries" config option Don't retry upload when server is full Improve error message to be more explicit for upload errors after retries run-tests.py: Fixes os error when trying to skip first tests run-tests.py: Added server profiles to skip tests depending on the server type Update CI test.yml to use last minio version and use python 3.10 instead of 3.10-dev Merge pull request #1281 from drakulavich/patch-1 Merge pull request #1282 from drakulavich/ci/pip_cache Cloudfront: Remove wildcard import for exceptions Cloudfront: Fixes paths with wildcard or special characters to not be invalidated Cloudfront: Fixes crash because of error reporting for retries Fixes #1256 - Add cfinval command to request cloudfront to invalidate paths Merge pull request #1034 from Jamstah/patch-1 Crypto: Remove old python legacy code Move hash functions into Crypto module Fixes #976 - Fixes recursive delete of objects named with whitespace Add to ls command a "DIROBJ" tag for directory objects in S3 remote Add "--keep-dirs" option to have the folder structure preserved on remote side SortedDict: Add support for reversed() and ensure SortedDictIterator is recognized as an iterable remote2sync: children files and folders have to be deleted before parent folders remote2local: Use rmdir to delete folders run-tests.py: Add test for keep-dirs feature Refactor to treat copy_pairs like remote_list and update_list for sync commands Set correct mime type when uploading directories Merge pull request #1287 from fviard/feature_keep_dirs Merge pull request #1289 from jahschwa/bug/abortmp-require-object Use "mkdir" prefix instead of "make dir" for remote2local output messages Fixes #1288 - More explicit errors when setting attributes for remote2local Cleanup of remote2local attributes setting code Don't use os.path for handling s3 paths Merge pull request #1284 from jeffwang0516/set-versioning Fixes #1292 - cloudfront: Parse all aws possible urls in httpurl_to_s3uri Merge pull request #1314 from xcir/fix/issue1313 CI: Add python 3.11 and 3.12 to the test matrix Fixes #1343 - bytes to be used for repl string in stripNameSpace cleanup: fixes "invalid escape sequence" Syntax Warnings Merge pull request #1324 from yarikoptic/enh-codespell cosmetic: comment typo fixes Merge pull request #1333 from PhilipNilsson7/master Merge pull request #1296 from kraai/curl-include-option Merge pull request #1297 from withings-sas/legalhold_and_retention cleanup: update copyright headers and other values Fixes #1359 - add setownership and setblockpublicaccess commands + related entries in info cosmetic: typo fix Merge pull request #1283 from jeffwang0516/tagging-support s3cmd: handle cases of empty key or value for settagging Update version to 2.4.0 Gavin John (1): Fixes error "'compute_content_md5' is not defined" for setversioning James Hewitt (1): Add missing return Joshua Haas (2): bug: require object in abortmp make error message consistent Matthew James Kraai (1): Pass --include to curl Michael Roth (2): Add FIPS support Move md5 to BaseUtils PhilipCNilsson7 (1): fix SignatureDoesNotMatch error when modifying an object on Cloudflare R2 Shohei Tanaka (1): Fix type error(#1313) Yaroslav Halchenko (5): Add github action to codespell master on push and PRs Add rudimentary codespell config ignores for codespell Fixed manually some ambigous typos [DATALAD RUNCMD] run codespell throughout jeffwang0516 (5): Add option to skip destination validation when applying notification config Update help message Support versioning configuration Refactoring Support bucket/object tagging command |
From: Florent V. (Sodria) <fl...@so...> - 2022-10-03 16:37:33
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.3.0 has been released on github, sourceforge and PyPi. This release should not contain any breaking change. Thanks again to all the contributors, the testers and the issue reporters. s3cmd-2.3.0 - 2022-10-03 =============== * Added "getnotification", "setnotification", and "delnotification" commands for notification policies (hrchu) * Added support for AWS_STS_REGIONAL_ENDPOINTS (#1218, #1228) (Johan Lanzrein) * Added ConnectionRefused [111] exit code to handle connection errors (Salar Nosrati-Ershad) * Added support for IMDSv2. Should work automatically on ec2 (Anthony Foiani) * Added --list-allow-unordered to list objects unordered. Only supported by Ceph based s3-compatible services (#1269) (Salar Nosrati-Ershad) * Fixed --exclude dir behavior for python >= 3.6 (Daniil Tararukhin) * Fixed Cloudfront invalidate retry issue (Yuan-Hsiang Lee) * Fixed 0 byte cache files crashing s3cmd (#1234) (Carlos Laviola) * Fixed --continue behavior for the "get" command (#1009) (Anton Ustyugov) * Fixed unicode issue with fixbucket (#1259) * Fixed CannotSendRequest and ConnectionRefusedError errors at startup (#1261) * Fixed error reporting for object info when the object does not exist * Fixed "setup.py test" to do nothing to avoid failure that could be problematic for distribution packaging (#996) * Improved expire command to use Rule/Filter/Prefix for LifecycleConfiguration (#1247) * Improved PASS/CHECK/INCLUDE/EXCLUDE debug log messages * Improved setup.py with python 3.9 and 3.10 support info(Ori Avtalion) * Many other bug fixes Shortlog below. Thanks, Florent Anthony Foiani (2): use IMDSv2 to retrieve instance metadata make IMDSv2 optional Carlos Laviola (1): treat 0 byte cache files as nonexistent Daniil Tararukhin (1): fix os walk when excluding directories Florent Viard (13): Update CI test.yml to run on PR Fixes #1261 - Always read a conn after a request to be able to reuse it Fixes #1259 - Fixes unicode issue with fixbucket Fixes #1259 - Correct typo when retrieving response from IMDSv2 Ensure 404 exc content is filled even for object_info Fixes #1009 - Fixes --continue behavior for the "get" command Fixes #996 - Avoid crashing when setup.py test is called Fixes #1247 - Update expire command to use Rule/Filter/Prefix for LifecycleConfiguration Fixes #1261 - Additional fix for CannotSendRequest at startup Fixes ConnectionRefusedError not found error for python 2 Use str instead of repr in FileList debug logs Use quotes around files in FileLists for debug logs Update version to 2.3.0 Johan Lanzrein (1): Take into account the AWS_STS_REGIONAL_ENDPOINTS Ori Avtalion (1): List support of Python 3.9, 3.10 Salar Nosrati-Ershad (3): excepting/handling tcp connection refused error feat: implement list objects allow-onordered option as a config feat: add `--list-allow-unorderd` command-line switch Yuan-Hsiang Lee (1): fixes cloudfront invalidate retry hrchu (1): Add notification commands |
From: Florent V. (Sodria) <fl...@so...> - 2021-09-27 19:19:39
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.2.0 has been released on github, sourceforge and PyPi. This release is mostly made of bug fixes and should not contain any breaking change. Thanks again to all the contributors, the testers and the issue reporters. s3cmd-2.2.0 - 2021-09-27 =============== * Added support for metadata modification of files bigger than 5 GiB * Added support for remote copy of files bigger than 5 GiB using MultiPart copy (Damian Martinez, Florent Viard) * Added progress info output for multipart copy and current-total info in output for cp, mv and modify * Added support for all special/foreign character names in object names to cp/mv/modify * Added support for SSL authentication (Aleksandr Chazov) * Added the http error 429 to the list of retryable errors (#1096) * Added support for listing and resuming of multipart uploads of more than 1000 parts (#346) * Added time based expiration for idle pool connections in order to avoid random broken pipe errors (#1114) * Added support for STS webidentity authentication (ie AssumeRole and AssumeRoleWithWebIdentity) (Samskeyti, Florent Viard) * Added support for custom headers to the mb command (#1197) (Sébastien Vajda) * Improved MultiPart copy to preserve acl and metadata of objects * Improved the server errors catching and reporting for cp/mv/modify commands * Improved resiliency against servers sending garbage responses (#1088, #1090, #1093) * Improved remote copy to have consistent copy of metadata in all cases: multipart or not, aws or not * Improved security by revoking public-write acl when private acl is set (#1151) (ruanzitao) * Improved speed when running on an EC2 instance (#1117) (Patrick Allain) * Reduced connection_max_age to 5s to avoid broken pipes as AWS closes https conns after around 6s (#1114) * Ensure that KeyboardInterrupt are always properly raised (#1089) * Changed sized of multipart copy chunks to 1 GiB * Fixed ValueError when using more than one ":" inside add_header in config file (#1087) * Fixed extra label issue when stdin used as source of a MultiPart upload * Fixed remote copy to allow changing the mime-type (ie content-type) of the new object * Fixed remote_copy to ensure that meta-s3cmd-attrs will be set based on the real source and not on the copy source * Fixed deprecation warnings due to invalid escape sequences (Karthikeyan Singaravelan) * Fixed getbucketinfo that was broken when the bucket lifecycle uses the filter element (Liu Lan) * Fixed RestoreRequest XML namespace URL (#1203) (Akete) * Fixed PARTIAL exit code that was not properly set when needed for object_get (#1190) * Fixed a possible inifinite loop when a file is truncated during hashsum or upload (#1125) (Matthew Krokosz, Florent Viard) * Fixed report_exception wrong error when LANG env var was not set (#1113) * Fixed wrong wiki url in error messages (Alec Barrett) * Py3: Fixed an AttributeError when using the "files-from" option * Py3: Fixed compatibility issues due to the removal of getchildren() from ElementTree in python 3.9 (#1146, #1157, #1162, # 1182, #1210) (Ondřej Budai) * Py3: Fixed compatibility issues due to the removal of encodestring() in python 3.9 (#1161, #1174) (Kentaro Kaneki) * Fixed a crash when the AWS_ACCESS_KEY env var is set but not AWS_SECRET_KEY (#1201) * Cleanup of check_md5 (Riccardo Magliocchetti) * Removed legacy code for dreamhost that should be necessary anymore * Migrated CI tests to use github actions (Arnaud J) * Improved README with a link to INSTALL.md (Sia Karamalegos) * Improved help content (Dmitrii Korostelev, Roland Van Laar) * Improvements for setup and build configurations * Many other bug fixes Shortlog below. Thanks, Florent AJP (1): Fix printing error in `report_exception(e)` Aaron Maxwell (17): Branch for Aaron / --follow-symlinks Provide --bucket-prefix for run-tests.py Test case for following symlinks (simplest case) (currently fails) Implementation of --follow-symlinks option (nonrecursive case) Test for s3cmd put --follow-symlinks Negative test for s3cmd put --follow-symlinks Manual page documentation for --follow-symlinks More thorough and precise tests for --follow-symlinks Guard against common error when using --bucket-prefix correct error in test use Autotest-3, not autotest-3 Compartmentalized symlink-safe directory walk, with all tests passing on python2.5 and python2.6 Fix leak in recursive-symlink-filtering algorithm document fswalk function name refactor to indicate _fswalk_follow_symlinks is an internal helper function Update Changelog Option to follow local symlinks for sync and put (--follow-symlinks option) including tests and documentation. Also addition to run-tests.py: --bucket-prefix option, to allow different developers to run tests in their own sandbox Abhishek Varshney (1): fix: handle empty md5 in find_md5_one Albert Bendicho (4): Off-by-one on MAX_CHUNKS verification max_chunks now a config option MAX_CHUNKS constant removed, now a config option Added multipart_max_chunks Alec Barrett (1): Update wiki URL in error message Aleksandr Chazov (4): Allow setting SSL authentication Update man pages make names more understandable allow skip key file config option for embedded keys case Alex Harford (1): Ensure testsuite-out is created before running dd Alex Offshore (1): Return decoded response in get_policy and get_cors Alexander Allakhverdiyev (1): Fix Python 3 bytes string encoding when getting IAM credentials Alexander Todorov (2): Disable hostname checks for SSL connections on RHEL 7.2 In case check_hostname isn't defined catch the error Andrew Gaul (7): Sort s3cmd ls output by bucket name Capitalize command descriptions for consistency Horizontally align object policy info Emit user metadata in object info Support signed URL content disposition type Emit storage class in info subcommand Handle missing ETag in subcmd_bucket_list Anton I. Sipos (1): Fix crash when cloudfront operations receive 500 errors. Retries parameter was not being sent to send_request as a keyword arg. Arnaud J (1): Migrate CI tests to use github actions Arto Jantunen (2): Add a close method to the ConnMan api Add option to disable connection pooling Arturas Moskvinas (1): Do not close stdout, it might be used when file is downloaded and destination is "-" Billy Crook (2): Added support for listing StorageClass attribute. Disabled by default. Enable with list_storageclass = True in .s3cfg Adding -l option for long_listing list_storageclass now provided by long_listing and -l. Minor whitespace cleanup. Bobby Impollonia (1): Add missing import of errno in S3.py Brady (2): Install instructions for PIP Correct typo in instructions Brandon Galbraith (1): Update s3cmd.1 Brendan O'Connor (1): Bugfix: use extra headers -- allows Requester Pays for buckets you don't own. Patch from http://arxiv.org/help/bulk_data_s3 Brian C. Lane (1): Fix handling of config tri-state bool values (like acl_public) Børge Nerdal Kjelsrud (1): Add option to ignore error on copy when key no longer exists Carlo Cabanilla (1): fix bug in datetime tuple concatenation Castedo Ellerman (1): content_encoding != charset, bug #153 Charlie Schluting (2): s3cmd du can gobble gigs of RAM on a bucket with millions of objects. Re-worked 'du' to traverse the structure and store only the sum at each level, allowing python to free memory at each folder. Went from 4GB consumption (and being killed) on a test bucket with ~50M objects in thousands of directories, to a max of 80M usage. that's not necessary any more. Chris Lamb (1): Support relative expiry times in signurl. Christian Rodriguez (2): Fix #849 by adding requester pays header to HEAD requests Merge branch 'bug/849' Christopher Noyes (3): allow adding encoding to content type in s3 don't add charset if it's already there, make param a comma delimited str, not multiple remove comments, add space after ; Christopher Porter (1): Fixing bug 3091912 - KeyError when copying multiple keys Cisco Guillaume (1): Merge pull request #1 from GuillaumeCisco/gcisco-fix-appdata-directory Craig Ringer (1): Add support for creating signed S3 URLs with the new signurl command Dale Lovelace (1): Allow --acl-grant on AWS groups e.g. Damian Gerow (2): Support proxying SSL connections Only support proxied SSL with Python >= 2.7 Damian Martinez (1): Adding MultiPartCopy class and enable for object_copy when remote file is greater than 5gb. Dan Carroll (1): Fixed typo: paramters -> parameters Daniel Gryniewicz (1): Add getlifecycle command Daniel Harris (1): Fix potential divide by zero errors Dave Cole (1): Apply include/exclude to destinations too, via mdomsch/s3cmd@268e7c3bd37cf60920025d556449fc1b93a1c590 Refs s3tools/s3cmd#109. David Eckelkamp (1): Let Makefile determine version number from PkgInfo.py file David Kohen (2): Add support for IAM roles and temp tokens IAM role typo fixes David Liu (1): Fail silently on SIGPIPE Devon Jones (3): added ability to set home based on environment variable added S3CMD_CONFIG documentation to the man page Merge remote-tracking branch 'upstream/master' Dmitry Kamenka (1): Copy uri_params in bucket_list Dongmao Zhang (2): add new option --limit-rate to throttle download/upload speed only gettimeofday when limitrate is required Doug Crozier (4): Changed "Buckets list" test to use pbucket rather than hardcoded names. This fixes issue #1066. Allow use of Docker to run test cases with multiple Python versions Fix for 'dictionary changed size during iteration' error on Python 3 with --cache-file (Issue #945) Test cases added for --cache-file EC2 Default User (1): .gitignore: add s3cmd.egg-info Emmanuel (1): managed files that was not remote copy to upload them directly EnlytonJ (1): Windows fix re / vs os.path.sep Eric Connell (1): added the ability to upload files by reading from stdin Eric Drechsel (1): Fix for ignored --default-mime-type option Eric Mill (1): rewrite of README into markdown Etienne Michon (1): Typo in manual Eugene Brevdo (4): Added --continue-put and --upload-id, allowing skipping of files or part files during upload when their size and md5sum match the file being uploaded. Minor bugfix to --upload-id check Fixed a small bug in MultiPart upload Fixed a upload corruption bug in MultiPart continue. Florent Viard (403): BugFix: Fix dateS3toPython to gracefully handle returned date that are not in GMT to respect iso8601. Ex.: 2014-03-05T18:45:50+00:00 instead of 2014-03-05T18:45:50Z Fixes date functions to return timestamps based on UTC time value. + so simplified formatDateTime(). Minor changes in README to fix points 1 and 2 in my comment at https://github.com/s3tools/s3cmd/pull/369. Cosmetic: removed one empty space in excess of end of run summary. Fix MD5 comparison with s3 compatible servers that are not Amazon S3. Small cleanup of code, using "in" instead of "find" for strings when possible. Fix an error in my previous commit: b5b61abc8eb159f5932b936c7ca47e618615f6a3. (.find() forgotten to be removed) Improve the exit code of s3cmd in the case of an error from S3 to give a clearer status of the reason of the failure. Some existing exit codes were modified when they were not already in use. Disable dns support for buckets when "host_bucket" hostname doesn't have a %(bucket)s item in it. Backported changes from the following pull request: https://github.com/s3tools/s3cmd/pull/369 regarding aws pricing. Updated README.md with latest changes of S3 prices: https://aws.amazon.com/fr/blogs/aws/aws-data-transfer-price-reduction/?sc_ichannel=EM&sc_icountry=global&sc_icampaigntype=launch&sc_icampaign=EM_128506540&sc_idetail=em_1778261631&ref_=pe_411040_128506540_12 Improvement of error reporting in the case of an issue when importing S3 sub-files. Now correctly cover cases like syntax errors in these files. Fixes #423 (https://github.com/s3tools/s3cmd/issues/423). Autodetected locale default should be set in default config for only being used when this field is not overriden, neither by command line or by a config file. Fixes #448 : Fixes S3UriS3 regexp to also recognize "s3://" as a valid S3UriS3 entity with no bucket and no object. Renaming some functions for the code to be more understandable. Fixes #424 : In python 2.7, parsing invalid xml now raise a ParserError(that didn't exist in python <2.7) instead of a ExpatError + cleanup in Utils.py (see issue #424 for more details) Improved handling for errors 501 and 504: Error 501: It means that a feature is not implemented in the server, so no need for useless retries. Error 504: Gateway timeout situation, exit code should indicate a server error. Cleanup: Add prologue to files to state that they are UTF-8 encoded. Fix run-test script to take into account the config file that is specified on the command line. Fixes encoding/decoding issues and so support for all strange characters (chinese, japanese, ...) in file names, folder or path arguments (Should fix bug like #417i) when UTF-8 is not the system def$ Now, in theory, everything that gets in should be converted to unicoded and then things would be converted back to string (user defined encoding) at the outmost parts when dealing with the system. When talking to S3, convert unicode to strings in utf-8 encoding. Remove useless error import Pass2 of "fixes encoding/decoding issues": os.walk replaced to handle unicodise conversion + fixed the removal of most of useless or invalid unicodise/decodise in the code. Fix for issue #432. Read the response from httplib even if there is a 0 byte to read. So httplib doesn't think that there is a response waiting to be read when the next request come. Straightforward to fix thanks to the detailed analysis of Edwintorok. Use deunicodise consistentely in object_copy and object_modify fix encoding : Added encode_to_s3 and decode_from_s3 function + switched to "unicode" call instead of "decode" as it is supposed to be more efficient. Fix wrong exception used for EOF from S3 error during recv_file. (S3Error expect an http response, not a text message) For copy and modify, take care to check if there is an error after the start of the operation (reported by a status 200 but an "Error" root entry in the xml body) In addition, for the move, It is good to check for the CopyObject xml reply to see if it is a success. But, if there is a status of 200 and no "data" in response, don't fail. Other S3 server implementations use only status code and don't return any body when there is no error. Another fix related to fix encoding issues. (file.name / stream.name are bytes, so unicodise them also) Fix invalid file transfer progress info in case of local2remote and remote2remote: If there was failed copy, seq number was preserved but previously uploaded file number was forgotten, thus user could have faced an invalid indication like: 14/8 when there is 6 upload following 6 remote copy failures. Add a link to the known issues page (on github) within the header and footer of s3cmd generated error reports. The goal is to reduce the number of duplicated "bug reports" for known issues. Fixes #528 (https://github.com/s3tools/s3cmd/issues/528). There is a lot of different "python-magic" libraries/versions, all doing different things, so all cases need to be covered. file-5.11 has be switched with "filemagic" as it should be more commonly used. Introspection in each python magic lib has been done to ensure to handle encoding or not only when it is needed. Fixes issue #576 (https://github.com/s3tools/s3cmd/issues/576). EX_SOMEFAILED was still used in a PR that was merged instead of EX_PARTIAL. Merge pull request #572 from fviard/fix_encoding_magic Merge pull request #574 from koenrh/https-by-default Don't slurp whole file while computing SHA256 checksum. Rework of the PR of petr-tichy. When using large --multipart-chunk-size-mb or --disable-multipart, the whole file was read into memory. Merge pull request #580 from Tim-Anderson/error_stop Merge pull request #583 from Tim-Anderson/issue577 Merge pull request #586 from Tim-Anderson/remove_ignore_failed_copy Merge pull request #587 from taz77/master Merge pull request #589 from jcwilson/master Merge pull request #560 from Flipboard/iter Merge pull request #593 from jrschumacher/set-cors Merge pull request #603 from mdomsch/bug/testsuite Following #603 : WARNING OUTPUT FORMAT CHANGE. Changed the format of output strings to be more consistant. Warning, the format of output for a lot of actions was changed. Thus, any automatic string for parsing the output could have to be fixed when using s3cmd after this checkin. Ex: "File hello/world.txt stored as s3://mybucket/world.txt" will now be: "upload: 'hello/world.txt' -> 's3://mybucket/world.txt' Fixes #602 (and complete #603) Fix run-tests.py to have all tests passing against HEAD If s3error message is none, don't print an useless ": None" Return NotFound error with message s3Error: NoSuchBucket when there is no bucket found to delete Raise standard S3Error in S3.py instead of simply exiting. So the normal error logic can't operate on any unexpected error. Avoid to have empty errors in output like: "ERROR: S3 error:" (Because of an 400 InvalidBucketName error returned by a non AWS3 server) Merge pull request #590 from andrewgaul/content-disposition Fix stop_on_error config type Attempt to fix the first bug of issue #600 : When a subfolder "xxx" was selected as source, the sync comparison was done against the destination base whereas it had to be done based on the "xxx" subfolder of the destination. Fixes run-tests following the fix for https://github.com/s3tools/s3cmd/issues/600 Merge pull request #616 from mdomsch/bug/612 Fix bad help info for the "--no-check-certificate" switch. Thanks to @psi29a for proposing the fix. Merge pull request #618 from mdomsch/bug/futurewarning Fixes #592 : Ignore any leading slash before bucket_name in an S3Uri. Original issue: "Running s3cmd ls s3:///foo lists owned buckets instead of emitting an error." Fix for error reporting in case of upload error. (Following the fix for #577) Fixes #581 regarding the reporting of SSL verification issues. Other problems of this issues should already be fixed. Merge pull request #621 from mdomsch/bug/ssl-checking Merge pull request #619 from fviard/second_fix_for_577 Merge pull request #617 from fviard/fix_subfolder_source Merge pull request #623 from mdomsch/bug/testsuite2 Reworked Upload and Download errors to be more consistent. Add detection and handling of different OSErrors like "File name too long." Merge pull request #628 from mdomsch/bug/curl Merge pull request #627 from fviard/upload_download_errors_rework Merge pull request #630 from mdomsch/feature/ia-class Merge pull request #611 from bsapiro/master Added --server-side-encryption-kms-id config option. Following PR #611. Fixes bug in PR #611 Fixes #636. Also remove the temporary file when encountering an S3Error. Merge pull request #599 from shigezobriand/master Fix "info" command for servers not supporting the "request_payer" feature. Based on PR #640 of Qiankunzheng. Fixes #644 : Minor fix to not output "u quoted" invalid parameter in the case of an invalid command. Reworked source and destination input of "remote2local" to cover gracefully more cases. Block "-" (stdin / stdout) to be used in parameters for sync as it has no sense. Only warn for the non regular files that are skipped. In the case of local2remote, raise an error if a source arg is missing. Fix automatic tests for recent changes. Merge pull request #650 from fviard/fix-remote2local-input Don't fail the copy if a non AWS s3 server doesn't support setting ACLs. Have extra labels coherent when progress is enabled or not and for multipart uploads. Added the --stats option that is similar to the one of rsync. Added "--stats" info for sync commands (remote2local, remote2remote, local2remote) Merge pull request #652 from s3tools/bug/598 Merge pull request #655 from fviard/rework_progress Fix duplicated code for the error alert message. Based on @edunham PR: https://github.com/s3tools/s3cmd/pull/609 Fixes #661 : Don't try to perform a delete if there is nothing to delete. + Restore warning that was in 1.5.2. Fixes #659 : Don't fail if "content-type" field is missing in info like for eucalyptus walrus. Fixes #646: number of "files to update" are not reported in the summary (there was only the new files to be added) Fixes #646 : remote2remote sync - "Updated" files were not reported as to be remotely copied when in dry-run mode. Merge pull request #667 from kobake/skip-restore-already-in-progress When using --no-preserve, still add 'md5' custom attribute if md5 check is enabled. Fixes #674 : When using AWS_CREDENTIAL_FILE, crash in case of invalid key in .aws/credentials file + updated the code for actual name of the keys. Merge pull request #711 from Shamanoid/patch-2 Merge pull request #559 from jmif/master Merge pull request #701 from makery/master Don't ask to set access_token in run_configure as it should not be used very often. Merge pull request #720 from harshavardhana/etag Merge pull request #684 from harshavardhana/try-fail Fix for local2remote in the case of "." as a source. Fixes #706 Fixes #709 : Fixes sync command with windows + cleanup Fix the error message to avoid confusion in a special cases like "no space left on device". Merge pull request #702 from artms/quiet Fixes #744 - Ensure to check non None or empty bucket names before calling check_bucket_name. Merge pull request #755 from andrewgaul/list-missing-etag Merge pull request #714 from alces-software/fix/generalize-wildcard-certificate-forgiveness Minor optimization and cosmetic fix in ConnMan. Fix Issue #672 - Patch based on PR #729 proposed by MLopez-Ibanez. Merge pull request #763 from dang/getlifecycle Fixes getBucketFromHostname for when host_bucket config has no "bucket" element. Fix the infamous "UnboundLocalError" for conn exception hidding meaningful exceptions Added an explicit error message when the servers hostname are wrong or unresolvable. Avoid having an error when user uses "https://" prefix to host_base... Fixes #741 Issues validating ssl certs when using a custom port for host_bucket address. Real fix for #772. Use urlparse to extract hostnames in connman Merge pull request #779 from cybojanek/master Follow up on #779, not using a CONNECT "tunnel" for http proxies as it is not the most common case. Fixes #777 : Fix __repr__ returning unicode and sign_v2 not encoded to s3. Merge pull request #757 from mozawa/max-ls-keys Merge pull request #781 from kwarunek/master Fixes recursive delete that was broken by the new "truncated" tag. Fixes orphan var in "du" exception handler. run-tests.py Update expirations tests dates to the future. Merge pull request #784 from mozawa/configure-endpoint Merge pull request #785 from shaform/py3k Fix failure of copy or rename for s3-like servers that don't have support for acl. Fix error of "info" command when s3-like servers don't implement "acl" or another feature. Merge pull request #789 from shaform/py3k Merge pull request #787 from Abhishekvrshny/fix-empty-md5 Avoid errors with modify command when s3-like servers don't support acl. Fix possible bug in previous change for info command and get_lifecycle. Fixes #722 : basename fonction was broken for local files for windows as the os path separator is not the same on linux. Aws sometimes uses 301 code for redirects + redirect to aws base host. Fix signature v4 when aws redirect to aws base host. Fixes #778 : Workaround for aws sending early errors before requests are completed + support for http expect continue feature. Updated maintainer in setup.py. Fixes #797 : unicodisation missing for exclude patterns coming from files. Fixes #800 (second part): "cfg.include" was not checked like is cfg.exclude to use the non memory optimal del subcmd. Fix for part of #660: fix detection for "sync single-file.txt s3://mybucket/single-file.txt" Fix to supporting python 1.6. Use "List" for http methods and not a kind of dict. Backport more code from httplib for compatibility with python 2.6 Fix previous patch on custom httpconnection, it is _send_output and not send_output. Attemps to fix last issues of #769 . Initial version of a .travis.yml and it's .travis.s3cfg configuration for CI with travis-ci. custom_httplib27 - Missing symbols in python 2.6 Missing bit in previous commit. [travis] Fixes allow_failures not being strings. [travis] Added run-test-minio.py + have travis run it. Added travis-ci build status of master in the github displayed readme. Merge pull request #806 from shaform/py3k Merge pull request #808 from robd003/master Merge pull request #809 from shaform/py3k Fixes #810 : Also handle redirects for path styles syntax. Fixes #816 : Fix recv_file 3xx,4xx, handlers + add region extraction of 3xx redirections. Fixes #798 : HEAD request can't parse xml response to get redirection url. Fix missing imports in custom_httplib27. Move urlencode func into Utils and fix encoding issues when having uid groupid with special chars. Small cleanup in S3.py Py3 support: remove usage of deprecated has_key. Py3 support: fix SortedDict (keys() are now views and iterators need __next__) Update README to indicate python versions supported by s3cmd. Changed unicodise/deunicodise wrappers to fit with python3 string model. Python3 support: Fixed stderrr, stdout unicode output. Remove some code that was there for supporting py version < 2.6 Use the stdout/stderr buffer directly instead of detaching. Python3 support: fix error reporting at import time and unicode. Python3 support WIP: Added Custom_httplib3x to patch python3 version of httplib as it is done for py2. [Py3 WIP] Fix for compatibility. PY3 support: fixed S3Uri and Exceptions regarding unicode/str compatibility. Missing change that was needed for custom_httplib functions... PY3 support: more unicode related things. PY3 support: long was removed in py3. Replace by int PY3 support: ensure to use the adequate / sign for division with python3. PY3 support: Added a warning at runtime to warn of WIP PY3 support: Added silent versions for unicodise/deunicodise funcs Small cosmetic changes in Custom_httplib(s) Remove legacy code. s3cmd is now only compatible py >2.6 Py3 support: Reworked request/signature regarding unicode/bytes mix. Py3 support: signature v2 - limit list of subresources url that are signed Py3 support: ensure that request body are encoded when "unicode" strings Signature v2: add "cors" to list of subresources to include in signature Improve get_bucket_location to be able to work all the time with aws Use pretty print everywhere in debug for request responses Getlocation for each type of request when region is not set and signature v4 used PY3 support: more fixes for str/unicode related issues. Typo fix in debug log message Merge pull request #837 from alexjh/allow-skip-tests Merge pull request #851 from burstworks/master PY3 support: use getitem instead of getslices for slices PY3 support: Fix for regexp/unicode in runtest-minio Use io buffers everywhere for files or stdin/stdout + Use custom stream_name attribute. Fixes signurl (v2) for python3. run-tests-minio: re-activated a test that is now working. Urlencode string only when formatting the url and not earlier. Get ride of the "extra" parameter for requests and use a clean "params" dict for all the query strings. Ensure to not have "{}" in functions parameters to be safe. In the v4 canonical query string, parameter without values should still have an equal sign. Fixes signature v2 invalid call to format_param_str. PY3 support: Fixes acl.py use of repr. Remove bad duplicated code. Little performance + memory improvement. + Fixes randomless in auto tests. Typo fixes. Sort headers in requests to always have the same order. Fixes failing tests with special chars for PY3 Small changes on run-tests.py Change .travis.yml now all tests are passing with minio. Merge pull request #870 from yossi-stratoscale/issue-869 Merge pull request #858 from chauffer/patch-1 Documentation: Fixes #872 . Updated following the proposal of @cthorley. Merge pull request #857 from guygurari/cp-mv-bug Fix error message to be accurate with all cases + Fixed auto tests for last change. Update version to 2.0.0 Update setup.py classifiers to indicate that Python3 is supported. Changed version to 2.0.0+ for github nightly versions Fixes #887 - Invalid code inside cf __str__ breaking cf invalidation. Fixes #889 - info output cosmetic fix. Fixes #885 - Specific error message when getting policy by non owner. Small cleanup of legacy checks in Config.py Fixes #888 - Fixes cfinvalinfo broken in 2.0.0 Fixes #886 - Don't force to have a / on last parameter for "modify" command. Fixes #890 - Missing json import after cleanup patch. Fixes #892 - Ensure to add "LICENSE" file to release tarballs. Fixes #894 - Detect and report error 403 in getpolicy for info command Fixes #895 - Refactored py3 source to be byte-compilable with py2 to fix warnings during "setup.py install" Remove python3 support warning. [PY3 support] Fixes --dump-config broken because of "verbosity" Cosmetic - use "with" instead of an open/close Fix encoding issues with config files + PY3 support for --configure Use "unicode" for config attributes Small rework for pre existing hash cache file detection PY3 fix: Open/Save hash cache file in "binary" mode Fix for non string cmdline parameters being ignored Cleanup: Don't have anymore cfg as a global Fixes missing local cfg in copy/move Fixes last missing local cfg Fixes #905 - Fixes https with proxy when no port defined for server. Fixes #908 - Fixes unicode in req headers for py2 and py3 Merge pull request #917 from thelan/master Fixes #921 - fixes gpg_command for python3 Fixed #922 - getenv return unicode in py3 and bytes in py2 Merge pull request #923 from harshavardhana/fix-etag Fix unicode issue in report in the case of an error Fixes #920 Always treat hostnames in lower case. Signature issues. Merge pull request #925 from jmaitrehenry/marker Fix #922 - Handle libmagic missing on windows Fixes getBucketFromHostname not using "lowercase" host_bucket. Avoid crashing python when encountering a CertificateError Update version to 2.0.1 Fixes sourceforge mailing list addr in RELEASE_INSTRUCTIONS file. Update version to 2.0.1+ for MASTER dev. Fixes #929 - strip namespace well even without xml declaration Fixes #933 Fix dst copy when the destination file was modified Merge pull request #938 from Nossnevs/feature/fix_nr_of_paths Fixes #525 - Fixed no invalidate of remotely copied files during sync Fixes error reporting for import error and utf8 characters as cmdline arguments Added clarification comment for the socket_timeout configuration value OS limit Partial fix for #941 - Retry for all kind of requests in case of OS network timeout events Fixes #941 - Restore retries for some 400 errors like RequestTimeout Partial fix for #913 - Only throttle retries for s3 slowdown errors Fixes #913 - Fix limitrate to recover of sudden changes in bandwidth and to not slow down too much the transfers Merge pull request #959 from mouchar/2.0.1 Little help clarification for --delete-after. Fixes #958 Merge pull request #947 from alexallah/master Addition to #947 - ensure it works with py2. aws requests always returns utf-8 encoded data Merge pull request #940 from bcl/master-acl_public Fixes bug created by #940 - cmdline set values are real booleans Complete #940 - upload_id, content_disposition and content_type default value is now "" and not anymore None Merge pull request #968 from LocutusOfBorg/avoid-distutils Merge pull request #970 from radeksimko/b-exclude-connection-header Merge pull request #975 from cybojanek/add_restore_subresource Merge pull request #966 from Trane9991/master Merge pull request #973 from mcv21/nulldisplay Fixes #960 - Do a retry if exc has errno but errno is None. Python3 support. Fixes auto-test following recent aws changes regarding bucket names. See: #957 Fixes #967 - In addition with patch for #960, ensure that error logging with not crash Update version to 2.0.2 Fixes #987 - Fixes using IAM with python3. Config options should be unicode. Fixes #987 - Before py36, py3 json was only able to load unicode str Merge pull request #986 from vpresnyak/output_alignment_fix Complete the fix #986 - To satisfy py2 and p3, there should be float before and after the division Issue #956 - Fixes size reporting using k instead of K as it a multiple of 1024 Fixes #956 - fix size and alignement of du and ls output reporting Merge changes of release 2.0.2 Merge pull request #994 from harshavardhana/no-need Merge pull request #1007 from hrchu/patch-1 Merge pull request #1010 from starrify/fix-minor-help-message-typo Convert anything given to getTreeFromXml to utf8 bytes if needed Fixes #1006 - fixes multiple python3 encoding issues with cloudfront functions Merge pull request #1021 from offshore/patch-1 Merge pull request #1033 from junousi/public_url_https Fixes #1006 - missing return in previous commit 222af5d8ce90bbb2f7784360c019f5a6af2ef8bd Merge pull request #1042 from serprex/master Merge pull request #1061 from k0ste/help Merge pull request #1063 from timgates42/bugfix/typo_unnecessarily Change version to development 2.0.2+ setup.cfg: set universal bdist_wheel flag setup.py: update meta files with support for recent python versions Merge pull request #1011 from zmedico/signurl_use_dns_based_on_bucket_host travis-ci: Disable tests for python 2.6 as not supported anymore by travis-ci default ubuntu distribution Merge pull request #1071 from nitrocode/patch-1 Complete the previous change to rename INSTALL file to INSTALL.md Merge pull request #1067 from NotTheDr01ds/bug/1066 Merge pull request #1068 from NotTheDr01ds/feature/run-tests-in-docker run-tests.py: Don't fail ACL tests if public_url_use_https is used svnignore: update to match latest gitignore content Little cleanup. TODO is out of date and obsolete Add a .dockerignore because of this stupid docker that does not know how to exclude files he has no business with Update run-tests.docker.README.md with a small display fix Create FUNDING.yml Factorize the region detection inner request and stop retrying requests that fail the ssl certificate checks Fixes #1059 - Hack for SignatureDoesNotMatch error when host port 80 or 443 is specified, due to stupid servers Merge pull request #1014 from Quiq/master Merge pull request #1032 from vainu-arto/make-connection-pooling-configurable run-test-*: Minor fixes Small cleanup Fixes #377 - Avoid crashing when a file disappeared during cache generation Fixes #1080 - url encode bucket_name for path-style requests. Fixes an issue with CephRGW and the tenant feature. Fixes #1081 - Don't force signature v4 if bucket_name uses special chars Fixes #1082 - Allow to use signature v4 only, even for a command with no target bucket specified Fixes #1054 - Relax limitation on special chars for --add-header key names Fixes #1020 - Update storage-class cmd help msg with new aws s3 classes Fixes open file descriptors leaking by not being closed on time Added a comment about how to escape the comment char in exclude files Fixes #989 - in aws_credential_file, fixes read_string only available with py3 + refactor for possible unicode issues Fixes #847 - set a few str as unicode to fix UnicodeDecodeError with python2 when doing a sync remote 2 remote Merge pull request #1084 from prabhakar267/master Fixes py headers to use system default python and not force python2 anymore setup.py: Update authors and cleanup Update version to 2.1.0 Change version to development 2.1.0+ Fixes #1087 - fixes ValueError when using more than one ":" inside add_header in config file Fixes #1089 - Ensure that KeyboardInterrupt are always properly raised Update RELEASE_INSTRUCTIONS with the new procedure to upload to Pypi Rework and cleanup of MultiPart and MultiPartCopy (following the merge of PR #74) Add progress info output for multipart copy and current-total info in output for cp, mv and modify Move s3_quote to Utils module Multipart copy - lower multipart copy chunk size to 1 GiB Move cmd - take care in that copy s3 xml result can be a multipart copy result Url encode source for copy/mv/modify to support all special/foreign character names in object names Cosmetic cleanup MultipartCopy avoid retrieving again object info when we already know its size Ensure to catch and display server errors for cmd cp,mv,modify errors Cleanup - no message was set in 2 cases for S3UploadError MultiPart - Fixes extra label issue with stdin as source MultiPartCopy - Also copy acl like direct copy MultipartCopy - Preserve metadata of the object that is copied Fixes #1091 - Ugly hack to have modify work with files bigger than 5Gb Allows to set mime-type (ie content-type) also for remote copies also Allows to set public ACL with "modify" to match "copy" Remove legacy code for dreamhost. I guess that they are now supporting acl Small cleanup of _build_attr_header Fixes typo/failure in multipart cp when progress meter is off Refactored object_copy to avoid duplicated code in object_modify Fixes #685 - Grammatical correction a debug message Metadata for copy should now be consistent for multipart or not and for aws or not cases Fix remote_copy to ensure that meta-s3cmd-attrs will be set based on the real source and not on the copy source Fixes _build_attr_header for case of wrong old value being use for wrong attribute Simplify the code to use _build_attr_header in remote_copy Fixes #1088 - Handle servers indicating (lying) that listObject result is truncated, when it is not the case Fixes #1090 - Be more resilient to a server sending random data in ?lifecycle response Fixes #1093 - Fixes error encountered with Minio send empty content instead of a valid ?requestPayment response. Merge pull request #1051 from chazer/ssl-auth Fixes #1094 - Fixes typo bug + mini cleanup Fixes #1096 - Add http error 429 to the retryable errors Fixes #1101 - "files-from" option was resulting in an AttributeError with Python3 Cleanup of a few lines ending with a ";" Cleanup - Remove unreachable code and an useless test Fixes #1113 - Fixes report_exception wrong error when LANG env var was not set Fixes #346 - Add support for listing and resume of multipart uploads of more than 1000 parts Utils/parseNodes: cleanup - avoid returning empty dicts inside the result list Ref #1114 - Add a time based expiration for idle pool connections in order to avoid random broken pipe errors Fixes #1114 - Reduce connection_max_age to 5s as it looks like that aws closes https conns after around 6s Merge pull request #1118 from Patouche/issues/1117 Merge pull request #1119 from Patouche/fix/datetime-tz Merge pull request #1120 from anbnyc/patch-1 Merge pull request #1124 from tirkarthi/fix-warnings Merge pull request #1137 from ondrejbudai/python39-fixes Merge pull request #1141 from onagurna/master Merge pull request #1145 from xrmx/cleanup_check_md5 travis.yml : Add python 3.9 to the ci test matrix Merge pull request #1144 from k3ntako/master Change import to have the py3 import first Merge pull request #1160 from siakaramalegos/patch-1 Merge pull request #1165 from FFKL/master Merge pull request #1163 from Rjerk/wip-fix-empty-lc-prefix Merge pull request #1152 from orztt/wip-private-acl Change dev version to 2.1.0-dev to respect PEP 440 that is now enforced by pip Fixes #1125 - avoid infinite loops when files are truncated during hashsum or upload Config: Get token expiration from AssumeRoleWithWebIdentityResult response Minor fixes in s3cmd Create a new BaseUtils module for helper functions that don't need access to Config Ensure that "https" url namespaces are also stripped in xml Complete work for AssumeRole And AssumeRoleWithWebIdentity support Use explicit raw strings for regexp patterns in Config Cleanup: Factor unicodise/deunicodise to be reused inside Config Fixes #1190 - Set properly PARTIAL exit code for object_get when needed Merge pull request #1199 from arnaudj/dev/migrateTravisToGitHubActions CI: Add python 3.10 to test matrix Merge pull request #1193 from rvanlaar/patch-1 Merge pull request #1204 from akete/issue-1203 Merge pull request #1198 from sebvajda/master Fixes #1201 - Raise an explicit error when only env var AWS_ACCESS_KEY is set Update version to 2.2.0 Francois Gaudin (2): Fix custom header support for cp Modify command equivalent to a self copy to modify metadata Francois Lebel (1): Fix the example for recursive listing. François Wirion (1): Fix missing slash in directory semantics documentation George Hickman (2): Don't include python files in the manifest. Add a makefile for releasing George Melika (1): Added a --disable-content-encoding to disable adding the content-encoding header to the pushed objects Gianfranco Costamagna (2): Fix typo (overriden -> overridden) Add fallback code to make it compatible with Python2 implementation (we could simplify by adding whichcraft, but a new dependency just for one which command seems an overkill) Gianni Chiappetta (1): Always attempt to set the content-type Giorgos Logiotatidis (1): If response['headers'] is empty, then don't update dict. GuillaumeCisco (3): Fix app data directory Add .idea directory to gitignore Add fallback for old windows versions Guy Gur-Ari (4): run-tests.py: use lowercase bucket name don't allow mv/cp of multiple files to single file raise exception for missing '/' during mv, cp remove unreachable code Harshavardhana (5): s3cmd: Fix s3 path style signature v4 handling. Missing etag in HTTP headers, handle gracefully. build: Do not throw exceptions if no arguments were passed for setup.py Don't need to compute ETag md5sum match if it is not a real md5sum Do not convert all regions to lower case Haruka Iwao (1): * S3/Config.py Get user input aws access key, secret and gpg passphrase out of debug format string James Brown (1): Respect the $TZ environment variable to show local times if requested Jan Kasiak (2): Change proxy SSL URI to relative Add restore subresource to V2 signature Jason Dalton (8): Replaced mktemp with alternative temp file generation based on secure randoms in order to avoid owner-restricted permission issues with mktemp files Added cleanup of temp file if error or keyboard interrupt occurs during transfer. cleanup Revert "cleanup" Revert "Replaced mktemp with alternative temp file generation based on secure randoms in order to avoid owner-restricted permission issues with mktemp files" Set permission on copied file based on umask. Updated references to temp file in exception handlers Relaxed timeout for http connection during IAM role lookup Jason Godfrey (1): Quick fix for cfcreate and cf modify not working. These commands would generate a KeyError for 'Origin'. (Bug 3536626) Jens Braeuer (2): Added support for "Static Websites" Remove recursion detection for symlinks. Jerome Leclanche (11): Avoid catching SystemExit in a try block Add __repr__ for S3Uri class Add S3.MultiPart module with basic functionality Make use of S3.MultiPart functionality by default Rename MultiPartUpload.bucket into MultiPartUpload.s3 Implement multipart threading Use the ThreadPool interface to thread multipart uploads and return a proper response Dynamically increase the chunk size depending on the file size Add the --enable-multipart option to s3cmd Add multipart_num_threads and multipart_chunk_size to Config Properly pass multipart enabling, default enabled on > 100MB Joe Fiorini (1): Added command to set bucket access policy Joe Mifsud (2): Add support for invalidating multiple CF distributions with --cf-inval when the bucket is the origin for multiple distributions Switching to set() Jose Fernandez (magec) (2): Make use of TMP variable to change tempdir Document Environment variables Josep del Rio (4): When using default index, invalidate path Corrected the new flags for CF index invalidation Removed debug messages, make sure default index is defined Fixed exception for root document Josh C. Wilson (1): move progress_meter default setting down to Config class Jukka Nousiainen (2): Set public_url() protocol based on use_https Parameter for public URL protocol Julien Maitrehenry (1): Change the marker value for NextMarker in bucket_list_streaming Julien Recurt (1): Adding option signurl_use_https for signurl to use https prefix Ka-Hing Cheung (1): enlarge write and read buffers Karl Matthias (1): Add support for putting arbitrary additional headers in s3cfg file. Karsten Sperling (15): Use python-magic for guessing MIME types if available Catch the right exception, doh. Added example /etc/magic file with better support for web file types Add support for setting Content-Encoding header where it can be guessed, in particular Content-Encoding: gzip is handled. Flush stdout after logging messages, in case buffered pipes are involved Merge branch 'master' of git://github.com/s3tools/s3cmd Added example /etc/magic file with better support for web file types Add support for setting Content-Encoding header where it can be guessed, in particular Content-Encoding: gzip is handled. Flush stdout after logging messages, in case buffered pipes are involved Fix up code that was merged incorrectly Merge branch 'mime-guessing' Add -M short option for --guess-mime-type Fix ValueError when magic module is not present Add support for http://pypi.python.org/pypi/filemagic version of python-magic Merge branch 'mime-guessing' Karthikeyan Singaravelan (1): Fix deprecation warnings due to invalid escape sequences. Kelly McLaughlin (2): Handle empty return bodies when processing S3 errors. Reuse a single connection when doing a bucket list Kentaro (2): Update base64.encodestring for Python 3.9 compatibility encodestring() method was removed from base64 in Python 3.9. Update base64 import to support Python 2 and 3 Kevin Daub (5): Fixed typo in output messages. Added --server-side-encryption as an option and adds the header to put and put-copy. Reverted out manpage update Added Server-Side Encryption (SSE) to output of --info Fixed formatting Kiana McNellis (2): Fixed bug where the error report will not show if there was an import error due to dependency on unicodise which was not successfully imported Changed the exit code for import errors to not rely on dependency of EX_GENERAL being defined when not imported Koen Rouwhorst (1): Use HTTPS by default Konstantin Shalygin (1): s3cmd: python 3.8 compat Krzysztof Warunek (1): support endpoint with custom path Lance Batson (1): fix issue with error display after config modify Leandro Ostera (1): Update s3cmd --configure Liu Lan (1): fix get bucket info error when LC use the filter element Luke Winslow (1): Fixed issue with mixed path separators with s3cmd get --recursive on Windows. Marco Jakob (7): Get the access token from the environment Do not refresh role Do not refresh role Add option for token-refresh Add option for token-refresh Add option for token-refresh Add option for token-refresh Mark Titorenko (2): Generalize wildcard certificate forgiveness Updated based on feedback. Masashi Ozawa (2): add "limit" option to "ls" and "la" commands to return the specified number of objects instead of returning all objects in the bucket. Enable to configure "host_base" as "S3 Endpoint" in interactive configuration. Matt Domsch (485): Apply excludes/includes at local os.walk() time add --delete-after option for sync add more --delete-after to sync variations Merge remote-tracking branch 'origin/master' into merge Merge branch 'delete-after' into merge add Config.delete_after Merge branch 'delete-after' into merge fix os.walk() exclusions for new upstream code Merge branch 'master' into merge Merge branch 'master' of git://github.com/s3tools/s3cmd add --delay-updates option finish merge Handle hardlinks and duplicate files hardlink/copy fix remote_copy() doesn't need to know dst_list anymore handle remote->local transfers with local hardlink/copy if possible sync: add --add-destination, parallelize uploads to multiple destinations add local tree MD5 caching fix getting uid Apply excludes/includes at local os.walk() time add --delete-after option for sync add more --delete-after to sync variations add Config.delete_after fix os.walk() exclusions for new upstream code add --delay-updates option Handle hardlinks and duplicate files hardlink/copy fix remote_copy() doesn't need to know dst_list anymore handle remote->local transfers with local hardlink/copy if possible sync: add --add-destination, parallelize uploads to multiple destinations add local tree MD5 caching fix getting uid HashCache: add missing break during purge merge manpage conflict] Merge branch 'cache-local-md5' into hardlink-fixes sync: refactor parent/child and single process code Merge branch 'econnell/master' into econnell-merge merge relistan/master Merge remote-tracking branch 'brendano/master' into brendano-merge Merge remote-tracking branch 'ksperling/master' into ksperling-merge Merge remote-tracking branch 'michaeltyson/master' into michaeltyson-merge Merge remote-tracking branch 'joseprio/master' into joseprio-merge Merge remote-tracking branch 'ghickman/packaging' into ghickman-merge Merge remote-tracking branch 'mattswe/patch-1' into mattswe-merge Merge remote-tracking branch 'joefiorini/access-policy' into joefiorini-merge Merge remote-tracking branch 'Roguelazer/tz_support' into Roguelazer-merge Merge remote-tracking branch 'godfreja/fix_cf_keyerror' into godfreja-merge Merge remote-tracking branch 'manos/master' into manos-merge Merge remote-tracking branch 'ghickman/manifest-cleanup' into merge fix mime detection for Fedora python-mime fix multipart uploads fix --cf-invalidate after merge cannot use both --cf-invalidate and --add-destination note mdomsch merge branch in version string Merge remote-tracking branch 'ejnoyes/master' into test-merge Merge remote-tracking branch 'spikegronim/fix-hang-bug' into merge Merge remote-tracking branch 'tycho/master' into merge finish merge fix up failed merge of --delete-after-fetch update mdomsch-merge version string fixup trivial merge conflict remove unused and incorrect function hash_file() wrap get_md5() with try/except IOError remove debug output that mixes unicode and ascii strings poorly be verbose when both hardlinking and copying of local files fails update_acl(): fix output message Handle local hardlink and copy failures by downloading the objects add output on remote copy, fixes test cases 27 and 28 add --files-from=FILE to allow transfer of select files only accept --files-from=- to read from stdin add --files-from to manpage use --files-from only on local source (not remote, not dest) fix sync --dry-run traceback, bug #124 fix single file sync local2remote traceback, bug #128 sync remote2remote: set target_uri on updated files too, bug 131 Split md5/hardlink stuff out of SortedDict into FileDict fix sync single remote file to existing local file traceback, bug 88 add --max-delete for del and sync commands, fixes bug #155 sync: ignore --delete-removed if no source files, bug #155 provide a better exception / error reporting message, bug #158 Merge branch 'files-from' into merge, pull request #116 Merge remote-tracking branch 'borgenk/master' into merge, PR #143 Merge remote-tracking branch 'yar/master' into merge, PR #147 Merge remote-tracking branch 'aisipos/master' into merge, PR #151 Merge remote-tracking branch 'developmentseed/master' into merge, PR #152 Merge branch 'bug/155' into merge, PR #156 Merge branch 'bug/153' into merge, PR #157 Merge remote-tracking branch 'abuchanan-grio/master' into merge, PR #162 Merge remote-tracking branch 'tjasond/master' into merge, PR #163 add --xattr option referring to file system extended attribute containing md5sums Merge branch 'xattr' into merge fix conditional import xattr module Merge branch 'xattr' into merge push md5_xattr to Config only if xattr module was imported Merge branch 'xattr' into merge Don't emit "cowardly" message unless we have source files & delete files Merge branch 'bug/155' into merge Get md5 from remote attribute header in recursive syncs too Merge branch 'bug/remote-md5sum' into merge FileLists: get md5 from headers Merge branch 'bug/remote-md5sum' into merge move x-amz-meta-s3cmd-attrs handling into S3.py use metadata-stored md5 in bucket list, put, and info Merge branch 'bug/remote-md5sum' into merge look at metadata md5 for get as well Merge branch 'bug/remote-md5sum' into merge s3cmd get to stdout: set file_exists to avoid traceback, bug #175 don't purge the HashCache if using --files-from HashCache: catch KeyError in unmark_for_purge(), bug #194 HashCache and FileDict: ignore files with device = 0 or inode = 0 Merge branch 'files-from' into merge Merge branch 'bug/194' into merge fixup merge Merge remote-tracking branch 'ivebeenlinuxed/master' into merge Merge remote-tracking branch 'reed-sandberg/master' into merge Merge remote-tracking branch 'kuenishi/more_multipart' into merge Merge remote-tracking branch 'scharissis/fix_help_typo' into merge finish merge of #186 Merge remote-tracking branch 'orenhe/simplify_wildcard_handling' into merge Merge remote-tracking branch 'robysath/master' into merge Merge remote-tracking branch 'spartman/remote-copy-fix' into merge remote_copy(): fix value in failed_copy_list Merge branch 'no-mime-magic-option' of git://github.com/radomirml/s3cmd into merge Merge branch 'master' of git://github.com/brandongalbraith/s3cmd into merge Merge branch 'listallmybuckets-error' of git://github.com/leostera/s3cmd into merge Fix for ignored --default-mime-type option Merge branch 'bug/default-mime-type' into merge really merge @edrex https://github.com/s3tools/s3cmd/pull/244 include python version in error dump remove python-2.5-ism when guessing mimetype (bug #218) don't hardlink 0-length files encode filenames to system encoding before calling shutils.copy2() getting the file system encoding is loop invariant Merge pull request #250 from mdomsch/bug/python-version-on-error Merge pull request #251 from mdomsch/bug/218 Merge pull request #253 from mdomsch/bug/unicode-stat Print a useful error message when MemoryError is raised Merge pull request #259 from 36degrees/patch-1 fixes for Windows not including a pwd module fix divide-by-zero when no files were transferred handle OSError: No such file or directory in get_local_filelist Merge branch 'bug/memoryerror' into merge Merge branch 'bug/divide-by-zero' into merge Merge branch 'bug/lstat-removed-file' into merge Merge branch 'bug/windows-no-pwd' into merge Merge git://github.com/robd003/s3cmd into merge --no-check-md5 should disable md5sum calculation, including preserved attributes Merge branch 'bug/217' into merge fix put - s3:// (upload from stdin, #270) Merge branch 'bug/put-from-stdin' into merge os.gete[ug]id() doesn't exist on non-Unix Merge branch 'bug/put-from-stdin' into merge Merge git://github.com/flebel/s3cmd into merge add 'make tarball' and 'make rpm' to Makefile remove useless import of S3.Exceptions reduce "Process files that was not remote copied" to debug() add s3cmd.spec to .gitignore add a friendly error message on UnicodeEncodeError exception report LANG= environment variable on crash handle empty directory on S3 sync remote2local _download: Exception != OSError _download: handle S3Error Merge pull request #92 from ksauzz/support-get-operation-to-multibyte-objectname Ignore S3 objects with empty names Merge pull request #284 from swiftserve/fix_config_file_verbosity set content-type, content-charset, and not content-encoding use ~/.s3cfg in run-tests too move empty object name detection where it belongs delete unused keys_to_remove logic s/0o775/0755/ for python 2.4 compatibility Merge pull request #286 from swiftserve/fix_verbosity_config Merge pull request #291 from Knewton/master Merge pull request #293... [truncated message content] |
From: Florent V. (Sodria) <fl...@so...> - 2020-04-08 00:26:02
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.1.0 has been released on github, sourceforge and PyPi. This release is mostly made of bug fixes and should not contain any breaking change. Thanks again to all the contributors, the testers and the issue reporters. s3cmd-2.1.0 - 2020-04-07 =============== * Changed size reporting using k instead of K as it a multiple of 1024 (#956) * Added "public_url_use_https" config to generate public url using https (#551, #666) (Jukka Nousiainen) * Added option to make connection pooling configurable and improvements (Arto Jantunen) * Added support for path-style bucket access to signurl (Zac Medico) * Added docker configuration and help to run test cases with multiple Python versions (Doug Crozier) * Relaxed limitation on special chars for --add-header key names (#1054) * Fixed all regions that were automatically converted to lower case (Harshavardhana) * Fixed size and alignment of DU and LS output reporting (#956) * Fixes for SignatureDoesNotMatch error when host port 80 or 443 is specified, due to stupid servers (#1059) * Fixed the useless retries of requests that fail because of ssl cert checks * Fixed a possible crash when a file disappears during cache generation (#377) * Fixed unicode issues with IAM (#987) * Fixed unicode errors with bucked Policy/CORS requests (#847) (Alex Offshore) * Fixed unicode issues when loading aws_credential_file (#989) * Fixed an issue with the tenant feature of CephRGW. Url encode bucket_name for path-style requests (#1080) * Fixed signature v2 always used when bucket_name had special chars (#1081) * Allow to use signature v4 only, even for commands without buckets specified (#1082) * Fixed small open file descriptor leaks. * Py3: Fixed hash-bang in headers to not force using python2 when setup/s3cmd/run-test scripts are executed directly. * Py3: Fixed unicode issues with Cloudfront (#1006) * Py3: Fixed http.client.RemoteDisconnected errors (#1014) (Ryan Huddleston) * Py3: Fixed 'dictionary changed size during iteration' error when using a cache-file (#945) (Doug Crozier) * Py3: Fixed the display of file sizes (Vlad Presnyak) * Py3: Python 3.8 compatibility fixes (Konstantin Shalygin) * Py2: Fixed unicode errors sometimes crashing remote2remote sync (#847) * Added s3cmd.egg-info to .gitignore (Philip Dubé) * Improved run-test script to not use hard-coded bucket names(#1066) (Doug Crozier) * Renamed INSTALL to INSTALL.md and improvements (Nitro, Prabhakar Gupta) * Improved the restore command help (Hrchu) * Updated the storage-class command help with the recent aws s3 classes (#1020) * Fixed typo in the --continue-put help message (Pengyu Chen) * Fixed typo (#1062) (Tim Gates) * Improvements for setup and build configurations * Many other bug fixes Shortlog below. Thanks, Florent Alex Offshore (1): Return decoded response in get_policy and get_cors Arto Jantunen (2): Add a close method to the ConnMan api Add option to disable connection pooling Doug Crozier (4): Changed "Buckets list" test to use pbucket rather than hardcoded names. This fixes issue #1066. Allow use of Docker to run test cases with multiple Python versions Fix for 'dictionary changed size during iteration' error on Python 3 with --cache-file (Issue #945) Test cases added for --cache-file EC2 Default User (1): .gitignore: add s3cmd.egg-info Florent Viard (36): Fixes #987 - Fixes using IAM with python3. Config options should be unicode. Fixes #987 - Before py36, py3 json was only able to load unicode str Complete the fix #986 - To satisfy py2 and p3, there should be float before and after the division Issue #956 - Fixes size reporting using k instead of K as it a multiple of 1024 Fixes #956 - fix size and alignement of du and ls output reporting Convert anything given to getTreeFromXml to utf8 bytes if needed Fixes #1006 - fixes multiple python3 encoding issues with cloudfront functions Fixes #1006 - missing return in previous commit 222af5d8ce90bbb2f7784360c019f5a6af2ef8bd Change version to development 2.0.2+ setup.cfg: set universal bdist_wheel flag setup.py: update meta files with support for recent python versions travis-ci: Disable tests for python 2.6 as not supported anymore by travis-ci default ubuntu distribution Complete the previous change to rename INSTALL file to INSTALL.md run-tests.py: Don't fail ACL tests if public_url_use_https is used svnignore: update to match latest gitignore content Little cleanup. TODO is out of date and obsolete Add a .dockerignore because of this stupid docker that does not know how to exclude files he has no business with Update run-tests.docker.README.md with a small display fix Create FUNDING.yml Factorize the region detection inner request and stop retrying requests that fail the ssl certificate checks Fixes #1059 - Hack for SignatureDoesNotMatch error when host port 80 or 443 is specified, due to stupid servers run-test-*: Minor fixes Small cleanup Fixes #377 - Avoid crashing when a file disappeared during cache generation Fixes #1080 - url encode bucket_name for path-style requests. Fixes an issue with CephRGW and the tenant feature. Fixes #1081 - Don't force signature v4 if bucket_name uses special chars Fixes #1082 - Allow to use signature v4 only, even for a command with no target bucket specified Fixes #1054 - Relax limitation on special chars for --add-header key names Fixes #1020 - Update storage-class cmd help msg with new aws s3 classes Fixes open file descriptors leaking by not being closed on time Added a comment about how to escape the comment char in exclude files Fixes #989 - in aws_credential_file, fixes read_string only available with py3 + refactor for possible unicode issues Fixes #847 - set a few str as unicode to fix UnicodeDecodeError with python2 when doing a sync remote 2 remote Fixes py headers to use system default python and not force python2 anymore setup.py: Update authors and cleanup Update version to 2.1.0 Harshavardhana (1): Do not convert all regions to lower case Jukka Nousiainen (2): Set public_url() protocol based on use_https Parameter for public URL protocol Konstantin Shalygin (1): s3cmd: python 3.8 compat Pengyu Chen (1): Fixed: Minor typo in help message for `--continue-put` Prabhakar Gupta (1): Fix INSTALL.md formatting issues Ryan Huddleston (1): With this commit 4 years ago OSError was caught: https://github.com/s3tools/s3cmd/commit/889c5a7ba1ef8fc48f0c1be593e7ddf1e518a408 Tim Gates (1): Fix simple typo: unecessarily -> unnecessarily Vlad Presnyak (1): Fixing proper handling of file sizes. Will correctly return floating point or integer as appropriate. Zac Medico (1): Fix signurl to support path-style bucket access via check_bucket_name_dns_support hrchu (1): Update restore cmd help matteobar (1): Copyright year updated nitro (1): README: brew install and renamed to INSTALL.md |
From: Florent V. (Sodria) <fl...@so...> - 2018-07-15 14:18:33
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.0.2 has been released on github, sourceforge and PyPi. This release is mostly made of bug fixes. Thanks again to all the contributors, the testers and the issue reporters. s3cmd-2.0.2 - 2018-07-15 =============== * Fixed unexpected timeouts encountered during requests or transfers due to AWS strange connection short timeouts (#941) * Fixed a throttle issue slowing down too much transfers in some cases (#913) * Added support for $AWS_PROFILE (#966) (Taras Postument) * Added clarification comment for the socket_timeout configuration value OS limit * Avoid distutils usage at runtime (Matthias Klose) * Python 2 compatibility: Fixed import error of which with fallback code (Gianfranco Costamagna) * Fixed Python 3 bytes string encoding when getting IAM credentials (Alexander Allakhverdiyev) * Fixed handling of config tri-state bool values (like acl_public) (Brian C. Lane) * Fixed V2 signature when restore command is used (Jan Kasiak) * Fixed setting full_control on objects with public read access (Matthew Vernon) * Fixed a bug when only one path is supplied with Cloudfront. (Mikael Svensson) * Fixed signature errors with 'modify' requests (Radek Simko) * Fixes #936 - Fix setacl command exception (Robert Moucha) * Fixes error reporting if deleting a source object failed after a move (#929) * Many other bug fixes (#525, #933, #940, #947, #957, #958, #960, #967) Important info: AWS S3 doesn't allow anymore uppercases and underscores in bucket names since march 1, 2018 Shortlog below. Thanks, Florent Alexander Allakhverdiyev (1): Fix Python 3 bytes string encoding when getting IAM credentials Brian C. Lane (1): Fix handling of config tri-state bool values (like acl_public) Florent Viard (18): Fixes sourceforge mailing list addr in RELEASE_INSTRUCTIONS file. Update version to 2.0.1+ for MASTER dev. Fixes #929 - strip namespace well even without xml declaration Fixes #933 Fix dst copy when the destination file was modified Fixes #525 - Fixed no invalidate of remotely copied files during sync Fixes error reporting for import error and utf8 characters as cmdline arguments Added clarification comment for the socket_timeout configuration value OS limit Partial fix for #941 - Retry for all kind of requests in case of OS network timeout events Fixes #941 - Restore retries for some 400 errors like RequestTimeout Partial fix for #913 - Only throttle retries for s3 slowdown errors Fixes #913 - Fix limitrate to recover of sudden changes in bandwidth and to not slow down too much the transfers Little help clarification for --delete-after. Fixes #958 Addition to #947 - ensure it works with py2. aws requests always returns utf-8 encoded data Fixes bug created by #940 - cmdline set values are real booleans Complete #940 - upload_id, content_disposition and content_type default value is now "" and not anymore None Fixes #960 - Do a retry if exc has errno but errno is None. Python3 support. Fixes auto-test following recent aws changes regarding bucket names. See: #957 Fixes #967 - In addition with patch for #960, ensure that error logging with not crash Gianfranco Costamagna (1): Add fallback code to make it compatible with Python2 implementation (we could simplify by adding whichcraft, but a new dependency just for one which command seems an overkill) Jan Kasiak (1): Add restore subresource to V2 signature Matthew Vernon (1): Allow setting full_control on objects with public read access Matthias Klose (1): Avoid distutils usage at runtime Mikael Svensson (1): Fixes bug when only one path is supplied. Before it displayed the nr of characters in the string. Radek Simko (1): Avoid passing 'connection' header to 'modify' requests Robert Moucha (1): Fixes #936 - Fix setacl command exception Taras (1): replace `Config()` with `self` Taras Postument (6): Issue #906 add support for $AWS_PROFILE make it work with python2 Completed some PR tasks:- better imports handling- more debug logging- better error logging- use `Config().update_option()` to set keys pr tasks: - unicodize env variables as well - treat each param from aws credentials as optional - print errors from reading aws credentials file as warnings add helper function to read the deprecated credentials format add more comments |
From: Florent V. <flo...@se...> - 2017-10-21 20:21:51
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.0.1 has been released on github, sourceforge and PyPi. This release is mostly made of bug fixes. A lot of regressions for Python 3 have been fixed and Python 3 support should now be good enough. Thanks again to all the contributors, the testers and the issue reporters. s3cmd-2.0.1 - 2017-10-21 =============== * Support for Python 3 is now stable * Fixed signature issues due to upper cases in hostname (#920) * Improved support for Minio Azure gateway (Julien Maitrehenry, Harshavardhana) * Added signurl_use_https option to use https prefix for signurl (Julien Recurt) * Fixed a lot of remaining issues and regressions for Python 3 (#922, #921, #908) * Fixed --configure option with Python 3 * Fixed non string cmdline parameters being ignored * Windows support fixes (#922) * Don't force anymore to have a / on last parameter for the "modify" command (#886) * Removed the python3 support warning * Detect and report error 403 in getpolicy for info command (#894) * Added a specific error message when getting policy by non owner (#885) * Many other bug fixes (#905, #892, #890, #888, #889, #887) Shortlog below. Thanks, Florent Florent Viard (35): Changed version to 2.0.0+ for github nightly versions Fixes #887 - Invalid code inside cf __str__ breaking cf invalidation. Fixes #889 - info output cosmetic fix. Fixes #885 - Specific error message when getting policy by non owner. Small cleanup of legacy checks in Config.py Fixes #888 - Fixes cfinvalinfo broken in 2.0.0 Fixes #886 - Don't force to have a / on last parameter for "modify" command. Fixes #890 - Missing json import after cleanup patch. Fixes #892 - Ensure to add "LICENSE" file to release tarballs. Fixes #894 - Detect and report error 403 in getpolicy for info command Fixes #895 - Refactored py3 source to be byte-compilable with py2 to fix warnings during "setup.py install" Remove python3 support warning. [PY3 support] Fixes --dump-config broken because of "verbosity" Cosmetic - use "with" instead of an open/close Fix encoding issues with config files + PY3 support for --configure Use "unicode" for config attributes Small rework for pre existing hash cache file detection PY3 fix: Open/Save hash cache file in "binary" mode Fix for non string cmdline parameters being ignored Cleanup: Don't have anymore cfg as a global Fixes missing local cfg in copy/move Fixes last missing local cfg Fixes #905 - Fixes https with proxy when no port defined for server. Fixes #908 - Fixes unicode in req headers for py2 and py3 Merge pull request #917 from thelan/master Fixes #921 - fixes gpg_command for python3 Fixed #922 - getenv return unicode in py3 and bytes in py2 Merge pull request #923 from harshavardhana/fix-etag Fix unicode issue in report in the case of an error Fixes #920 Always treat hostnames in lower case. Signature issues. Merge pull request #925 from jmaitrehenry/marker Fix #922 - Handle libmagic missing on windows Fixes getBucketFromHostname not using "lowercase" host_bucket. Avoid crashing python when encountering a CertificateError Update version to 2.0.1 Harshavardhana (1): Don't need to compute ETag md5sum match if it is not a real md5sum Julien Maitrehenry (1): Change the marker value for NextMarker in bucket_list_streaming Julien Recurt (1): Adding option signurl_use_https for signurl to use https prefix matteobar (1): Added note on new Python 3+ support |
From: Florent V. <flo...@se...> - 2017-06-27 00:10:34
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.0.0 is now available on github, sourceforge, and PyPi. After almost 1 year and half, and the contributions from 23 different developers, this release is an important milestone because it adds the support for Python 3! This is my first release and so I would like to thanks Matt Domsch for his support and all the contributors, the testers and the issue reporters. s3cmd-2.0.0 - 2017-06-26 =============== * Added support for Python 3 (Shaform, Florent Viard) * Added getlifecycle command (Daniel Gryniewicz) * Added --cf-inval for invalidating multiple CF distributions (Joe Mifsud) * Added --limit to "ls" and "la" commands to return the specified number of objects (Masashi Ozawa) * Added --token-refresh and --no-token-refresh and get the access token from the environment (Marco Jakob) * Added --restore-priority and --restore-days for S3 Glacier (Robert Palmer) * Fixed requester pays header with HEAD requests (Christian Rodriguez) * Don't allow mv/cp of multiple files to single file (Guy Gur-Ari) * Generalize wildcard certificate forgiveness (Mark Titorenko) * Multiple fixes for SSL connections and proxies * Added support for HTTP 100-CONTINUE * Fixes for s3-like servers * Big cleanup and many unicode fixes * Many other bug fixes Shortlog below. Thanks, Florent Abhishek Varshney (1): fix: handle empty md5 in find_md5_one Alex Harford (1): Ensure testsuite-out is created before running dd Andrew Gaul (1): Handle missing ETag in subcmd_bucket_list Arturas Moskvinas (1): Do not close stdout, it might be used when file is downloaded and destination is "-" Christian Rodriguez (1): Fix #849 by adding requester pays header to HEAD requests Daniel Gryniewicz (1): Add getlifecycle command David Eckelkamp (1): Let Makefile determine version number from PkgInfo.py file Florent Viard (96): Don't ask to set access_token in run_configure as it should not be used very often. Fix for local2remote in the case of "." as a source. Fixes #706 Fixes #709 : Fixes sync command with windows + cleanup Fix the error message to avoid confusion in a special cases like "no space left on device". Fixes #744 - Ensure to check non None or empty bucket names before calling check_bucket_name. Minor optimization and cosmetic fix in ConnMan. Fix Issue #672 - Patch based on PR #729 proposed by MLopez-Ibanez. Fixes getBucketFromHostname for when host_bucket config has no "bucket" element. Fix the infamous "UnboundLocalError" for conn exception hidding meaningful exceptions Added an explicit error message when the servers hostname are wrong or unresolvable. Avoid having an error when user uses "https://" prefix to host_base... Fixes #741 Issues validating ssl certs when using a custom port for host_bucket address. Real fix for #772. Use urlparse to extract hostnames in connman Follow up on #779, not using a CONNECT "tunnel" for http proxies as it is not the most common case. Fixes #777 : Fix __repr__ returning unicode and sign_v2 not encoded to s3. Fixes recursive delete that was broken by the new "truncated" tag. Fixes orphan var in "du" exception handler. run-tests.py Update expirations tests dates to the future. Fix failure of copy or rename for s3-like servers that don't have support for acl. Fix error of "info" command when s3-like servers don't implement "acl" or another feature. Avoid errors with modify command when s3-like servers don't support acl. Fix possible bug in previous change for info command and get_lifecycle. Fixes #722 : basename fonction was broken for local files for windows as the os path separator is not the same on linux. Aws sometimes uses 301 code for redirects + redirect to aws base host. Fix signature v4 when aws redirect to aws base host. Fixes #778 : Workaround for aws sending early errors before requests are completed + support for http expect continue feature. Updated maintainer in setup.py. Fixes #797 : unicodisation missing for exclude patterns coming from files. Fixes #800 (second part): "cfg.include" was not checked like is cfg.exclude to use the non memory optimal del subcmd. Fix for part of #660: fix detection for "sync single-file.txt s3://mybucket/single-file.txt" Fix to supporting python 1.6. Use "List" for http methods and not a kind of dict. Backport more code from httplib for compatibility with python 2.6 Fix previous patch on custom httpconnection, it is _send_output and not send_output. Attemps to fix last issues of #769 . Initial version of a .travis.yml and it's .travis.s3cfg configuration for CI with travis-ci. custom_httplib27 - Missing symbols in python 2.6 Missing bit in previous commit. [travis] Fixes allow_failures not being strings. [travis] Added run-test-minio.py + have travis run it. Added travis-ci build status of master in the github displayed readme. Fixes #810 : Also handle redirects for path styles syntax. Fixes #816 : Fix recv_file 3xx,4xx, handlers + add region extraction of 3xx redirections. Fixes #798 : HEAD request can't parse xml response to get redirection url. Fix missing imports in custom_httplib27. Move urlencode func into Utils and fix encoding issues when having uid groupid with special chars. Small cleanup in S3.py Py3 support: remove usage of deprecated has_key. Py3 support: fix SortedDict (keys() are now views and iterators need __next__) Update README to indicate python versions supported by s3cmd. Changed unicodise/deunicodise wrappers to fit with python3 string model. Python3 support: Fixed stderrr, stdout unicode output. Remove some code that was there for supporting py version < 2.6 Use the stdout/stderr buffer directly instead of detaching. Python3 support: fix error reporting at import time and unicode. Python3 support WIP: Added Custom_httplib3x to patch python3 version of httplib as it is done for py2. [Py3 WIP] Fix for compatibility. PY3 support: fixed S3Uri and Exceptions regarding unicode/str compatibility. Missing change that was needed for custom_httplib functions... PY3 support: more unicode related things. PY3 support: long was removed in py3. Replace by int PY3 support: ensure to use the adequate / sign for division with python3. PY3 support: Added a warning at runtime to warn of WIP PY3 support: Added silent versions for unicodise/deunicodise funcs Small cosmetic changes in Custom_httplib(s) Remove legacy code. s3cmd is now only compatible py >2.6 Py3 support: Reworked request/signature regarding unicode/bytes mix. Py3 support: signature v2 - limit list of subresources url that are signed Py3 support: ensure that request body are encoded when "unicode" strings Signature v2: add "cors" to list of subresources to include in signature Improve get_bucket_location to be able to work all the time with aws Use pretty print everywhere in debug for request responses Getlocation for each type of request when region is not set and signature v4 used PY3 support: more fixes for str/unicode related issues. Typo fix in debug log message PY3 support: use getitem instead of getslices for slices PY3 support: Fix for regexp/unicode in runtest-minio Use io buffers everywhere for files or stdin/stdout + Use custom stream_name attribute. Fixes signurl (v2) for python3. run-tests-minio: re-activated a test that is now working. Urlencode string only when formatting the url and not earlier. Get ride of the "extra" parameter for requests and use a clean "params" dict for all the query strings. Ensure to not have "{}" in functions parameters to be safe. In the v4 canonical query string, parameter without values should still have an equal sign. Fixes signature v2 invalid call to format_param_str. PY3 support: Fixes acl.py use of repr. Remove bad duplicated code. Little performance + memory improvement. + Fixes randomless in auto tests. Typo fixes. Sort headers in requests to always have the same order. Fixes failing tests with special chars for PY3 Small changes on run-tests.py Change .travis.yml now all tests are passing with minio. Documentation: Fixes #872 . Updated following the proposal of @cthorley. Fix error message to be accurate with all cases + Fixed auto tests for last change. Update setup.py classifiers to indicate that Python3 is supported. Guy Gur-Ari (4): run-tests.py: use lowercase bucket name don't allow mv/cp of multiple files to single file raise exception for missing '/' during mv, cp remove unreachable code Harshavardhana (2): Missing etag in HTTP headers, handle gracefully. build: Do not throw exceptions if no arguments were passed for setup.py Jan Kasiak (1): Change proxy SSL URI to relative Joe Mifsud (2): Add support for invalidating multiple CF distributions with --cf-inval when the bucket is the origin for multiple distributions Switching to set() Krzysztof Warunek (1): support endpoint with custom path Marco Jakob (7): Get the access token from the environment Do not refresh role Do not refresh role Add option for token-refresh Add option for token-refresh Add option for token-refresh Add option for token-refresh Mark Titorenko (2): Generalize wildcard certificate forgiveness Updated based on feedback. Masashi Ozawa (2): add "limit" option to "ls" and "la" commands to return the specified number of objects instead of returning all objects in the bucket. Enable to configure "host_base" as "S3 Endpoint" in interactive configuration. Matt Domsch (2): Catch 'connection reset by peer' (errno 104) and retry pretty print server responses Robert Palmer (1): Added support for S3 Glacier archive retrieval speed options Shaform (11): use print function Fix has_key with 2to3 Fix except with 2to3 Fix numliterals with 2to3 update relative import actually run python3 for tests run python3 for run-tests.py fix relative imports fix imports fix logging fix has_key, numliterals, except, imports on s3cmd Simone Esposito (1): S3/Config: `os.environ.get` -> `os.getenv` Vincent (1): FIX: update s3cmd.1 --disable man entry option Yossi Abraham (1): Properly passing parsed hostname and port marcojakob (3): Remove optparser switches Make _access_token_refresh private and set it to true if access_token was already set Add CLI option for specifying access_token, see PR #694 |
From: Matt D. <ma...@do...> - 2016-01-20 18:25:42
|
s3cmd v1.6.1 is released to github, sourceforge, and PyPi. This primarily fixes bugs from previous releases, especially SSL library challenges introduced by the ever-changing python 2.7.x series SSL support. Thanks to Florent Viard for reviewing a lot of proposed patches. This round we had contributions from 8 different developers. Shortlog below. Thanks, Matt Albert Bendicho (4): Off-by-one on MAX_CHUNKS verification max_chunks now a config option MAX_CHUNKS constant removed, now a config option Added multipart_max_chunks Alexander Todorov (2): Disable hostname checks for SSL connections on RHEL 7.2 In case check_hostname isn't defined catch the error Andrew Gaul (1): Emit storage class in info subcommand Florent Viard (19): Fixes #636. Also remove the temporary file when encountering an S3Error. Fix "info" command for servers not supporting the "request_payer" feature. Based on PR #640 of Qiankunzheng. Fixes #644 : Minor fix to not output "u quoted" invalid parameter in the case of an invalid command. Reworked source and destination input of "remote2local" to cover gracefully more cases. Block "-" (stdin / stdout) to be used in parameters for sync as it has no sense. Only warn for the non regular files that are skipped. In the case of local2remote, raise an error if a source arg is missing. Fix automatic tests for recent changes. Don't fail the copy if a non AWS s3 server doesn't support setting ACLs. Have extra labels coherent when progress is enabled or not and for multipart uploads. Added the --stats option that is similar to the one of rsync. Added "--stats" info for sync commands (remote2local, remote2remote, local2remote) Fix duplicated code for the error alert message. Based on @edunham PR: https://github.com/s3tools/s3cmd/pull/609 Fixes #661 : Don't try to perform a delete if there is nothing to delete. + Restore warning that was in 1.5.2. Fixes #659 : Don't fail if "content-type" field is missing in info like for eucalyptus walrus. Fixes #646: number of "files to update" are not reported in the summary (there was only the new files to be added) Fixes #646 : remote2remote sync - "Updated" files were not reported as to be remotely copied when in dry-run mode. When using --no-preserve, still add 'md5' custom attribute if md5 check is enabled. Fixes #674 : When using AWS_CREDENTIAL_FILE, crash in case of invalid key in .aws/credentials file + updated the code for actual name of the keys. Harshavardhana (1): s3cmd: Fix s3 path style signature v4 handling. Matt Domsch (14): remote copies: copy ACL also (#547) only copy the ACL if --acl-public wasn't specified Allow --requester-pays for PUT commands too (#585) add "+" to PkgVersion so we can tell it's a git checkout Include version at the top of --debug output Update RELEASE_INSTRUCTIONS Add github release instructions Explicitly catch and raise CertificateError in send_request() Validate SSL certificate hostname against target bucket hostname (#598) Don't unconditionally disable hostname checking more hostname fixes, move debugs around Add --host and --host-bucket command line options Allow acl_public = True/False to be in config file Remove newer python-ism from S3/Utils.py Shigeaki Okazaki (1): Warn when skipping non-existing files or symbolic links Teemu Ikonen (2): Fix for #634. Partial get start position was dropped if requested fell back to signature v2 Fix start_position dropped on 307 redirect kobake (1): Ignore RestoreAlreadyInProgress exception on restore command wiof (2): Control MAX_CHUNKS Define MAX_CHUNKS = 10000 |
From: Matt D. <ma...@do...> - 2015-09-18 11:40:07
|
After seven months, and dozens of new features and bug fixes from 24 contributors, s3cmd v1.6.0 is now available. In this cycle, Florent Viard has stepped up to co-maintain the tool, and has done an outstanding job reviewing pull requests, fixing bugs, and doing everything a good maintainer should. Thank you Florent! Here's the high-level overview of all the cool new capabilities (including support for Infrequent Accessed storage announced this morning by AWS!): s3cmd-1.6.0 - 2015-09-18 =============== * Support signed URL content disposition type * Added 'ls -l' long listing including storage class * Added --limit-rate=RATE * Added --server-side-encryption-kms-id=KEY_ID * Added --storage-class=CLASS * Added --requester-pays, [payer] command * Added --[no-]check-hostname * Added --stop-on-error, removed --ignore-failed-copy * Added [setcors], [delcors] commands * Added support for cn-north-1 region hostname checks * Output strings may have changed. Scripts calling s3cmd expecting specific text may need to be updated. * HTTPS is now the default * Many unicode fixes * Many other bug fixes Pick up your copy wherever fine software is distributed: https://github.com/s3tools/s3cmd/releases/tag/v1.6.0 https://sourceforge.net/projects/s3tools/ Full shortlog since v1.5.2 appended. Thanks, Matt Andrew Gaul (3): Horizontally align object policy info Emit user metadata in object info Support signed URL content disposition type Billy Crook (2): Added support for listing StorageClass attribute. Disabled by default. Enable with list_storageclass = True in .s3cfg Adding -l option for long_listing list_storageclass now provided by long_listing and -l. Minor whitespace cleanup. Brady (2): Install instructions for PIP Correct typo in instructions David Liu (1): Fail silently on SIGPIPE Dongmao Zhang (2): add new option --limit-rate to throttle download/upload speed only gettimeofday when limitrate is required Florent Viard (53): Improved handling for errors 501 and 504: Error 501: It means that a feature is not implemented in the server, so no need for useless retries. Error 504: Gateway timeout situation, exit code should indicate a server error. Cleanup: Add prologue to files to state that they are UTF-8 encoded. Fix run-test script to take into account the config file that is specified on the command line. Fixes encoding/decoding issues and so support for all strange characters (chinese, japanese, ...) in file names, folder or path arguments (Should fix bug like #417i) when UTF-8 is not the system def$ Now, in theory, everything that gets in should be converted to unicoded and then things would be converted back to string (user defined encoding) at the outmost parts when dealing with the system. When talking to S3, convert unicode to strings in utf-8 encoding. Remove useless error import Pass2 of "fixes encoding/decoding issues": os.walk replaced to handle unicodise conversion + fixed the removal of most of useless or invalid unicodise/decodise in the code. Fix for issue #432. Read the response from httplib even if there is a 0 byte to read. So httplib doesn't think that there is a response waiting to be read when the next request come. Straightforward to fix thanks to the detailed analysis of Edwintorok. Use deunicodise consistentely in object_copy and object_modify fix encoding : Added encode_to_s3 and decode_from_s3 function + switched to "unicode" call instead of "decode" as it is supposed to be more efficient. Fix wrong exception used for EOF from S3 error during recv_file. (S3Error expect an http response, not a text message) For copy and modify, take care to check if there is an error after the start of the operation (reported by a status 200 but an "Error" root entry in the xml body) In addition, for the move, It is good to check for the CopyObject xml reply to see if it is a success. But, if there is a status of 200 and no "data" in response, don't fail. Other S3 server implementations use only status code and don't return any body when there is no error. Another fix related to fix encoding issues. (file.name / stream.name are bytes, so unicodise them also) Fix invalid file transfer progress info in case of local2remote and remote2remote: If there was failed copy, seq number was preserved but previously uploaded file number was forgotten, thus user could have faced an invalid indication like: 14/8 when there is 6 upload following 6 remote copy failures. Add a link to the known issues page (on github) within the header and footer of s3cmd generated error reports. The goal is to reduce the number of duplicated "bug reports" for known issues. Fixes #528 (https://github.com/s3tools/s3cmd/issues/528). There is a lot of different "python-magic" libraries/versions, all doing different things, so all cases need to be covered. file-5.11 has be switched with "filemagic" as it should be more commonly used. Introspection in each python magic lib has been done to ensure to handle encoding or not only when it is needed. Fixes issue #576 (https://github.com/s3tools/s3cmd/issues/576). EX_SOMEFAILED was still used in a PR that was merged instead of EX_PARTIAL. Merge pull request #572 from fviard/fix_encoding_magic Merge pull request #574 from koenrh/https-by-default Don't slurp whole file while computing SHA256 checksum. Rework of the PR of petr-tichy. When using large --multipart-chunk-size-mb or --disable-multipart, the whole file was read into memory. Merge pull request #580 from Tim-Anderson/error_stop Merge pull request #583 from Tim-Anderson/issue577 Merge pull request #586 from Tim-Anderson/remove_ignore_failed_copy Merge pull request #587 from taz77/master Merge pull request #589 from jcwilson/master Merge pull request #560 from Flipboard/iter Merge pull request #593 from jrschumacher/set-cors Merge pull request #603 from mdomsch/bug/testsuite Following #603 : WARNING OUTPUT FORMAT CHANGE. Changed the format of output strings to be more consistant. Warning, the format of output for a lot of actions was changed. Thus, any automatic string for parsing the output could have to be fixed when using s3cmd after this checkin. Ex: "File hello/world.txt stored as s3://mybucket/world.txt" will now be: "upload: 'hello/world.txt' -> 's3://mybucket/world.txt' Fixes #602 (and complete #603) Fix run-tests.py to have all tests passing against HEAD If s3error message is none, don't print an useless ": None" Return NotFound error with message s3Error: NoSuchBucket when there is no bucket found to delete Raise standard S3Error in S3.py instead of simply exiting. So the normal error logic can't operate on any unexpected error. Avoid to have empty errors in output like: "ERROR: S3 error:" (Because of an 400 InvalidBucketName error returned by a non AWS3 server) Merge pull request #590 from andrewgaul/content-disposition Fix stop_on_error config type Attempt to fix the first bug of issue #600 : When a subfolder "xxx" was selected as source, the sync comparison was done against the destination base whereas it had to be done based on the "xxx" subfolder of the destination. Fixes run-tests following the fix for https://github.com/s3tools/s3cmd/issues/600 Merge pull request #616 from mdomsch/bug/612 Fix bad help info for the "--no-check-certificate" switch. Thanks to @psi29a for proposing the fix. Merge pull request #618 from mdomsch/bug/futurewarning Fixes #592 : Ignore any leading slash before bucket_name in an S3Uri. Original issue: "Running s3cmd ls s3:///foo lists owned buckets instead of emitting an error." Fix for error reporting in case of upload error. (Following the fix for #577) Fixes #581 regarding the reporting of SSL verification issues. Other problems of this issues should already be fixed. Merge pull request #621 from mdomsch/bug/ssl-checking Merge pull request #619 from fviard/second_fix_for_577 Merge pull request #617 from fviard/fix_subfolder_source Merge pull request #623 from mdomsch/bug/testsuite2 Reworked Upload and Download errors to be more consistent. Add detection and handling of different OSErrors like "File name too long." Merge pull request #628 from mdomsch/bug/curl Merge pull request #627 from fviard/upload_download_errors_rework Merge pull request #630 from mdomsch/feature/ia-class Merge pull request #611 from bsapiro/master Added --server-side-encryption-kms-id config option. Following PR #611. Fixes bug in PR #611 François Wirion (1): Fix missing slash in directory semantics documentation Gianfranco Costamagna (1): Fix typo (overriden -> overridden) Giorgos Logiotatidis (1): If response['headers'] is empty, then don't update dict. Josh C. Wilson (1): move progress_meter default setting down to Config class Ka-Hing Cheung (1): enlarge write and read buffers Koen Rouwhorst (1): Use HTTPS by default Matt Domsch (74): signurl should honor Config.use_https Merge branch 'master' of github.com:s3tools/s3cmd into upstream-master fix signurl test retry recv_file() on IOError(EPIPE) Revert "signurl should honor Config.use_https" close http(s) connection after IOError(EPIPE) or other unknown error report_exception() on uncaught IOError before exit accept 'us-east-1' as valid Config.bucket_location in [mb] command Merge pull request #483 from mdomsch/bug/broken-pipe Merge pull request #478 from petrus-jvrensburg/patch-1 Merge pull request #459 from thesues/throttle s3cmd help: --limit-rate works for uploads too Merge pull request #480 from fviard/improve-errors-501-504 Merge pull request #468 from ppe/support-display_name-in-acl-revoke Sign x-emc-* headers (#486) Merge pull request #488 from glogiotatidis/master Ensure content-type header is treated as a string and long int Merge pull request #494 from andrewgaul/emit-user-metadata Merge pull request #493 from andrewgaul/align-policy Merge pull request #496 from fviard/fix_for_432 Merge pull request #495 from fviard/batch_encoding_fixes Merge pull request #504 from LocutusOfBorg/fix-typo Merge pull request #507 from LocutusOfBorg/debian-patches escape - and 'Cache-Control' items in manpage Merge branch 'master' of github.com:s3tools/s3cmd into upstream-master replace /tmp/ with ~/ in manpage examples remove references to donations update manpage for formatting Remove useless imports and unused variables Merge pull request #499 from fviard/fix_copy_modify_check_errors Merge pull request #508 from fviard/fix_encoding_multipart_filename more IOError EPIPE catches Merge pull request #512 from fviard/fix_invalid_progress_info add --requester-pays (#511) add Payer to [info] command output move setting requester-pays into S3Request() init Merge pull request #515 from mdomsch/feature/requestor-pays add [payer] command add requester-pays tests Merge branch 'BCrookAtRA-master' into upstream-master Merge pull request #532 from ddzialak/master [info] use last-modified value, not date note s3:GetBucketLocation permission is needed too Makefile: file is README.md now Merge pull request #537 from flisky/master Merge pull request #538 from kahing/bigger_chunk_size Merge pull request #569 from fviard/common_issues_link Merge pull request #568 from blindroot/master Try using Config.bucket_location if we haven't received a redirect Merge branch 'master' of github.com:s3tools/s3cmd into upstream-master Merge pull request #550 from iceboundflame/10513-sigpipe Merge pull request #571 from wirion/master Downloads: print destination file name not <fdopen> Fix testsuite for recent changes introduce class ExitScoreboard Fix Rename (NoSuchKey) test string tested for fix md5 checks on download of multipart-uploaded files (#604) Merge pull request #605 from mdomsch/bug/604 Fix use of scoreboard Don't check hostnames in SSL certs when --no-check-certificate Fix FutureWarnings from S3/AccessLog.py add --[no-]check-hostname option Allow cn-north-1 region SSL certificate to validate hostname Use already imported ssl.CERT_NONE Add single quotes around progress output filenames run-tests: s/'-'/'<stdin>'/ run-tests: use curl instead of wget more curl cleanups Use test_curl_HEAD() throughout, add --location flag run-tests: detect if curl is on path before using it add --storage-class option Remove unicode chars from --server-side-encryption-kms-id helptext Update copyright year to include 2015 Version 1.6.0 Mike Klaas (13): iterative du, rm bugfix fixed to conform to test suite typo slight cleanup of du code remove errant print statement Code review feedback: * Print all filenames deleted * Use batch mode if max-deletes is set (& remove logic from iterative delete) * Fix formatting of aggregate output of the deletion, and elide unless -v is specified _status() inner funtion wasnt really needed eliminate recurse() function which is unneeded and simplify subcmd_bucket_usage revert unnecessary change remove closure variable capture in batch_del_iterative; fix typo in subcmd_disk_usage stream groups of 1000 (or whatever is available) objects at a time, and delete that group directly fix non-streaming bucket_list() Mikhail Gusarov (3): Fix various typos spot by lintian Return non-zero code from 'put' command if some files failed to upload Do not loop endlessly if removal of bucket content fails Paul Wise (1): Use user home instead of tmp in examples Petri Pellinen (1): Add support for display names in --acl-revoke (#223) Petrus Janse van Rensburg (1): Update README.md Przemyslaw Hejman (1): Added dry-run to fixbucket command Ryan Schumacher (1): Add ability to set, get and del cors policy. Tim Anderson (7): Add stop_on_error Add better exit errors on stop_on_error Return PARTIAL in sync transfers Correct exit status handling for children Error check on multipart upload Change Multipart error so it can continue Replace ignore_failed_copy with stop_on_error Yin Jifeng (1): FIX #535 mark delay-updates obsolete ben.sapiro (1): basic SSE-KMS support dzialak (1): fix simply error in setlifecycle command |
From: Matt D. <ma...@do...> - 2015-02-08 15:14:04
|
I've put out a really quick update, 1.5.2. This fixes one big failure on Ubuntu 14.04, due to their SSL library not yet validating a server's SSL certificate. It also fixes some packaging issues (I had added, and have now removed, python ez_setup, which, as it happens, is more hassle than it's worth). github, PyPi, and SourceForge all updated. Fedora & EPEL testing repos have 1.5.1.2, and aren't affected by the SSL library failure that broke Ubuntu, so I don't intend to update those again. shortlog since 1.5.1: Matt Domsch (9): include ez_setup.py in MANIFEST.in Update version to 1.5.1.1 Change name of README.md in MANIFEST.in Update to version 1.5.1.2 setup.py: add install_requires python-magic make bundled s3cmd.spec.in closer to Fedora spec Handle unvalidated SSL certificate remove ez_setup Update version to 1.5.2 Thanks, Matt |
From: Matt D. <ma...@do...> - 2015-02-04 18:43:57
|
I've pushed out s3cmd 1.5.1.2 today. It's mostly a bugfix release over what was in 1.5.0. The two minor minor point releases were necessary to fix up PyPi packaging bugs that kept 'pip install s3cmd' from working correctly. Sourceforge, github, and PyPi are all updated now. Shortlog: Andrew Gaul (2): Sort s3cmd ls output by bucket name Capitalize command descriptions for consistency Chris Lamb (1): Support relative expiry times in signurl. Luke Winslow (1): Fixed issue with mixed path separators with s3cmd get --recursive on Windows. Matt Domsch (51): remember git push --tags for release fix S3 wildcard certificate checking Add docstring explanation into match_hostname_aws() Handle headers with spaces in their values properly (#460) Don't add --add-headers results into every request Fix lack of SSL certificate checking libraries on older python set content-type header for stdin from command line or Config() Fix multipart abortmp error message fix uploads from stdin (#464) in S3.send_file(), don't conditionalize on stdin Multipart: send 2nd+ chunks from start of buffer add test for put from stdin add multipart test testsuite: add Multipart Put large file from stdin use close_fds = True in calls to Popen() reduce use of os.system() in run-tests stdin tests reduce "Forwarding request to %s' % region" from warning to info Fix directory exclusions (#467) Correct exclude pattern for python 2.4 s/sign_url/sign_url_v2/ sign_url_v2 is in Crypto now, not Utils sign_string_v2 is now in Crypto, not Utils fix signurl run-tests: add sign, signurl tests Don't retry in response to HTTP 405 error (#422) run-tests: add -c <config file> run-tests: add test for [la] command setup.py: use install_requires it's called python-dateutil in pypi Don't crash when a proxy returns an invalid XML error document refresh manpage after capitolization patch move from distutils to setuputils invoke python2 setup.py build ... with python2 sign uploads to pypi update pypi metadata report the actual error XML when we can't parse it. Fix indentation causing failures remove useless debug() and print() message update NEWS for 1.5.1 update PkgInfo and manpage version no longer keeping sourceforge git tree in sync include ez_setup.py in MANIFEST.in Update version to 1.5.1.1 Change name of README.md in MANIFEST.in Update to version 1.5.1.2 Thanks, Matt |
From: S3Express <in...@s3...> - 2014-08-04 11:28:18
|
We have released s3cmd version 1.5-rc1, in preparation for a final 1.5.0 release, with all changes, fixes and improvements implemented since 1.5 beta. Thanks a lot to all for your contributions and fixes. Keep them coming! Special thanks to Matt Domsch for administering and leading the project. And to Michal Ludvig for putting together the 1.5-rc1 release. Year to date, s3cmd has been downloaded 781,248 times from SourceForge. See: http://sourceforge.net/projects/s3tools/files/s3cmd/stats/timeline?dates=2014-01-01+to+2014-07-29 You can download the latest s3cmd version from here: http://s3tools.org/s3cmd thank you, M.Bartoli |
From: Michael L. <ml...@lo...> - 2013-12-02 05:27:26
|
Hi Matt & all Thanks a lot for looking after s3cmd and once again my apologies for not being active in the community. I'm really happy to see s3cmd is alive and well. I have merged the mdomsch/s3cmd/merge branch to s3tools/s3cmd/master and released it as *s3cmd 1.5.0-beta1* (https://sourceforge.net/projects/s3tools/files/s3cmd-1.5.0-beta1.tar.gz). Please download, test and report back your success. Matt - Quite a long time ago I made you an admin of the github repo (s3tools/s3cmd) and just recently also gave you admin access to the SourceForge s3tools project so you can merge directly to the official master and make new releases if you wish. There are a few steps involved, we can discuss that offline if you're keen. Or keep updating your merge branch and ping me when it's ready for a release, I'm sure I can find the time to roll out a release when the code is ready. I just don't have enough time for the day to day participation. Thanks a lot for all your contributions guys, keep up the good work! And special thanks to Matt for putting it all together :) Michal On 27/11/13 04:50, Matt Domsch wrote: > The pull requests have been piling up for several months, with lots of > bugs being fixed by 21 different developers. Michal hasn't had time > to review, test, and merge these, so here's my attempt to help the > process along, in hope of getting more people to test and report success/failure. > > Appended is the shortlog of all changes pulled into my merge branch since > the current upstream-master 9ee3bdd320c8. I've also tried leaving a > comment on each of the existing pull requests. There were a few I > think need a little more work before they can be merged, or that > require Michal's judgment. > > The tree: https://github.com/mdomsch/s3cmd/tree/merge > (HEAD commit ae6030cd33d4) > ZIP archive: https://github.com/mdomsch/s3cmd/archive/merge.zip > Clone: git clone git://github.com/mdomsch/s3cmd.git > (you want the branch called 'merge') > > If this works for you, please leave a comment. If you were > particuarly affected by a bug that is now fixed, _please_ let us know > that as well. > > Special thanks to Will Tinsdeall (@ivebeenlinuxed) for reviewing and > merging six of these patches into his tree for testing this last month. > > Thanks, > Matt Domsch (@mdomsch) > > > |
From: Michal L. <mi...@lo...> - 2013-03-11 00:21:18
|
Hi s3cmd users! Another week passed, another release arrived - *s3cmd 1.5.0-alpha3 <http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha3/s3cmd-1.5.0-alpha3.tar.gz>* The main feature of this release is _Persistent reusable HTTP connections_ - an improvement that brings massive speedup for pretty much all Amazon S3 operations. Up until now s3cmd used to open a new connection for each file upload, download, delete, etc. That resulted in quite a big overhead, especially for short requests like remote file copy or delete. Even worse when HTTPS was turned on - setting up the SSL channel for each and every request was a very resource intensive and time consuming exercise. >From s3cmd 1.5.0-alpha3 on we support a much faster approach - we open the connection once and use it again and again and again without closing and re-opening for each subsequent request. That achieves a significant speedup in all operations. >From my office on a remote Pacific island with slow ADSL and huge latency to S3 I get: - Testsuite run without SSL from 3 min 54 sec down to 2' 54" - that's 26% speedup - Testsuite run with SSL (i.e. HTTPS) from 5' 59" down to 3' 11" = 47% speedup - Uploading 10000 files from 145' 43" down to 57' 43" = 61% speedup - Deleting 10000 files from 121' 41" down to 33' 6" - that's _72% speedup_!! Obviously if you are on a fat pipe with a low latency close to an Amazon datacentre you may not experience such an extreme improvement - I'm keen to hear how much (if at all) the new connection handling helps in your case! With regards to other changes in alpha3 - new switch --quiet / -q to suppress all "normal" output from s3cmd. (Siddarth Prakash) - s3cmd get now honours umask again - for some time the downloaded files were created with 0600 permissions - that's now corrected. (Jason Dalton) - Matt Domsch did a great job fixing small annoying bugs and closing resolved tickets - thanks! Please download, test and report any problems http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha3/s3cmd-1.5.0-alpha3.tar.gz The most up to date sourcecode is available on GitHub: https://github.com/s3tools/s3cmd Also check out our website for further updates: http://s3tools.org/s3cmd Have a nice day! Michal |
From: Michal L. <mi...@lo...> - 2013-03-05 04:49:09
|
Hi guys* **it's **release time again**- welcome s3cmd 1.5.0-alpha2 <http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha2/s3cmd-1.5.0-alpha2.tar.gz> :) *The major new feature in this release is *IAM roles support* thanks to David Kohen and Eric Dowd. Another new feature is the support for managing bucket policies (setpolicy, delpolicy and info for showing the policy) from Kota Uenishi. s3cmd 1.5.0-alpha2 works again with Python 2.4 On top of that we've got many minor fixes and improvements from various contributors. Download from here: http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha2/s3cmd-1.5.0-alpha2.tar.gz The most up to date sourcecode is available on GitHub: https://github.com/s3tools/s3cmd Also check out our website for further updates: http://s3tools.org/s3cmd Please test and report any issues you may find. Enjoy Michal & contributors |
From: Michal L. <mi...@lo...> - 2013-02-19 01:58:10
|
Surprise surprise! After more than a year of neglecting s3cmd and letting it rot unmaintained I thought I should spend some time bringing it back to life. Here you go: s3cmd 1.5.0-alpha1 <http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha1/s3cmd-1.5.0-alpha1.tar.gz> is now available Big thanks to Matt Domsch for integrating many pull requests over the year into his Fedora version of s3cmd. This s3cmd 1.5.0-alpha1 is very much based on his outstanding work. And of course many thanks and a big sorry to all the contributors who submitted new features and bug fixes and were left without a response, wondering if their work is ever going to be merged back into the official s3cmd codebase. Sorry guys, this indeed wasn't a prime example of a project maintainership :( So what has changed since 1.1.0-beta3 released a year ago? _s3cmd 1.5.0__-alpha1_ * Server-side copy for hardlinks/softlinks to improve performance (Matt Domsch) * New [signurl] command (Craig Ringer) * Improved symlink-loop detection (Michal Ludvig) * Add --delete-after option for sync (Matt Domsch) * Handle empty return bodies when processing S3 errors. (Kelly McLaughlin) * Upload from STDIN (Eric Connell) * Updated bucket locations (Stefhen Hovland) * Support custom HTTP headers (Brendan O'Connor, Karl Matthias) * Improved MIME support (Karsten Sperling, Christopher Noyes) * Added support for --acl-grant/--acl-revoke to 'sync' command (Michael Tyson) * CloudFront: Support default index and default root invalidation (Josep del Rio) * Command line options for access/secret keys (Matt Sweeney) * Support [setpolicy] for setting bucket policies (Joe Fiorini) * Respect the $TZ environment variable (James Brown) * Reduce memory consumption for [s3cmd du] (Charlie Schluting) * Rate limit progress updates (Steven Noonan) * Download from S3 to a temp file first (Sumit Kumar) * Reuse a single connection when doing a bucket list (Kelly McLaughlin) * Delete empty files if object_get() failed (Oren Held) That's a lot of changes all around the codebase. Although s3cmd passes the testsuite and works in the common situations I'm afraid we may have missed some corner cases. Please report any problems you experience when running s3cmd 1.5.0-alpha1 to make the final release as stable as possible. Download from here: http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha1/s3cmd-1.5.0-alpha1.tar.gz The most up to date sourcecode is available on GitHub: https://github.com/s3tools/s3cmd Also check out our website for further updates: http://s3tools.org/s3cmd Now download, test and enjoy! :) Michal & contributors |
From: Michal L. <ml...@lo...> - 2012-01-12 08:00:42
|
Hi all! I have just released s3cmd 1.1.0-beta3 with two important fixes: - Multipart now works for [sync] with a limitation in that it doesn't compare MD5 sums of the local and remote file. The reason is that Amazon doesn't provide a MD5 sum for multipart-uploaded files and therefore we have nothing to check the local MD5 sum against. For now s3cmd only compares the file sizes in that case, will add a timestamp check later as well. - The new MIME guessing turned out to a bit difficult to initialise properly across all the platforms. Not only that the python-magic module may not be installed, worse yet there are two incompatible python-magic modules currently shipped with Linux recent distros. It's quite a mess but hopefully it should now work as expected for everyone. And obviously a number of minor bugs was fixed as well. Thanks to all the reporters and contributors! Download from here: http://sourceforge.net/projects/s3tools/files/s3cmd/1.1.0-beta3/s3cmd-1.1.0-beta3.tar.gz/download Alternatively grab the latest source from the official git repository: git clone git://github.com/s3tools/s3cmd.git _**_As with any new release I'm keen to hear your feedback, positive or not. Please direct all emails to s3t...@li... as usual. Enjoy s3cmd 1.1.0-beta3 :) Michal |
From: Michal L. <ml...@lo...> - 2012-01-06 11:47:48
|
Hi all! It's been a looong time since the last major s3cmd release. It's been quiet but the project is not dead. A lot has happened since s3cmd 1.0.0 - most importantly the development was moved to GIT and is now primarily hosted on GitHub (https://github.com/s3tools/s3cmd). This move has sparked an unexpected activity among the users(-now-developers) of s3cmd who begun to contribute new features in a much higher rate than I was used to while the code was hosted in SourceForge's SVN. s3cmd 1.1.0 will be a true community effort - thanks folks! Here are the major changes... *Multipart Upload* It's by far the most requested feature and it's finally here. In this beta release it works only for the [put] command, not for [sync] because there are some technical difficulties yet to be sorted out. Multipart is enabled by default and kicks in for files bigger than 15MB. You can set this treshold as low as 5MB (Amazon's limit) with --multipart-chunk-size-mb=5 or to any other value between 5 and 5120 MB. Two "management" commands are still missing - [mplist] for listing all unfinished multipart uploads and [mpabort] for aborting those left-behind ones. S3cmd tries to cleanly abort the upload and free up all the resources on the S3 side if it's interrupted for any reason but there still may be occasions where an unfinished upload is left behind. A number of contributors submitted patches adding this functionality over the past few months, thank you all! Unfortunately none of submissions was complete and clean enough to make it through. In the end I used Jerome Leclanche's (Adys) work as a starting point and based my work on that. Thanks Jerome! *CloudFront invalidation* Another often requested feature. s3cmd now can invalidate objects in CloudFront distributions immediately during [sync] with --cf-invalidate switch. With this switch all uploaded files will be invalidated from CloudFront which makes the content delivery network consistent with the primary S3 storage. It takes some time to process the invalidation - use [s3cmd cfinvalinfo cf://dist-id] command to query the status. See CloudFront documentation <http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html> for details. *Static WebSite support* Jens Braeuer contributed a patch for creating and deleting "static websites" in S3/CF. See S3 documentation <http://docs.amazonwebservices.com/AmazonS3/latest/dev/WebsiteHosting.html> for details. *Config settings from Environment* Many users are not comfortable with keeping their access and secret keys in the config file. Ori Bar provided a patch that allows any config value to be taken from the process environment. For example to obtain the access key from the environment put this in your ~/.s3cfg file: access_key=$S3_ACCESS_KEY The S3_ACCESS_KEY name is just an example - use anything you like, just prepend it with "$" sign. For instance access_key=$ABRACADABRA is just as valid as $S3_ACCESS_KEY *MIME options made saner* The previous semantic of --default-mime-type and --guess-mime-type was a bit ... illogical. It actually caught me by surprise a couple of times, despite the fact that I was the author of such an ill-logic ;) I believe it's a lot cleaner now: --guess-mime-type tries to guess the MIME type by the extension or, if available, by using the "magic" file (python-magic support contributed by Karsten Sperling). If guessing fails the type is set to binary/octet-stream or to whatever is set in --default-mime-type. Explicit setting including the mime-type parameters can be achieved with --mime-type="text/plain; charset=utf-8". *Download multiple files to stdout* Rob Wills contributed a support for downloading multiple files to standard output. I'm not quite sure what is it good for to be honest ;) Anyway the patch was straightforward and so I accepted it. *s3cmd --configure can test access to a specified bucket *Listing buckets requires the S3 ListAllMyBuckets permission which is typically not available to delegated IAM accounts. With this change, s3cmd --configure accepts an (optional) bucket uri as a parameter and if provided, the access check will just verify access to this bucket individually. Contributed by Mike Repass. *Minor changes* Obviously this release also comes with a number of smaller fixes and improvements, tweaked config values here and there, stability improvements, documentation fixes, etc. Too much to list here individually. _*Contribute to s3cmd project on GitHub*_ All hands are welcome to contribute to the s3cmd project. Whether you can code new features in Python or add/improve documentation you're welcome to "fork" the https://github.com/s3tools/s3cmd project, do your changes in your private s3cmd workspace and once ready submit them for inclusion to the official repository by sending a "Pull Request" from the GitHub web interface. I'm not going into more details about Git, GitHub workflow, forks, pull requests, etc - it's well out of scope of this announcement. Google it, Try it, Like it and Contribute :) _*Download s3cmd 1.1.0-beta2*_ Download the s3cmd 1.0.0-beta2 package from SourceForge: http://sourceforge.net/projects/s3tools/files/s3cmd/1.1.0-beta2/s3cmd-1.1.0-beta2.tar.gz/download Alternatively grab the latest source from the official git repository: git clone git://github.com/s3tools/s3cmd.git _*Donate to s3cmd project*_ The s3cmd script is a free software, no licence or fees are required to use it. However if you find s3cmd useful and worth a buck please make a donation. Any amount, small or big, will be greatly appreciated! Follow this link for a PayPal or Credit Card donation: http://s3tools.org/donate _*Etc... *_As with any new release I'm keen to hear your feedback, positive or not. Please direct all emails to s3t...@li... as usual. Enjoy s3cmd 1.1.0-beta2 :) Michal |
From: Michal L. <ml...@lo...> - 2011-04-12 00:30:06
|
Hi all s3cmd fans! The long requested *CloudFront invalidation* support is now available in s3cmd-1.1.0-beta1 thanks to a sponsorship from Philipp Winkler and Interactive 21 <http://interactive21.de>. CF Invalidation is integrated with /s3cmd sync/ command: ~$ s3cmd *--cf-invalidate* --acl-public sync /local/path/ s3://<bucket>/remote/path/ [... uploading ...] Created invalidation request for 4 paths Check progress with: s3cmd cfinvalinfo cf://E9RB4B71C06M7/IJK5N4YQBAG8K It takes a couple of minutes for the invalidation requests to be processed on the Amazon side. Can check the status with s3cmd cfinvalinfo: ~$ s3cmd *cfinvalinfo* cf://E9RB4B71C06M7/IJK5N4YQBAG8K URI: cf://E9RB4B71C06M7/IJK5N4YQBAG8K Status: InProgress Created: 2011-04-11T03:05:55.270Z Nr of paths: 4 Reference: E9RB4B71C06M7.20110411150554.2741 As there can be up to 3 invalidation requests running in parallel you can list them all (using either s3://bucket or cf://dist-id identifier): ~$ s3cmd cfinvalinfo s3://public.example.com URI: cf://E9RB4B71C06M7/IJK5N4YQBAG8K Status: *InProgress* Created: 2011-04-11T03:05:55.270Z Nr of paths: 4 Reference: E9RB4B71C06M7.20110411150554.2741 URI: cf://E9RB4B71C06M7/I1TA7E8XR7J70P Status: *Completed* Created: 2011-04-08T14:59:02.714Z Nr of paths: 20 Reference: E9RB4B71C06M7.20110409025901.2531 [...] There is also a couple of limitations: - invalidation is only available for sync, there is no dedicated /s3cmd cfinval/ command that would invalidate without uploading. Yet. - the maximum number of paths invalidated in one run is 999 (that's a single request size limit in Amazon). If more than 999 paths are to be invalidated the list is saved in a file and the user will have to deal with that in some other way, possibly using the future /s3cmd cfinval/ command. Download the package from this link: http://sourceforge.net/projects/s3tools/files/testing/s3cmd-1.1.0-beta1.tar.gz/download For testing it is /not/ necessary to install s3cmd into the system. Simply unpack it and run from that directory. For instance instead of invoking the installed s3cmd run /tmp/s3cmd-1.1.0-beta1/s3cmd (if that's where you unpacked the tarball). S3cmd 1.1.0-beta1 should work as expected and there should be no regressions against s3cmd 1.0.0 but please test it and let me know. Enjoy! Michal |
From: Michal L. <ml...@lo...> - 2011-01-20 02:24:10
|
(resending - corrected URL) Hi everyone! We have been maintaining installation repositories for RPM-based distributions, namely RHEL, Fedora, CentOS and OpenSUSE / SLES for quite some time now and the service became pretty popular - only yesterday we have seen over 1500 unique IP addresses checking for updates! However one of the frequent questions from the community - "*Is there a repository for Debian / Ubuntu?*" - used to have a negative answer. Although both Ubuntu and Debian have s3cmd in their standard repositories the versions there are never updated once released which is especially annoying for distributions with a long support like Ubuntu LTS. Now the things have changed - the S3tools project announces an *Official s3cmd DEB repository* from which you can always get the latest greatest s3cmd for your Debian or Ubuntu. Follow these steps to enable the repo from command line: 1) Import the repository key: wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key | sudo apt-key add - 2) Add the repo to sources.list: sudo wget -O/etc/apt/sources.list.d/s3tools.list http://s3tools.org/repo/deb-all/stable/s3tools.list 3) Refresh package cache: sudo aptitude update && sudo aptitude install -V s3cmd 4) Test if it works as expected and let me know if not :) Although I'm not a DEB packaging expert the repo seems to work fine, tested on Ubuntu 10.4 LTS x86 (Lucid Lynx), on Debian 5 amd64 (Lenny) and on Debian 6 (Squeeze) running on quite an exotic HP-PA machine. Should work just fine with Ubuntu 10.10 and newer and most likely with some older Ubuntus too. Success stories are indeed welcome :) Enjoy it! Michal |
From: Michal L. <ml...@lo...> - 2011-01-20 02:05:29
|
Hi everyone! We have been maintaining installation repositories for RPM-based distributions, namely RHEL, Fedora, CentOS and OpenSUSE / SLES for quite some time now and the service became pretty popular - only yesterday we have seen over 1500 unique IP addresses checking for updates! However one of the frequent questions from the community - "*Is there a repository for Debian / Ubuntu?*" - used to have a negative answer. Although both Ubuntu and Debian have s3cmd in their standard repositories the versions there are never updated once released which is especially annoying for distributions with a long support like Ubuntu LTS. Now the things have changed - the S3tools project announces an *Official s3cmd DEB repository* from which you can always get the latest greatest s3cmd for your Debian or Ubuntu. Follow these steps to enable the repo from command line: 1) Import the repository key: wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key <http://s3tools.org/repo/deb-py26/stable/s3tools.key> | sudo apt-key add - 2) Add the repo to sources.list: sudo wget -O/etc/apt/sources.list.d/s3tools.list http://s3tools.org/repo/deb-all/stable/s3tools.list 3) Refresh package cache: sudo aptitude update && sudo aptitude install -V s3cmd 4) Test if it works as expected and let me know if not :) Although I'm not a DEB packaging expert the repo seems to work fine, tested on Ubuntu 10.4 LTS x86 (Lucid Lynx), on Debian 5 amd64 (Lenny) and on Debian 6 (Squeeze) running on quite an exotic HP-PA machine. Should work just fine with Ubuntu 10.10 and newer and most likely with some older Ubuntus too. Success stories are indeed welcome :) Enjoy it! Michal |
From: Michal L. <ml...@lo...> - 2011-01-10 01:48:57
|
Hello World :) The time has come, _s3cmd 1.0.0 has been released_! Four years after the first public version s3cmd is now a stable, production ready, easy to use piece of software. It supports almost all features of Amazon S3 and Amazon CloudFront, is designed for use in shell scripts and automated processes, great both for sensitive encrypted backups to Amazon S3 as well as for publishing web content via Amazon CloudFront content delivery network. And all in between. The list of all currently supported features is very long, let me just include the changes from s3cmd 0.9.9.91: * [sync] now supports --no-check-md5 * [sync] now supports bucket-to-bucket synchronisation * Added [accesslog] command. * Added access logging for CloudFront distributions using [cfmodify --log] * Added --acl-grant and --acl-revoke [thx Timothee Groleau] * Allow s3:// URI as well as cf:// URI as a distribution name for most CloudFront related commands. * Support for Reduced Redundancy Storage (--reduced-redundancy) * Follow symlinks in [put] and [sync] with --follow-symlinks * Support for CloudFront DefaultRootObject [thx Luke Andrew] * Network connections now have 10s timeout _Download_ Obviously this is the best s3cmd release ever - all users are advised to upgrade. See http://s3tools.org/download for download links or go directly to the SourceForge download page: https://sourceforge.net/projects/s3tools/files/s3cmd/ _RPM repositories_ If you run a RedHat, CentOS, Fedora or one of the SuSE distributions the best way to keep your s3cmd up to date is to install it from our RPM repository. See http://s3tools.org/repositories for more informations and instructions. _Development support_ If you find s3cmd useful, and especially if you use it for your business, you may want to support further development by giving us a small donation. Visit http://s3tools.org/donate for more info. _Future development_ s3cmd 1.0.0 is an important milestone but we're not yet done. The most requested features that will be addressed in the near future include: * Multipart upload - more reliable than current file-at-once and allowing over 5GB files. * CloudFront invalidation - integrated with 'sync' as well as standalone 'cfinvalidate' command. * and a lot more :) _Etc..._ As with any new release I'm keen to hear your feedback, positive or not. Direct any emails to s3t...@li... as usually. Enjoy s3cmd 1.0.0 and have a great day! Michal |
From: Michal L. <mi...@lo...> - 2009-10-19 10:49:45
|
Hi all, It has been a long time since the last s3cmd release. I hope that means that s3cmd is now a pretty stable, mature piece of software and that we’re on track to 1.0.0 release :-) Therefore take this 0.9.9.91 release as a -rc1 for 1.0.0. What's new? =========== == Added --recursive support for cp and mv == It is finally possible to 'rename' buckets by moving the whole contents from one bucket to another on the server side (ie without downloading to your server and uploading back to S3). == Added --include/--exclude and --dry-run for cp, mv, del and setacl == All file manipulation commands now understand the standard set of parameters for including and excluding files, test runs with --dry-run, etc. == New command 'sign' == This is useful for signing for instance POST upload policies in order to allow uploading to a bucket through an ordinary web page. == Added --verbatim mode == This mode prevents most filenames pre-processing, for instance converting them to Unicode, converting to XML entities, etc. It's an invaluable for fixing broken buckets, however successful use assumes some decent knowledge of Amazon S3 internals. Most users will probably prefer... == New command 'fixbucket' == New command for fixing invalid filenames in a bucket. Many people somehow seem to manage to upload files for instance with a backspace character that later lead to an invalid XML returned from Amazon. This command should be able to fix such buckets. Download -------- Users of RPM-based Linux distributions are advised to grab one of the RPMs provided in our repositories and eventually add our repos to their package manager. Details are here: http://s3tools.org/repositories All others should install from a source package. Get it from SourceForge: https://sourceforge.net/projects/s3tools/files/ Many thanks to all of you who reported problems encountered in s3cmd 0.9.9. Without you I wouldn't know what needs to get fixed. And indeed many thanks to those who contributed with a donation (http://s3tools.org/donations). Your help is much appreciated! Any questions or feedback? Please send an email to the mailing list: s3t...@li... Enjoy! Michal |
From: Michal L. <mi...@lo...> - 2009-02-19 00:22:10
|
Hi all, over 3 months after the last release of s3cmd it's time for an upgrade again ;-) As you may have already noticed on SourceForge or Freshmeat, s3cmd 0.9.9 has finally been released with a load of new features and improvements. To learn more about Amazon S3 and s3cmd visit http://s3tools.org/s3cmd Major new features in s3cmd 0.9.9: --------------------------------- == Improved [put], [get] and [sync] == All these three commands now can do recursive transfer, accept multiple sources and support wildcards even for the direction from S3 to local dir. All of them also support --exclude / --include and eventually --skip-existing filtering. For instance: s3cmd put --recursive dir1 dir2 file3.txt s3://bkt/backup/ s3cmd get --exclude "tmp*.jpg" s3://bkt/backup/*.jpg /tmp/ Be aware that these commands may potentially behave slightly differently to what you were used to in older s3cmd versions. To make sure that it does what you meant and that your exclude / include patterns are correct use --dry-run mode. With that option s3cmd will only print what files should be transferred without actually transferring them. Check out http://s3tools.org/s3cmd-sync for more details. == CloudFront support == Amazon recently released their CloudFront Content Delivery Network for public and thanks to a sponsorship from Joseph Denne s3cmd now can manage CloudFront for you - the commands are [cfcreate], [cfdelete], [cfmodify] and [cfinfo]. See more at http://s3tools.org/s3cmd-cloudfront == Progress meter == Uploading or downloading large files often meant many minutes wait without any way to check how far the transfer has gone. Now we've got a new progress meter that'll keep you updated on the progress: testfile.pdf -> s3://bkt/testfile.pdf [1 of 1] 495616 of 2304411 21% in 4s 112.52 kB/s When s3cmd is running on a terminal it by default uses the progress meter. If it's running from a cron job or otherwise without a terminal it won't. These defaults can be overriden with --progress and --no-progress switches. == Commands for copying and moving remote files == Andrew Ryan contributed support for Copy and Move operations, accessible as [cp] and [mv] commands. You can copy or move objects within a bucket or from one bucket to another, even from US to EU data center. At the moment these two commands don't support recursive operation or multiple sources, such a functionality will be added in the upcoming version. == New command [setacl] for setting ACL on existing objects == To enable public access to your files over HTTP they need to have a "Public ACL". You can do it either during upload with --acl-public parameter or later on using [setacl] command. Again, sponsored by Joseph Denne. Use it with --acl-public or --acl-private to set the desired Access Control. == New --add-header option == One more feature for web hosters - occasionally you may want to set a specific headers to your files that are to be accessed over HTTP. Such as "Content-encoding:gzip" for, say, your gzipped javascript, or perhaps Expires or Cache-control headers (not only) for use with CloudFront CDN. With --add-header, that is available with [put] and [sync] as well as with [cp] and [mv], you can do all that: s3cmd put blah.js.gz s3://public-bkt/js/ \ --mime-type "application/javascript" \ --add-header "Content-Encoding: gzip" \ --acl-public == Recursive [del] and removal of non-empty buckets == Many people wanted an easy way to delete a subtrees from S3 or to remove non-empty buckets. Both is now possible - [del] understands --recursive and [rb] honours --force (which in fact will do a recursive delete first internally): ~$ s3cmd rb --force s3://bkt-test WARNING: Bucket is not empty. Removing all the objects from it first. This may take some time... File s3://bkt-test/testfile.txt deleted Bucket 's3://bkt-test/' removed == Support for non-recursive [ls] == Listing large buckets with thousands of files used to be pretty difficult with s3cmd ls. With the non-recursive [ls], which is a default from now, listings will look like this: ~$ s3cmd ls s3://bkt/ DIR s3://bkt/subdir1/ DIR s3://bkt/subdir2/ 2009-02-11 03:32 2304411 s3://bkt/testfile.pdf The old "list everything" behaviour is still available with --recursive option. == Added --list-md5 option for [ls] == One more improvement for [ls] - print out MD5 sums of the remote files: ~$ s3cmd ls s3://bkt/ DIR s3://bkt/subdir1/ 2009-02-11 03:32 2304411 6e4469df...7a8b88 s3://bkt/tst.pdf == Improvements in handling non-ASCII filenames == Ever lasting troubles with non-ASCII characters in filenames should now be gone (or at least reduced). If you're on a UTF-8 system and all your files have UTF-8 encoded filenames s3cmd should work just fine. In all other cases ... let me know ;-) Your system encoding should be autodetected correctly, but in case of cron jobs it may eventually fail at times. In such a case --encoding option to set the encoding explicitly: ~$ s3cmd put --encoding UTF-8 číča.txt s3://bkt/ Minor features and bugfixes: ---------------------------- * Improved resistance to communication errors (Connection reset by peer, etc.) * Continue [get] of partially downloaded files with --continue * Fixed GPG (--encrypt) compatibility with Python 2.6. * Always send Content-Length header to satisfy some http proxies. * Fixed installation on Windows and Mac OS X. * Don't print a nasty backtrace on KeyboardInterrupt. Quite an bunch of new stuff, isn't it? Download -------- Users of RPM-based Linux distributions are advised to grab one of the RPMs provided in our repositories and eventually add our repos to their package manager. Details are here: http://s3tools.org/repositories All others should install from a source package. Get it from SourceForge: https://sourceforge.net/project/showfiles.php?group_id=178907&package_id=218690 Many thanks to all of you who were testing the -pre and -rc releases and reported back the problems encountered. And indeed many thanks to those who contributed with a donation (http://s3tools.org/donations). Your help is much appreciated! Any questions or feedback? Please send an email to the mailing list: s3t...@li... Enjoy! Michal |
From: Michal L. <mi...@lo...> - 2009-02-02 03:28:38
|
Hi all, s3cmd 0.9.9-rc3 is now available for download. It fixes three problems: 1) S3cmd crashed on upload instead of restarting the failed operation when Amazon returned Internal Error. The crash message typically was: AttributeError: 'S3Error' object has no attribute '_message' Fixed now. 2) Under some circumstances s3cmd crashed on put / get / sync when there were no files to transfer. Fixed now. 3) s3cmd sync with --dry-run always reported files scheduled for deletion, thus confusing users. Now these files are only printed out when --delete-removed is also used. Let me know if you encounter any other problems please. Download from here: https://sourceforge.net/project/showfiles.php?group_id=178907&package_id=218690 Enjoy Michal |