You can subscribe to this list here.
2006 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
---|---|---|---|---|---|---|---|---|---|---|---|---|
2007 |
Jan
|
Feb
(13) |
Mar
(3) |
Apr
|
May
(3) |
Jun
(6) |
Jul
(1) |
Aug
(15) |
Sep
(6) |
Oct
|
Nov
(7) |
Dec
|
2008 |
Jan
|
Feb
(5) |
Mar
(2) |
Apr
|
May
|
Jun
(12) |
Jul
(15) |
Aug
(7) |
Sep
(11) |
Oct
(12) |
Nov
(45) |
Dec
(6) |
2009 |
Jan
(11) |
Feb
(24) |
Mar
(24) |
Apr
(3) |
May
(11) |
Jun
(4) |
Jul
(12) |
Aug
(20) |
Sep
(2) |
Oct
(10) |
Nov
(6) |
Dec
(15) |
2010 |
Jan
(5) |
Feb
|
Mar
(3) |
Apr
(11) |
May
(11) |
Jun
(6) |
Jul
(5) |
Aug
(13) |
Sep
(5) |
Oct
(18) |
Nov
(37) |
Dec
(11) |
2011 |
Jan
(33) |
Feb
(4) |
Mar
(14) |
Apr
(8) |
May
(3) |
Jun
(6) |
Jul
(4) |
Aug
(1) |
Sep
|
Oct
(4) |
Nov
(2) |
Dec
(5) |
2012 |
Jan
(8) |
Feb
(2) |
Mar
(4) |
Apr
(3) |
May
(4) |
Jun
(5) |
Jul
(22) |
Aug
(10) |
Sep
(3) |
Oct
(16) |
Nov
(2) |
Dec
(16) |
2013 |
Jan
(4) |
Feb
(9) |
Mar
(6) |
Apr
|
May
(14) |
Jun
(4) |
Jul
(12) |
Aug
(21) |
Sep
|
Oct
(3) |
Nov
(7) |
Dec
(3) |
2014 |
Jan
(1) |
Feb
(10) |
Mar
(19) |
Apr
(43) |
May
(7) |
Jun
(6) |
Jul
(2) |
Aug
(1) |
Sep
(1) |
Oct
(1) |
Nov
(1) |
Dec
(10) |
2015 |
Jan
(5) |
Feb
(18) |
Mar
(17) |
Apr
(16) |
May
(8) |
Jun
|
Jul
(1) |
Aug
(4) |
Sep
(4) |
Oct
(5) |
Nov
(7) |
Dec
|
2016 |
Jan
(7) |
Feb
(2) |
Mar
(4) |
Apr
|
May
(1) |
Jun
(2) |
Jul
|
Aug
|
Sep
|
Oct
(3) |
Nov
|
Dec
|
2017 |
Jan
|
Feb
|
Mar
(3) |
Apr
(3) |
May
(1) |
Jun
(4) |
Jul
|
Aug
(1) |
Sep
|
Oct
(3) |
Nov
(1) |
Dec
|
2018 |
Jan
(1) |
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
(2) |
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
(3) |
Dec
|
2019 |
Jan
|
Feb
|
Mar
(7) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(1) |
2020 |
Jan
|
Feb
|
Mar
|
Apr
(3) |
May
(1) |
Jun
(1) |
Jul
(1) |
Aug
|
Sep
|
Oct
(3) |
Nov
(1) |
Dec
(7) |
2021 |
Jan
|
Feb
(2) |
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
2022 |
Jan
|
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
|
Dec
|
2024 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
From: <sil...@sr...> - 2024-09-04 21:33:28
|
Hello group, having problem with pre-signed URLs on B2. Setup: - application key with read access to the bucket and `listAllBucketNames` enabled - private bucket - file in that bucket - visible with `s3cmd ls s3://bucket-name/` - downloadable with `s3cmd get s3://bucket-name/file-name` - no special characters in bucket-name nor file-name. Just plain alphanumeric ASCII Now I try to get that file with a pre-signed URL: $ s3cmd signurl s3://bucket-name/file-name `gdate -d 'now + 1 year' +%s` https://bucket-name.s3.us-west-001.backblazeb2.com/file-name?AWSAccessKeyId=<correct-access-key-id-here>&Expires=1757021192&Signature=aJtG2JFU3xrWIMZlD%2F0wwWYhTIs%3D But trying to use this URL returns always <Error> <Code>UnauthorizedAccess</Code> <Message>bucket is not authorized: bucket-name</Message> </Error> Any clues as to what I might be doing wrong? -- SD! |
From: Florent V. (Sodria) <fl...@so...> - 2022-10-03 16:27:40
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.3.0 has been released on github, sourceforge and PyPi. This release should not contain any breaking change. Thanks again to all the contributors, the testers and the issue reporters. s3cmd-2.3.0 - 2022-10-03 =============== * Added "getnotification", "setnotification", and "delnotification" commands for notification policies (hrchu) * Added support for AWS_STS_REGIONAL_ENDPOINTS (#1218, #1228) (Johan Lanzrein) * Added ConnectionRefused [111] exit code to handle connection errors (Salar Nosrati-Ershad) * Added support for IMDSv2. Should work automatically on ec2 (Anthony Foiani) * Added --list-allow-unordered to list objects unordered. Only supported by Ceph based s3-compatible services (#1269) (Salar Nosrati-Ershad) * Fixed --exclude dir behavior for python >= 3.6 (Daniil Tararukhin) * Fixed Cloudfront invalidate retry issue (Yuan-Hsiang Lee) * Fixed 0 byte cache files crashing s3cmd (#1234) (Carlos Laviola) * Fixed --continue behavior for the "get" command (#1009) (Anton Ustyugov) * Fixed unicode issue with fixbucket (#1259) * Fixed CannotSendRequest and ConnectionRefusedError errors at startup (#1261) * Fixed error reporting for object info when the object does not exist * Fixed "setup.py test" to do nothing to avoid failure that could be problematic for distribution packaging (#996) * Improved expire command to use Rule/Filter/Prefix for LifecycleConfiguration (#1247) * Improved PASS/CHECK/INCLUDE/EXCLUDE debug log messages * Improved setup.py with python 3.9 and 3.10 support info(Ori Avtalion) * Many other bug fixes Shortlog below. Thanks, Florent Anthony Foiani (2): use IMDSv2 to retrieve instance metadata make IMDSv2 optional Carlos Laviola (1): treat 0 byte cache files as nonexistent Daniil Tararukhin (1): fix os walk when excluding directories Florent Viard (13): Update CI test.yml to run on PR Fixes #1261 - Always read a conn after a request to be able to reuse it Fixes #1259 - Fixes unicode issue with fixbucket Fixes #1259 - Correct typo when retrieving response from IMDSv2 Ensure 404 exc content is filled even for object_info Fixes #1009 - Fixes --continue behavior for the "get" command Fixes #996 - Avoid crashing when setup.py test is called Fixes #1247 - Update expire command to use Rule/Filter/Prefix for LifecycleConfiguration Fixes #1261 - Additional fix for CannotSendRequest at startup Fixes ConnectionRefusedError not found error for python 2 Use str instead of repr in FileList debug logs Use quotes around files in FileLists for debug logs Update version to 2.3.0 Johan Lanzrein (1): Take into account the AWS_STS_REGIONAL_ENDPOINTS Ori Avtalion (1): List support of Python 3.9, 3.10 Salar Nosrati-Ershad (3): excepting/handling tcp connection refused error feat: implement list objects allow-onordered option as a config feat: add `--list-allow-unorderd` command-line switch Yuan-Hsiang Lee (1): fixes cloudfront invalidate retry hrchu (1): Add notification commands |
From: Felipe G. <fel...@gm...> - 2022-04-15 12:03:55
|
Hello, I am trying to use the `s3cmd` tool to create a bucket in a different region than the default US. And I am using a config file for that. The docs [1] say that there are other regions available. ``` --region=REGION, --bucket-location=REGION Region to create bucket in. As of now the regions are: us-east-1, us-west-1, us-west-2, eu-west-1, eu- central-1, ap-northeast-1, ap-southeast-1, ap- southeast-2, sa-east-1 ``` But when I try to use a region different than `US` I receive an error message. Here is the line of my config file to use another region. I based this file also on the official doc [2]. ``` [default] access_key = ****** access_token = add_encoding_exts = add_headers = bucket_location = eu-central-1 ``` The error seems to be the same of [3] but I dont understand how they solve to create the bucket in a different region. I also used the ` --bucket-location=":default-placement" ` and the bucket is still created at the US region. ``` ERROR: S3 error: 400 (InvalidLocationConstraint) ``` Could someone help me? [1] https://s3tools.org/usage [2] https://s3tools.org/kb/item14.htm [3] https://github.com/rook/rook/issues/2945 Thanks, Felipe |
From: Florent V. (Sodria) <fl...@so...> - 2021-09-27 19:19:39
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.2.0 has been released on github, sourceforge and PyPi. This release is mostly made of bug fixes and should not contain any breaking change. Thanks again to all the contributors, the testers and the issue reporters. s3cmd-2.2.0 - 2021-09-27 =============== * Added support for metadata modification of files bigger than 5 GiB * Added support for remote copy of files bigger than 5 GiB using MultiPart copy (Damian Martinez, Florent Viard) * Added progress info output for multipart copy and current-total info in output for cp, mv and modify * Added support for all special/foreign character names in object names to cp/mv/modify * Added support for SSL authentication (Aleksandr Chazov) * Added the http error 429 to the list of retryable errors (#1096) * Added support for listing and resuming of multipart uploads of more than 1000 parts (#346) * Added time based expiration for idle pool connections in order to avoid random broken pipe errors (#1114) * Added support for STS webidentity authentication (ie AssumeRole and AssumeRoleWithWebIdentity) (Samskeyti, Florent Viard) * Added support for custom headers to the mb command (#1197) (Sébastien Vajda) * Improved MultiPart copy to preserve acl and metadata of objects * Improved the server errors catching and reporting for cp/mv/modify commands * Improved resiliency against servers sending garbage responses (#1088, #1090, #1093) * Improved remote copy to have consistent copy of metadata in all cases: multipart or not, aws or not * Improved security by revoking public-write acl when private acl is set (#1151) (ruanzitao) * Improved speed when running on an EC2 instance (#1117) (Patrick Allain) * Reduced connection_max_age to 5s to avoid broken pipes as AWS closes https conns after around 6s (#1114) * Ensure that KeyboardInterrupt are always properly raised (#1089) * Changed sized of multipart copy chunks to 1 GiB * Fixed ValueError when using more than one ":" inside add_header in config file (#1087) * Fixed extra label issue when stdin used as source of a MultiPart upload * Fixed remote copy to allow changing the mime-type (ie content-type) of the new object * Fixed remote_copy to ensure that meta-s3cmd-attrs will be set based on the real source and not on the copy source * Fixed deprecation warnings due to invalid escape sequences (Karthikeyan Singaravelan) * Fixed getbucketinfo that was broken when the bucket lifecycle uses the filter element (Liu Lan) * Fixed RestoreRequest XML namespace URL (#1203) (Akete) * Fixed PARTIAL exit code that was not properly set when needed for object_get (#1190) * Fixed a possible inifinite loop when a file is truncated during hashsum or upload (#1125) (Matthew Krokosz, Florent Viard) * Fixed report_exception wrong error when LANG env var was not set (#1113) * Fixed wrong wiki url in error messages (Alec Barrett) * Py3: Fixed an AttributeError when using the "files-from" option * Py3: Fixed compatibility issues due to the removal of getchildren() from ElementTree in python 3.9 (#1146, #1157, #1162, # 1182, #1210) (Ondřej Budai) * Py3: Fixed compatibility issues due to the removal of encodestring() in python 3.9 (#1161, #1174) (Kentaro Kaneki) * Fixed a crash when the AWS_ACCESS_KEY env var is set but not AWS_SECRET_KEY (#1201) * Cleanup of check_md5 (Riccardo Magliocchetti) * Removed legacy code for dreamhost that should be necessary anymore * Migrated CI tests to use github actions (Arnaud J) * Improved README with a link to INSTALL.md (Sia Karamalegos) * Improved help content (Dmitrii Korostelev, Roland Van Laar) * Improvements for setup and build configurations * Many other bug fixes Shortlog below. Thanks, Florent AJP (1): Fix printing error in `report_exception(e)` Aaron Maxwell (17): Branch for Aaron / --follow-symlinks Provide --bucket-prefix for run-tests.py Test case for following symlinks (simplest case) (currently fails) Implementation of --follow-symlinks option (nonrecursive case) Test for s3cmd put --follow-symlinks Negative test for s3cmd put --follow-symlinks Manual page documentation for --follow-symlinks More thorough and precise tests for --follow-symlinks Guard against common error when using --bucket-prefix correct error in test use Autotest-3, not autotest-3 Compartmentalized symlink-safe directory walk, with all tests passing on python2.5 and python2.6 Fix leak in recursive-symlink-filtering algorithm document fswalk function name refactor to indicate _fswalk_follow_symlinks is an internal helper function Update Changelog Option to follow local symlinks for sync and put (--follow-symlinks option) including tests and documentation. Also addition to run-tests.py: --bucket-prefix option, to allow different developers to run tests in their own sandbox Abhishek Varshney (1): fix: handle empty md5 in find_md5_one Albert Bendicho (4): Off-by-one on MAX_CHUNKS verification max_chunks now a config option MAX_CHUNKS constant removed, now a config option Added multipart_max_chunks Alec Barrett (1): Update wiki URL in error message Aleksandr Chazov (4): Allow setting SSL authentication Update man pages make names more understandable allow skip key file config option for embedded keys case Alex Harford (1): Ensure testsuite-out is created before running dd Alex Offshore (1): Return decoded response in get_policy and get_cors Alexander Allakhverdiyev (1): Fix Python 3 bytes string encoding when getting IAM credentials Alexander Todorov (2): Disable hostname checks for SSL connections on RHEL 7.2 In case check_hostname isn't defined catch the error Andrew Gaul (7): Sort s3cmd ls output by bucket name Capitalize command descriptions for consistency Horizontally align object policy info Emit user metadata in object info Support signed URL content disposition type Emit storage class in info subcommand Handle missing ETag in subcmd_bucket_list Anton I. Sipos (1): Fix crash when cloudfront operations receive 500 errors. Retries parameter was not being sent to send_request as a keyword arg. Arnaud J (1): Migrate CI tests to use github actions Arto Jantunen (2): Add a close method to the ConnMan api Add option to disable connection pooling Arturas Moskvinas (1): Do not close stdout, it might be used when file is downloaded and destination is "-" Billy Crook (2): Added support for listing StorageClass attribute. Disabled by default. Enable with list_storageclass = True in .s3cfg Adding -l option for long_listing list_storageclass now provided by long_listing and -l. Minor whitespace cleanup. Bobby Impollonia (1): Add missing import of errno in S3.py Brady (2): Install instructions for PIP Correct typo in instructions Brandon Galbraith (1): Update s3cmd.1 Brendan O'Connor (1): Bugfix: use extra headers -- allows Requester Pays for buckets you don't own. Patch from http://arxiv.org/help/bulk_data_s3 Brian C. Lane (1): Fix handling of config tri-state bool values (like acl_public) Børge Nerdal Kjelsrud (1): Add option to ignore error on copy when key no longer exists Carlo Cabanilla (1): fix bug in datetime tuple concatenation Castedo Ellerman (1): content_encoding != charset, bug #153 Charlie Schluting (2): s3cmd du can gobble gigs of RAM on a bucket with millions of objects. Re-worked 'du' to traverse the structure and store only the sum at each level, allowing python to free memory at each folder. Went from 4GB consumption (and being killed) on a test bucket with ~50M objects in thousands of directories, to a max of 80M usage. that's not necessary any more. Chris Lamb (1): Support relative expiry times in signurl. Christian Rodriguez (2): Fix #849 by adding requester pays header to HEAD requests Merge branch 'bug/849' Christopher Noyes (3): allow adding encoding to content type in s3 don't add charset if it's already there, make param a comma delimited str, not multiple remove comments, add space after ; Christopher Porter (1): Fixing bug 3091912 - KeyError when copying multiple keys Cisco Guillaume (1): Merge pull request #1 from GuillaumeCisco/gcisco-fix-appdata-directory Craig Ringer (1): Add support for creating signed S3 URLs with the new signurl command Dale Lovelace (1): Allow --acl-grant on AWS groups e.g. Damian Gerow (2): Support proxying SSL connections Only support proxied SSL with Python >= 2.7 Damian Martinez (1): Adding MultiPartCopy class and enable for object_copy when remote file is greater than 5gb. Dan Carroll (1): Fixed typo: paramters -> parameters Daniel Gryniewicz (1): Add getlifecycle command Daniel Harris (1): Fix potential divide by zero errors Dave Cole (1): Apply include/exclude to destinations too, via mdomsch/s3cmd@268e7c3bd37cf60920025d556449fc1b93a1c590 Refs s3tools/s3cmd#109. David Eckelkamp (1): Let Makefile determine version number from PkgInfo.py file David Kohen (2): Add support for IAM roles and temp tokens IAM role typo fixes David Liu (1): Fail silently on SIGPIPE Devon Jones (3): added ability to set home based on environment variable added S3CMD_CONFIG documentation to the man page Merge remote-tracking branch 'upstream/master' Dmitry Kamenka (1): Copy uri_params in bucket_list Dongmao Zhang (2): add new option --limit-rate to throttle download/upload speed only gettimeofday when limitrate is required Doug Crozier (4): Changed "Buckets list" test to use pbucket rather than hardcoded names. This fixes issue #1066. Allow use of Docker to run test cases with multiple Python versions Fix for 'dictionary changed size during iteration' error on Python 3 with --cache-file (Issue #945) Test cases added for --cache-file EC2 Default User (1): .gitignore: add s3cmd.egg-info Emmanuel (1): managed files that was not remote copy to upload them directly EnlytonJ (1): Windows fix re / vs os.path.sep Eric Connell (1): added the ability to upload files by reading from stdin Eric Drechsel (1): Fix for ignored --default-mime-type option Eric Mill (1): rewrite of README into markdown Etienne Michon (1): Typo in manual Eugene Brevdo (4): Added --continue-put and --upload-id, allowing skipping of files or part files during upload when their size and md5sum match the file being uploaded. Minor bugfix to --upload-id check Fixed a small bug in MultiPart upload Fixed a upload corruption bug in MultiPart continue. Florent Viard (403): BugFix: Fix dateS3toPython to gracefully handle returned date that are not in GMT to respect iso8601. Ex.: 2014-03-05T18:45:50+00:00 instead of 2014-03-05T18:45:50Z Fixes date functions to return timestamps based on UTC time value. + so simplified formatDateTime(). Minor changes in README to fix points 1 and 2 in my comment at https://github.com/s3tools/s3cmd/pull/369. Cosmetic: removed one empty space in excess of end of run summary. Fix MD5 comparison with s3 compatible servers that are not Amazon S3. Small cleanup of code, using "in" instead of "find" for strings when possible. Fix an error in my previous commit: b5b61abc8eb159f5932b936c7ca47e618615f6a3. (.find() forgotten to be removed) Improve the exit code of s3cmd in the case of an error from S3 to give a clearer status of the reason of the failure. Some existing exit codes were modified when they were not already in use. Disable dns support for buckets when "host_bucket" hostname doesn't have a %(bucket)s item in it. Backported changes from the following pull request: https://github.com/s3tools/s3cmd/pull/369 regarding aws pricing. Updated README.md with latest changes of S3 prices: https://aws.amazon.com/fr/blogs/aws/aws-data-transfer-price-reduction/?sc_ichannel=EM&sc_icountry=global&sc_icampaigntype=launch&sc_icampaign=EM_128506540&sc_idetail=em_1778261631&ref_=pe_411040_128506540_12 Improvement of error reporting in the case of an issue when importing S3 sub-files. Now correctly cover cases like syntax errors in these files. Fixes #423 (https://github.com/s3tools/s3cmd/issues/423). Autodetected locale default should be set in default config for only being used when this field is not overriden, neither by command line or by a config file. Fixes #448 : Fixes S3UriS3 regexp to also recognize "s3://" as a valid S3UriS3 entity with no bucket and no object. Renaming some functions for the code to be more understandable. Fixes #424 : In python 2.7, parsing invalid xml now raise a ParserError(that didn't exist in python <2.7) instead of a ExpatError + cleanup in Utils.py (see issue #424 for more details) Improved handling for errors 501 and 504: Error 501: It means that a feature is not implemented in the server, so no need for useless retries. Error 504: Gateway timeout situation, exit code should indicate a server error. Cleanup: Add prologue to files to state that they are UTF-8 encoded. Fix run-test script to take into account the config file that is specified on the command line. Fixes encoding/decoding issues and so support for all strange characters (chinese, japanese, ...) in file names, folder or path arguments (Should fix bug like #417i) when UTF-8 is not the system def$ Now, in theory, everything that gets in should be converted to unicoded and then things would be converted back to string (user defined encoding) at the outmost parts when dealing with the system. When talking to S3, convert unicode to strings in utf-8 encoding. Remove useless error import Pass2 of "fixes encoding/decoding issues": os.walk replaced to handle unicodise conversion + fixed the removal of most of useless or invalid unicodise/decodise in the code. Fix for issue #432. Read the response from httplib even if there is a 0 byte to read. So httplib doesn't think that there is a response waiting to be read when the next request come. Straightforward to fix thanks to the detailed analysis of Edwintorok. Use deunicodise consistentely in object_copy and object_modify fix encoding : Added encode_to_s3 and decode_from_s3 function + switched to "unicode" call instead of "decode" as it is supposed to be more efficient. Fix wrong exception used for EOF from S3 error during recv_file. (S3Error expect an http response, not a text message) For copy and modify, take care to check if there is an error after the start of the operation (reported by a status 200 but an "Error" root entry in the xml body) In addition, for the move, It is good to check for the CopyObject xml reply to see if it is a success. But, if there is a status of 200 and no "data" in response, don't fail. Other S3 server implementations use only status code and don't return any body when there is no error. Another fix related to fix encoding issues. (file.name / stream.name are bytes, so unicodise them also) Fix invalid file transfer progress info in case of local2remote and remote2remote: If there was failed copy, seq number was preserved but previously uploaded file number was forgotten, thus user could have faced an invalid indication like: 14/8 when there is 6 upload following 6 remote copy failures. Add a link to the known issues page (on github) within the header and footer of s3cmd generated error reports. The goal is to reduce the number of duplicated "bug reports" for known issues. Fixes #528 (https://github.com/s3tools/s3cmd/issues/528). There is a lot of different "python-magic" libraries/versions, all doing different things, so all cases need to be covered. file-5.11 has be switched with "filemagic" as it should be more commonly used. Introspection in each python magic lib has been done to ensure to handle encoding or not only when it is needed. Fixes issue #576 (https://github.com/s3tools/s3cmd/issues/576). EX_SOMEFAILED was still used in a PR that was merged instead of EX_PARTIAL. Merge pull request #572 from fviard/fix_encoding_magic Merge pull request #574 from koenrh/https-by-default Don't slurp whole file while computing SHA256 checksum. Rework of the PR of petr-tichy. When using large --multipart-chunk-size-mb or --disable-multipart, the whole file was read into memory. Merge pull request #580 from Tim-Anderson/error_stop Merge pull request #583 from Tim-Anderson/issue577 Merge pull request #586 from Tim-Anderson/remove_ignore_failed_copy Merge pull request #587 from taz77/master Merge pull request #589 from jcwilson/master Merge pull request #560 from Flipboard/iter Merge pull request #593 from jrschumacher/set-cors Merge pull request #603 from mdomsch/bug/testsuite Following #603 : WARNING OUTPUT FORMAT CHANGE. Changed the format of output strings to be more consistant. Warning, the format of output for a lot of actions was changed. Thus, any automatic string for parsing the output could have to be fixed when using s3cmd after this checkin. Ex: "File hello/world.txt stored as s3://mybucket/world.txt" will now be: "upload: 'hello/world.txt' -> 's3://mybucket/world.txt' Fixes #602 (and complete #603) Fix run-tests.py to have all tests passing against HEAD If s3error message is none, don't print an useless ": None" Return NotFound error with message s3Error: NoSuchBucket when there is no bucket found to delete Raise standard S3Error in S3.py instead of simply exiting. So the normal error logic can't operate on any unexpected error. Avoid to have empty errors in output like: "ERROR: S3 error:" (Because of an 400 InvalidBucketName error returned by a non AWS3 server) Merge pull request #590 from andrewgaul/content-disposition Fix stop_on_error config type Attempt to fix the first bug of issue #600 : When a subfolder "xxx" was selected as source, the sync comparison was done against the destination base whereas it had to be done based on the "xxx" subfolder of the destination. Fixes run-tests following the fix for https://github.com/s3tools/s3cmd/issues/600 Merge pull request #616 from mdomsch/bug/612 Fix bad help info for the "--no-check-certificate" switch. Thanks to @psi29a for proposing the fix. Merge pull request #618 from mdomsch/bug/futurewarning Fixes #592 : Ignore any leading slash before bucket_name in an S3Uri. Original issue: "Running s3cmd ls s3:///foo lists owned buckets instead of emitting an error." Fix for error reporting in case of upload error. (Following the fix for #577) Fixes #581 regarding the reporting of SSL verification issues. Other problems of this issues should already be fixed. Merge pull request #621 from mdomsch/bug/ssl-checking Merge pull request #619 from fviard/second_fix_for_577 Merge pull request #617 from fviard/fix_subfolder_source Merge pull request #623 from mdomsch/bug/testsuite2 Reworked Upload and Download errors to be more consistent. Add detection and handling of different OSErrors like "File name too long." Merge pull request #628 from mdomsch/bug/curl Merge pull request #627 from fviard/upload_download_errors_rework Merge pull request #630 from mdomsch/feature/ia-class Merge pull request #611 from bsapiro/master Added --server-side-encryption-kms-id config option. Following PR #611. Fixes bug in PR #611 Fixes #636. Also remove the temporary file when encountering an S3Error. Merge pull request #599 from shigezobriand/master Fix "info" command for servers not supporting the "request_payer" feature. Based on PR #640 of Qiankunzheng. Fixes #644 : Minor fix to not output "u quoted" invalid parameter in the case of an invalid command. Reworked source and destination input of "remote2local" to cover gracefully more cases. Block "-" (stdin / stdout) to be used in parameters for sync as it has no sense. Only warn for the non regular files that are skipped. In the case of local2remote, raise an error if a source arg is missing. Fix automatic tests for recent changes. Merge pull request #650 from fviard/fix-remote2local-input Don't fail the copy if a non AWS s3 server doesn't support setting ACLs. Have extra labels coherent when progress is enabled or not and for multipart uploads. Added the --stats option that is similar to the one of rsync. Added "--stats" info for sync commands (remote2local, remote2remote, local2remote) Merge pull request #652 from s3tools/bug/598 Merge pull request #655 from fviard/rework_progress Fix duplicated code for the error alert message. Based on @edunham PR: https://github.com/s3tools/s3cmd/pull/609 Fixes #661 : Don't try to perform a delete if there is nothing to delete. + Restore warning that was in 1.5.2. Fixes #659 : Don't fail if "content-type" field is missing in info like for eucalyptus walrus. Fixes #646: number of "files to update" are not reported in the summary (there was only the new files to be added) Fixes #646 : remote2remote sync - "Updated" files were not reported as to be remotely copied when in dry-run mode. Merge pull request #667 from kobake/skip-restore-already-in-progress When using --no-preserve, still add 'md5' custom attribute if md5 check is enabled. Fixes #674 : When using AWS_CREDENTIAL_FILE, crash in case of invalid key in .aws/credentials file + updated the code for actual name of the keys. Merge pull request #711 from Shamanoid/patch-2 Merge pull request #559 from jmif/master Merge pull request #701 from makery/master Don't ask to set access_token in run_configure as it should not be used very often. Merge pull request #720 from harshavardhana/etag Merge pull request #684 from harshavardhana/try-fail Fix for local2remote in the case of "." as a source. Fixes #706 Fixes #709 : Fixes sync command with windows + cleanup Fix the error message to avoid confusion in a special cases like "no space left on device". Merge pull request #702 from artms/quiet Fixes #744 - Ensure to check non None or empty bucket names before calling check_bucket_name. Merge pull request #755 from andrewgaul/list-missing-etag Merge pull request #714 from alces-software/fix/generalize-wildcard-certificate-forgiveness Minor optimization and cosmetic fix in ConnMan. Fix Issue #672 - Patch based on PR #729 proposed by MLopez-Ibanez. Merge pull request #763 from dang/getlifecycle Fixes getBucketFromHostname for when host_bucket config has no "bucket" element. Fix the infamous "UnboundLocalError" for conn exception hidding meaningful exceptions Added an explicit error message when the servers hostname are wrong or unresolvable. Avoid having an error when user uses "https://" prefix to host_base... Fixes #741 Issues validating ssl certs when using a custom port for host_bucket address. Real fix for #772. Use urlparse to extract hostnames in connman Merge pull request #779 from cybojanek/master Follow up on #779, not using a CONNECT "tunnel" for http proxies as it is not the most common case. Fixes #777 : Fix __repr__ returning unicode and sign_v2 not encoded to s3. Merge pull request #757 from mozawa/max-ls-keys Merge pull request #781 from kwarunek/master Fixes recursive delete that was broken by the new "truncated" tag. Fixes orphan var in "du" exception handler. run-tests.py Update expirations tests dates to the future. Merge pull request #784 from mozawa/configure-endpoint Merge pull request #785 from shaform/py3k Fix failure of copy or rename for s3-like servers that don't have support for acl. Fix error of "info" command when s3-like servers don't implement "acl" or another feature. Merge pull request #789 from shaform/py3k Merge pull request #787 from Abhishekvrshny/fix-empty-md5 Avoid errors with modify command when s3-like servers don't support acl. Fix possible bug in previous change for info command and get_lifecycle. Fixes #722 : basename fonction was broken for local files for windows as the os path separator is not the same on linux. Aws sometimes uses 301 code for redirects + redirect to aws base host. Fix signature v4 when aws redirect to aws base host. Fixes #778 : Workaround for aws sending early errors before requests are completed + support for http expect continue feature. Updated maintainer in setup.py. Fixes #797 : unicodisation missing for exclude patterns coming from files. Fixes #800 (second part): "cfg.include" was not checked like is cfg.exclude to use the non memory optimal del subcmd. Fix for part of #660: fix detection for "sync single-file.txt s3://mybucket/single-file.txt" Fix to supporting python 1.6. Use "List" for http methods and not a kind of dict. Backport more code from httplib for compatibility with python 2.6 Fix previous patch on custom httpconnection, it is _send_output and not send_output. Attemps to fix last issues of #769 . Initial version of a .travis.yml and it's .travis.s3cfg configuration for CI with travis-ci. custom_httplib27 - Missing symbols in python 2.6 Missing bit in previous commit. [travis] Fixes allow_failures not being strings. [travis] Added run-test-minio.py + have travis run it. Added travis-ci build status of master in the github displayed readme. Merge pull request #806 from shaform/py3k Merge pull request #808 from robd003/master Merge pull request #809 from shaform/py3k Fixes #810 : Also handle redirects for path styles syntax. Fixes #816 : Fix recv_file 3xx,4xx, handlers + add region extraction of 3xx redirections. Fixes #798 : HEAD request can't parse xml response to get redirection url. Fix missing imports in custom_httplib27. Move urlencode func into Utils and fix encoding issues when having uid groupid with special chars. Small cleanup in S3.py Py3 support: remove usage of deprecated has_key. Py3 support: fix SortedDict (keys() are now views and iterators need __next__) Update README to indicate python versions supported by s3cmd. Changed unicodise/deunicodise wrappers to fit with python3 string model. Python3 support: Fixed stderrr, stdout unicode output. Remove some code that was there for supporting py version < 2.6 Use the stdout/stderr buffer directly instead of detaching. Python3 support: fix error reporting at import time and unicode. Python3 support WIP: Added Custom_httplib3x to patch python3 version of httplib as it is done for py2. [Py3 WIP] Fix for compatibility. PY3 support: fixed S3Uri and Exceptions regarding unicode/str compatibility. Missing change that was needed for custom_httplib functions... PY3 support: more unicode related things. PY3 support: long was removed in py3. Replace by int PY3 support: ensure to use the adequate / sign for division with python3. PY3 support: Added a warning at runtime to warn of WIP PY3 support: Added silent versions for unicodise/deunicodise funcs Small cosmetic changes in Custom_httplib(s) Remove legacy code. s3cmd is now only compatible py >2.6 Py3 support: Reworked request/signature regarding unicode/bytes mix. Py3 support: signature v2 - limit list of subresources url that are signed Py3 support: ensure that request body are encoded when "unicode" strings Signature v2: add "cors" to list of subresources to include in signature Improve get_bucket_location to be able to work all the time with aws Use pretty print everywhere in debug for request responses Getlocation for each type of request when region is not set and signature v4 used PY3 support: more fixes for str/unicode related issues. Typo fix in debug log message Merge pull request #837 from alexjh/allow-skip-tests Merge pull request #851 from burstworks/master PY3 support: use getitem instead of getslices for slices PY3 support: Fix for regexp/unicode in runtest-minio Use io buffers everywhere for files or stdin/stdout + Use custom stream_name attribute. Fixes signurl (v2) for python3. run-tests-minio: re-activated a test that is now working. Urlencode string only when formatting the url and not earlier. Get ride of the "extra" parameter for requests and use a clean "params" dict for all the query strings. Ensure to not have "{}" in functions parameters to be safe. In the v4 canonical query string, parameter without values should still have an equal sign. Fixes signature v2 invalid call to format_param_str. PY3 support: Fixes acl.py use of repr. Remove bad duplicated code. Little performance + memory improvement. + Fixes randomless in auto tests. Typo fixes. Sort headers in requests to always have the same order. Fixes failing tests with special chars for PY3 Small changes on run-tests.py Change .travis.yml now all tests are passing with minio. Merge pull request #870 from yossi-stratoscale/issue-869 Merge pull request #858 from chauffer/patch-1 Documentation: Fixes #872 . Updated following the proposal of @cthorley. Merge pull request #857 from guygurari/cp-mv-bug Fix error message to be accurate with all cases + Fixed auto tests for last change. Update version to 2.0.0 Update setup.py classifiers to indicate that Python3 is supported. Changed version to 2.0.0+ for github nightly versions Fixes #887 - Invalid code inside cf __str__ breaking cf invalidation. Fixes #889 - info output cosmetic fix. Fixes #885 - Specific error message when getting policy by non owner. Small cleanup of legacy checks in Config.py Fixes #888 - Fixes cfinvalinfo broken in 2.0.0 Fixes #886 - Don't force to have a / on last parameter for "modify" command. Fixes #890 - Missing json import after cleanup patch. Fixes #892 - Ensure to add "LICENSE" file to release tarballs. Fixes #894 - Detect and report error 403 in getpolicy for info command Fixes #895 - Refactored py3 source to be byte-compilable with py2 to fix warnings during "setup.py install" Remove python3 support warning. [PY3 support] Fixes --dump-config broken because of "verbosity" Cosmetic - use "with" instead of an open/close Fix encoding issues with config files + PY3 support for --configure Use "unicode" for config attributes Small rework for pre existing hash cache file detection PY3 fix: Open/Save hash cache file in "binary" mode Fix for non string cmdline parameters being ignored Cleanup: Don't have anymore cfg as a global Fixes missing local cfg in copy/move Fixes last missing local cfg Fixes #905 - Fixes https with proxy when no port defined for server. Fixes #908 - Fixes unicode in req headers for py2 and py3 Merge pull request #917 from thelan/master Fixes #921 - fixes gpg_command for python3 Fixed #922 - getenv return unicode in py3 and bytes in py2 Merge pull request #923 from harshavardhana/fix-etag Fix unicode issue in report in the case of an error Fixes #920 Always treat hostnames in lower case. Signature issues. Merge pull request #925 from jmaitrehenry/marker Fix #922 - Handle libmagic missing on windows Fixes getBucketFromHostname not using "lowercase" host_bucket. Avoid crashing python when encountering a CertificateError Update version to 2.0.1 Fixes sourceforge mailing list addr in RELEASE_INSTRUCTIONS file. Update version to 2.0.1+ for MASTER dev. Fixes #929 - strip namespace well even without xml declaration Fixes #933 Fix dst copy when the destination file was modified Merge pull request #938 from Nossnevs/feature/fix_nr_of_paths Fixes #525 - Fixed no invalidate of remotely copied files during sync Fixes error reporting for import error and utf8 characters as cmdline arguments Added clarification comment for the socket_timeout configuration value OS limit Partial fix for #941 - Retry for all kind of requests in case of OS network timeout events Fixes #941 - Restore retries for some 400 errors like RequestTimeout Partial fix for #913 - Only throttle retries for s3 slowdown errors Fixes #913 - Fix limitrate to recover of sudden changes in bandwidth and to not slow down too much the transfers Merge pull request #959 from mouchar/2.0.1 Little help clarification for --delete-after. Fixes #958 Merge pull request #947 from alexallah/master Addition to #947 - ensure it works with py2. aws requests always returns utf-8 encoded data Merge pull request #940 from bcl/master-acl_public Fixes bug created by #940 - cmdline set values are real booleans Complete #940 - upload_id, content_disposition and content_type default value is now "" and not anymore None Merge pull request #968 from LocutusOfBorg/avoid-distutils Merge pull request #970 from radeksimko/b-exclude-connection-header Merge pull request #975 from cybojanek/add_restore_subresource Merge pull request #966 from Trane9991/master Merge pull request #973 from mcv21/nulldisplay Fixes #960 - Do a retry if exc has errno but errno is None. Python3 support. Fixes auto-test following recent aws changes regarding bucket names. See: #957 Fixes #967 - In addition with patch for #960, ensure that error logging with not crash Update version to 2.0.2 Fixes #987 - Fixes using IAM with python3. Config options should be unicode. Fixes #987 - Before py36, py3 json was only able to load unicode str Merge pull request #986 from vpresnyak/output_alignment_fix Complete the fix #986 - To satisfy py2 and p3, there should be float before and after the division Issue #956 - Fixes size reporting using k instead of K as it a multiple of 1024 Fixes #956 - fix size and alignement of du and ls output reporting Merge changes of release 2.0.2 Merge pull request #994 from harshavardhana/no-need Merge pull request #1007 from hrchu/patch-1 Merge pull request #1010 from starrify/fix-minor-help-message-typo Convert anything given to getTreeFromXml to utf8 bytes if needed Fixes #1006 - fixes multiple python3 encoding issues with cloudfront functions Merge pull request #1021 from offshore/patch-1 Merge pull request #1033 from junousi/public_url_https Fixes #1006 - missing return in previous commit 222af5d8ce90bbb2f7784360c019f5a6af2ef8bd Merge pull request #1042 from serprex/master Merge pull request #1061 from k0ste/help Merge pull request #1063 from timgates42/bugfix/typo_unnecessarily Change version to development 2.0.2+ setup.cfg: set universal bdist_wheel flag setup.py: update meta files with support for recent python versions Merge pull request #1011 from zmedico/signurl_use_dns_based_on_bucket_host travis-ci: Disable tests for python 2.6 as not supported anymore by travis-ci default ubuntu distribution Merge pull request #1071 from nitrocode/patch-1 Complete the previous change to rename INSTALL file to INSTALL.md Merge pull request #1067 from NotTheDr01ds/bug/1066 Merge pull request #1068 from NotTheDr01ds/feature/run-tests-in-docker run-tests.py: Don't fail ACL tests if public_url_use_https is used svnignore: update to match latest gitignore content Little cleanup. TODO is out of date and obsolete Add a .dockerignore because of this stupid docker that does not know how to exclude files he has no business with Update run-tests.docker.README.md with a small display fix Create FUNDING.yml Factorize the region detection inner request and stop retrying requests that fail the ssl certificate checks Fixes #1059 - Hack for SignatureDoesNotMatch error when host port 80 or 443 is specified, due to stupid servers Merge pull request #1014 from Quiq/master Merge pull request #1032 from vainu-arto/make-connection-pooling-configurable run-test-*: Minor fixes Small cleanup Fixes #377 - Avoid crashing when a file disappeared during cache generation Fixes #1080 - url encode bucket_name for path-style requests. Fixes an issue with CephRGW and the tenant feature. Fixes #1081 - Don't force signature v4 if bucket_name uses special chars Fixes #1082 - Allow to use signature v4 only, even for a command with no target bucket specified Fixes #1054 - Relax limitation on special chars for --add-header key names Fixes #1020 - Update storage-class cmd help msg with new aws s3 classes Fixes open file descriptors leaking by not being closed on time Added a comment about how to escape the comment char in exclude files Fixes #989 - in aws_credential_file, fixes read_string only available with py3 + refactor for possible unicode issues Fixes #847 - set a few str as unicode to fix UnicodeDecodeError with python2 when doing a sync remote 2 remote Merge pull request #1084 from prabhakar267/master Fixes py headers to use system default python and not force python2 anymore setup.py: Update authors and cleanup Update version to 2.1.0 Change version to development 2.1.0+ Fixes #1087 - fixes ValueError when using more than one ":" inside add_header in config file Fixes #1089 - Ensure that KeyboardInterrupt are always properly raised Update RELEASE_INSTRUCTIONS with the new procedure to upload to Pypi Rework and cleanup of MultiPart and MultiPartCopy (following the merge of PR #74) Add progress info output for multipart copy and current-total info in output for cp, mv and modify Move s3_quote to Utils module Multipart copy - lower multipart copy chunk size to 1 GiB Move cmd - take care in that copy s3 xml result can be a multipart copy result Url encode source for copy/mv/modify to support all special/foreign character names in object names Cosmetic cleanup MultipartCopy avoid retrieving again object info when we already know its size Ensure to catch and display server errors for cmd cp,mv,modify errors Cleanup - no message was set in 2 cases for S3UploadError MultiPart - Fixes extra label issue with stdin as source MultiPartCopy - Also copy acl like direct copy MultipartCopy - Preserve metadata of the object that is copied Fixes #1091 - Ugly hack to have modify work with files bigger than 5Gb Allows to set mime-type (ie content-type) also for remote copies also Allows to set public ACL with "modify" to match "copy" Remove legacy code for dreamhost. I guess that they are now supporting acl Small cleanup of _build_attr_header Fixes typo/failure in multipart cp when progress meter is off Refactored object_copy to avoid duplicated code in object_modify Fixes #685 - Grammatical correction a debug message Metadata for copy should now be consistent for multipart or not and for aws or not cases Fix remote_copy to ensure that meta-s3cmd-attrs will be set based on the real source and not on the copy source Fixes _build_attr_header for case of wrong old value being use for wrong attribute Simplify the code to use _build_attr_header in remote_copy Fixes #1088 - Handle servers indicating (lying) that listObject result is truncated, when it is not the case Fixes #1090 - Be more resilient to a server sending random data in ?lifecycle response Fixes #1093 - Fixes error encountered with Minio send empty content instead of a valid ?requestPayment response. Merge pull request #1051 from chazer/ssl-auth Fixes #1094 - Fixes typo bug + mini cleanup Fixes #1096 - Add http error 429 to the retryable errors Fixes #1101 - "files-from" option was resulting in an AttributeError with Python3 Cleanup of a few lines ending with a ";" Cleanup - Remove unreachable code and an useless test Fixes #1113 - Fixes report_exception wrong error when LANG env var was not set Fixes #346 - Add support for listing and resume of multipart uploads of more than 1000 parts Utils/parseNodes: cleanup - avoid returning empty dicts inside the result list Ref #1114 - Add a time based expiration for idle pool connections in order to avoid random broken pipe errors Fixes #1114 - Reduce connection_max_age to 5s as it looks like that aws closes https conns after around 6s Merge pull request #1118 from Patouche/issues/1117 Merge pull request #1119 from Patouche/fix/datetime-tz Merge pull request #1120 from anbnyc/patch-1 Merge pull request #1124 from tirkarthi/fix-warnings Merge pull request #1137 from ondrejbudai/python39-fixes Merge pull request #1141 from onagurna/master Merge pull request #1145 from xrmx/cleanup_check_md5 travis.yml : Add python 3.9 to the ci test matrix Merge pull request #1144 from k3ntako/master Change import to have the py3 import first Merge pull request #1160 from siakaramalegos/patch-1 Merge pull request #1165 from FFKL/master Merge pull request #1163 from Rjerk/wip-fix-empty-lc-prefix Merge pull request #1152 from orztt/wip-private-acl Change dev version to 2.1.0-dev to respect PEP 440 that is now enforced by pip Fixes #1125 - avoid infinite loops when files are truncated during hashsum or upload Config: Get token expiration from AssumeRoleWithWebIdentityResult response Minor fixes in s3cmd Create a new BaseUtils module for helper functions that don't need access to Config Ensure that "https" url namespaces are also stripped in xml Complete work for AssumeRole And AssumeRoleWithWebIdentity support Use explicit raw strings for regexp patterns in Config Cleanup: Factor unicodise/deunicodise to be reused inside Config Fixes #1190 - Set properly PARTIAL exit code for object_get when needed Merge pull request #1199 from arnaudj/dev/migrateTravisToGitHubActions CI: Add python 3.10 to test matrix Merge pull request #1193 from rvanlaar/patch-1 Merge pull request #1204 from akete/issue-1203 Merge pull request #1198 from sebvajda/master Fixes #1201 - Raise an explicit error when only env var AWS_ACCESS_KEY is set Update version to 2.2.0 Francois Gaudin (2): Fix custom header support for cp Modify command equivalent to a self copy to modify metadata Francois Lebel (1): Fix the example for recursive listing. François Wirion (1): Fix missing slash in directory semantics documentation George Hickman (2): Don't include python files in the manifest. Add a makefile for releasing George Melika (1): Added a --disable-content-encoding to disable adding the content-encoding header to the pushed objects Gianfranco Costamagna (2): Fix typo (overriden -> overridden) Add fallback code to make it compatible with Python2 implementation (we could simplify by adding whichcraft, but a new dependency just for one which command seems an overkill) Gianni Chiappetta (1): Always attempt to set the content-type Giorgos Logiotatidis (1): If response['headers'] is empty, then don't update dict. GuillaumeCisco (3): Fix app data directory Add .idea directory to gitignore Add fallback for old windows versions Guy Gur-Ari (4): run-tests.py: use lowercase bucket name don't allow mv/cp of multiple files to single file raise exception for missing '/' during mv, cp remove unreachable code Harshavardhana (5): s3cmd: Fix s3 path style signature v4 handling. Missing etag in HTTP headers, handle gracefully. build: Do not throw exceptions if no arguments were passed for setup.py Don't need to compute ETag md5sum match if it is not a real md5sum Do not convert all regions to lower case Haruka Iwao (1): * S3/Config.py Get user input aws access key, secret and gpg passphrase out of debug format string James Brown (1): Respect the $TZ environment variable to show local times if requested Jan Kasiak (2): Change proxy SSL URI to relative Add restore subresource to V2 signature Jason Dalton (8): Replaced mktemp with alternative temp file generation based on secure randoms in order to avoid owner-restricted permission issues with mktemp files Added cleanup of temp file if error or keyboard interrupt occurs during transfer. cleanup Revert "cleanup" Revert "Replaced mktemp with alternative temp file generation based on secure randoms in order to avoid owner-restricted permission issues with mktemp files" Set permission on copied file based on umask. Updated references to temp file in exception handlers Relaxed timeout for http connection during IAM role lookup Jason Godfrey (1): Quick fix for cfcreate and cf modify not working. These commands would generate a KeyError for 'Origin'. (Bug 3536626) Jens Braeuer (2): Added support for "Static Websites" Remove recursion detection for symlinks. Jerome Leclanche (11): Avoid catching SystemExit in a try block Add __repr__ for S3Uri class Add S3.MultiPart module with basic functionality Make use of S3.MultiPart functionality by default Rename MultiPartUpload.bucket into MultiPartUpload.s3 Implement multipart threading Use the ThreadPool interface to thread multipart uploads and return a proper response Dynamically increase the chunk size depending on the file size Add the --enable-multipart option to s3cmd Add multipart_num_threads and multipart_chunk_size to Config Properly pass multipart enabling, default enabled on > 100MB Joe Fiorini (1): Added command to set bucket access policy Joe Mifsud (2): Add support for invalidating multiple CF distributions with --cf-inval when the bucket is the origin for multiple distributions Switching to set() Jose Fernandez (magec) (2): Make use of TMP variable to change tempdir Document Environment variables Josep del Rio (4): When using default index, invalidate path Corrected the new flags for CF index invalidation Removed debug messages, make sure default index is defined Fixed exception for root document Josh C. Wilson (1): move progress_meter default setting down to Config class Jukka Nousiainen (2): Set public_url() protocol based on use_https Parameter for public URL protocol Julien Maitrehenry (1): Change the marker value for NextMarker in bucket_list_streaming Julien Recurt (1): Adding option signurl_use_https for signurl to use https prefix Ka-Hing Cheung (1): enlarge write and read buffers Karl Matthias (1): Add support for putting arbitrary additional headers in s3cfg file. Karsten Sperling (15): Use python-magic for guessing MIME types if available Catch the right exception, doh. Added example /etc/magic file with better support for web file types Add support for setting Content-Encoding header where it can be guessed, in particular Content-Encoding: gzip is handled. Flush stdout after logging messages, in case buffered pipes are involved Merge branch 'master' of git://github.com/s3tools/s3cmd Added example /etc/magic file with better support for web file types Add support for setting Content-Encoding header where it can be guessed, in particular Content-Encoding: gzip is handled. Flush stdout after logging messages, in case buffered pipes are involved Fix up code that was merged incorrectly Merge branch 'mime-guessing' Add -M short option for --guess-mime-type Fix ValueError when magic module is not present Add support for http://pypi.python.org/pypi/filemagic version of python-magic Merge branch 'mime-guessing' Karthikeyan Singaravelan (1): Fix deprecation warnings due to invalid escape sequences. Kelly McLaughlin (2): Handle empty return bodies when processing S3 errors. Reuse a single connection when doing a bucket list Kentaro (2): Update base64.encodestring for Python 3.9 compatibility encodestring() method was removed from base64 in Python 3.9. Update base64 import to support Python 2 and 3 Kevin Daub (5): Fixed typo in output messages. Added --server-side-encryption as an option and adds the header to put and put-copy. Reverted out manpage update Added Server-Side Encryption (SSE) to output of --info Fixed formatting Kiana McNellis (2): Fixed bug where the error report will not show if there was an import error due to dependency on unicodise which was not successfully imported Changed the exit code for import errors to not rely on dependency of EX_GENERAL being defined when not imported Koen Rouwhorst (1): Use HTTPS by default Konstantin Shalygin (1): s3cmd: python 3.8 compat Krzysztof Warunek (1): support endpoint with custom path Lance Batson (1): fix issue with error display after config modify Leandro Ostera (1): Update s3cmd --configure Liu Lan (1): fix get bucket info error when LC use the filter element Luke Winslow (1): Fixed issue with mixed path separators with s3cmd get --recursive on Windows. Marco Jakob (7): Get the access token from the environment Do not refresh role Do not refresh role Add option for token-refresh Add option for token-refresh Add option for token-refresh Add option for token-refresh Mark Titorenko (2): Generalize wildcard certificate forgiveness Updated based on feedback. Masashi Ozawa (2): add "limit" option to "ls" and "la" commands to return the specified number of objects instead of returning all objects in the bucket. Enable to configure "host_base" as "S3 Endpoint" in interactive configuration. Matt Domsch (485): Apply excludes/includes at local os.walk() time add --delete-after option for sync add more --delete-after to sync variations Merge remote-tracking branch 'origin/master' into merge Merge branch 'delete-after' into merge add Config.delete_after Merge branch 'delete-after' into merge fix os.walk() exclusions for new upstream code Merge branch 'master' into merge Merge branch 'master' of git://github.com/s3tools/s3cmd add --delay-updates option finish merge Handle hardlinks and duplicate files hardlink/copy fix remote_copy() doesn't need to know dst_list anymore handle remote->local transfers with local hardlink/copy if possible sync: add --add-destination, parallelize uploads to multiple destinations add local tree MD5 caching fix getting uid Apply excludes/includes at local os.walk() time add --delete-after option for sync add more --delete-after to sync variations add Config.delete_after fix os.walk() exclusions for new upstream code add --delay-updates option Handle hardlinks and duplicate files hardlink/copy fix remote_copy() doesn't need to know dst_list anymore handle remote->local transfers with local hardlink/copy if possible sync: add --add-destination, parallelize uploads to multiple destinations add local tree MD5 caching fix getting uid HashCache: add missing break during purge merge manpage conflict] Merge branch 'cache-local-md5' into hardlink-fixes sync: refactor parent/child and single process code Merge branch 'econnell/master' into econnell-merge merge relistan/master Merge remote-tracking branch 'brendano/master' into brendano-merge Merge remote-tracking branch 'ksperling/master' into ksperling-merge Merge remote-tracking branch 'michaeltyson/master' into michaeltyson-merge Merge remote-tracking branch 'joseprio/master' into joseprio-merge Merge remote-tracking branch 'ghickman/packaging' into ghickman-merge Merge remote-tracking branch 'mattswe/patch-1' into mattswe-merge Merge remote-tracking branch 'joefiorini/access-policy' into joefiorini-merge Merge remote-tracking branch 'Roguelazer/tz_support' into Roguelazer-merge Merge remote-tracking branch 'godfreja/fix_cf_keyerror' into godfreja-merge Merge remote-tracking branch 'manos/master' into manos-merge Merge remote-tracking branch 'ghickman/manifest-cleanup' into merge fix mime detection for Fedora python-mime fix multipart uploads fix --cf-invalidate after merge cannot use both --cf-invalidate and --add-destination note mdomsch merge branch in version string Merge remote-tracking branch 'ejnoyes/master' into test-merge Merge remote-tracking branch 'spikegronim/fix-hang-bug' into merge Merge remote-tracking branch 'tycho/master' into merge finish merge fix up failed merge of --delete-after-fetch update mdomsch-merge version string fixup trivial merge conflict remove unused and incorrect function hash_file() wrap get_md5() with try/except IOError remove debug output that mixes unicode and ascii strings poorly be verbose when both hardlinking and copying of local files fails update_acl(): fix output message Handle local hardlink and copy failures by downloading the objects add output on remote copy, fixes test cases 27 and 28 add --files-from=FILE to allow transfer of select files only accept --files-from=- to read from stdin add --files-from to manpage use --files-from only on local source (not remote, not dest) fix sync --dry-run traceback, bug #124 fix single file sync local2remote traceback, bug #128 sync remote2remote: set target_uri on updated files too, bug 131 Split md5/hardlink stuff out of SortedDict into FileDict fix sync single remote file to existing local file traceback, bug 88 add --max-delete for del and sync commands, fixes bug #155 sync: ignore --delete-removed if no source files, bug #155 provide a better exception / error reporting message, bug #158 Merge branch 'files-from' into merge, pull request #116 Merge remote-tracking branch 'borgenk/master' into merge, PR #143 Merge remote-tracking branch 'yar/master' into merge, PR #147 Merge remote-tracking branch 'aisipos/master' into merge, PR #151 Merge remote-tracking branch 'developmentseed/master' into merge, PR #152 Merge branch 'bug/155' into merge, PR #156 Merge branch 'bug/153' into merge, PR #157 Merge remote-tracking branch 'abuchanan-grio/master' into merge, PR #162 Merge remote-tracking branch 'tjasond/master' into merge, PR #163 add --xattr option referring to file system extended attribute containing md5sums Merge branch 'xattr' into merge fix conditional import xattr module Merge branch 'xattr' into merge push md5_xattr to Config only if xattr module was imported Merge branch 'xattr' into merge Don't emit "cowardly" message unless we have source files & delete files Merge branch 'bug/155' into merge Get md5 from remote attribute header in recursive syncs too Merge branch 'bug/remote-md5sum' into merge FileLists: get md5 from headers Merge branch 'bug/remote-md5sum' into merge move x-amz-meta-s3cmd-attrs handling into S3.py use metadata-stored md5 in bucket list, put, and info Merge branch 'bug/remote-md5sum' into merge look at metadata md5 for get as well Merge branch 'bug/remote-md5sum' into merge s3cmd get to stdout: set file_exists to avoid traceback, bug #175 don't purge the HashCache if using --files-from HashCache: catch KeyError in unmark_for_purge(), bug #194 HashCache and FileDict: ignore files with device = 0 or inode = 0 Merge branch 'files-from' into merge Merge branch 'bug/194' into merge fixup merge Merge remote-tracking branch 'ivebeenlinuxed/master' into merge Merge remote-tracking branch 'reed-sandberg/master' into merge Merge remote-tracking branch 'kuenishi/more_multipart' into merge Merge remote-tracking branch 'scharissis/fix_help_typo' into merge finish merge of #186 Merge remote-tracking branch 'orenhe/simplify_wildcard_handling' into merge Merge remote-tracking branch 'robysath/master' into merge Merge remote-tracking branch 'spartman/remote-copy-fix' into merge remote_copy(): fix value in failed_copy_list Merge branch 'no-mime-magic-option' of git://github.com/radomirml/s3cmd into merge Merge branch 'master' of git://github.com/brandongalbraith/s3cmd into merge Merge branch 'listallmybuckets-error' of git://github.com/leostera/s3cmd into merge Fix for ignored --default-mime-type option Merge branch 'bug/default-mime-type' into merge really merge @edrex https://github.com/s3tools/s3cmd/pull/244 include python version in error dump remove python-2.5-ism when guessing mimetype (bug #218) don't hardlink 0-length files encode filenames to system encoding before calling shutils.copy2() getting the file system encoding is loop invariant Merge pull request #250 from mdomsch/bug/python-version-on-error Merge pull request #251 from mdomsch/bug/218 Merge pull request #253 from mdomsch/bug/unicode-stat Print a useful error message when MemoryError is raised Merge pull request #259 from 36degrees/patch-1 fixes for Windows not including a pwd module fix divide-by-zero when no files were transferred handle OSError: No such file or directory in get_local_filelist Merge branch 'bug/memoryerror' into merge Merge branch 'bug/divide-by-zero' into merge Merge branch 'bug/lstat-removed-file' into merge Merge branch 'bug/windows-no-pwd' into merge Merge git://github.com/robd003/s3cmd into merge --no-check-md5 should disable md5sum calculation, including preserved attributes Merge branch 'bug/217' into merge fix put - s3:// (upload from stdin, #270) Merge branch 'bug/put-from-stdin' into merge os.gete[ug]id() doesn't exist on non-Unix Merge branch 'bug/put-from-stdin' into merge Merge git://github.com/flebel/s3cmd into merge add 'make tarball' and 'make rpm' to Makefile remove useless import of S3.Exceptions reduce "Process files that was not remote copied" to debug() add s3cmd.spec to .gitignore add a friendly error message on UnicodeEncodeError exception report LANG= environment variable on crash handle empty directory on S3 sync remote2local _download: Exception != OSError _download: handle S3Error Merge pull request #92 from ksauzz/support-get-operation-to-multibyte-objectname Ignore S3 objects with empty names Merge pull request #284 from swiftserve/fix_config_file_verbosity set content-type, content-charset, and not content-encoding use ~/.s3cfg in run-tests too move empty object name detection where it belongs delete unused keys_to_remove logic s/0o775/0755/ for python 2.4 compatibility Merge pull request #286 from swiftserve/fix_verbosity_config Merge pull request #291 from Knewton/master Merge pull request #293... [truncated message content] |
From: Russell G. <rus...@gm...> - 2021-03-03 21:41:45
|
I had this series of error messages: Traceback (most recent call last): File "/usr/local/bin/s3cmd", line 4, in <module> __import__('pkg_resources').run_script('s3cmd==2.1.0', 's3cmd') File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3088, in <module> @_call_aside File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3072, in _call_aside f(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3101, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 574, in _build_master ws.require(__requires__) File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 892, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 778, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 's3cmd==2.1.0' distribution was not found and is required by the application This is Linux Mint 19.3. I noticed a Python 2.7 update come through so perhaps this caused the issue. Alternatively I have been messing with Python 3.6 which I guess I would prefer to use if there is a choice. At one time recently I changed my python link to python3 from python2 but found it borked another app I use so I changed it back. Maybe that was the issue, Please suggest what the issue is - any suggestions gratefully received. Does s3cmd work on Python3? Russell |
From: Florent V. (Sodria) <fl...@so...> - 2021-02-06 23:13:41
|
Hi David, Yes, the "sync" command will preserve users, creation and modification dates if you use the following flag: -p, --preserve Take care that you need to use it both when uploading to the server and downloading. And any way, it would be better for you to give a try with test files. Florent On Sat, Feb 6, 2021 at 11:34 AM David Roth <dav...@gm...> wrote: > Is there a way for s3cmd to restore the original Linux file time > stamp when it is restored to a Linux server? I'm concerned about using > s3cmd for legacy files, and losing the original time stamp on the files. > Thanks. > > David Roth > > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > |
From: David R. <dav...@gm...> - 2021-02-06 10:34:03
|
Is there a way for s3cmd to restore the original Linux file time stamp when it is restored to a Linux server? I'm concerned about using s3cmd for legacy files, and losing the original time stamp on the files. Thanks. David Roth |
From: Sharadha K. <sha...@hi...> - 2020-12-19 07:00:19
|
Hi, These are database files. So I am not sure about its content. So it means that the files are the same but with different names and it is trying to upload it only once? But that fails and so both the files are copied individually. Is my understanding correct? Regards, Sharadha On Fri, Dec 18, 2020 at 8:40 PM Florent Viard (Sodria) <fl...@so...> wrote: > Hi, > > Indeed there is no worry to have with this warning. > > To be smart, s3cmd identify files that have the same content, and in that > case try to upload it only once and do remote copies to create the other > files more efficiently. > But some servers do not support remote copies, so, in that case there is > this warning and we fallback to upload all the files individually. > > Regards, > Florent > > > Le ven. 18 déc. 2020 à 15:59, William Smith <w_...@co...> a > écrit : > >> Just a WAG: Are the two files the same? Maybe it's detected the fact >> that they are the same file (MD5 or such) and decided to make a local copy >> rather than upload a new one... >> >> William P. N. Smith >> ComputerSmiths Consulting, Inc. >> 978-600-1621 >> >> >> >> >> On Dec 18, 2020, at 8:51 AM, Sharadha Krishna < >> sha...@hi...> wrote: >> >> Hi, >> >> I am using the s3cmd sync command to sync a backup folder to the S3 >> bucket. I am testing this using a new s3 bucket. >> >> When I execute *s3cmd sync backups/ $S3_PATH/backups/* >> >> I get the following error message that delays my sync of just a few files >> to 3-5 mins. >> >> WARNING: Unable to remote copy files >> 's3://<bucket-name>/backups/mc-13-big-Filter.db' -> >> 's3://<bucket-name>/backups/mc-17-big-Filter.db' >> >> Why is it trying to sync a different file to another one? Can someone >> help me with this? >> >> Thanks, >> Sharadha >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> >> >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > |
From: Florent V. (Sodria) <fl...@so...> - 2020-12-18 15:10:09
|
Hi, Indeed there is no worry to have with this warning. To be smart, s3cmd identify files that have the same content, and in that case try to upload it only once and do remote copies to create the other files more efficiently. But some servers do not support remote copies, so, in that case there is this warning and we fallback to upload all the files individually. Regards, Florent Le ven. 18 déc. 2020 à 15:59, William Smith <w_...@co...> a écrit : > Just a WAG: Are the two files the same? Maybe it's detected the fact > that they are the same file (MD5 or such) and decided to make a local copy > rather than upload a new one... > > William P. N. Smith > ComputerSmiths Consulting, Inc. > 978-600-1621 > > > > > On Dec 18, 2020, at 8:51 AM, Sharadha Krishna <sha...@hi...> > wrote: > > Hi, > > I am using the s3cmd sync command to sync a backup folder to the S3 > bucket. I am testing this using a new s3 bucket. > > When I execute *s3cmd sync backups/ $S3_PATH/backups/* > > I get the following error message that delays my sync of just a few files > to 3-5 mins. > > WARNING: Unable to remote copy files > 's3://<bucket-name>/backups/mc-13-big-Filter.db' -> > 's3://<bucket-name>/backups/mc-17-big-Filter.db' > > Why is it trying to sync a different file to another one? Can someone help > me with this? > > Thanks, > Sharadha > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > |
From: William S. <w_...@co...> - 2020-12-18 14:58:58
|
Just a WAG: Are the two files the same? Maybe it's detected the fact that they are the same file (MD5 or such) and decided to make a local copy rather than upload a new one... William P. N. Smith ComputerSmiths Consulting, Inc. 978-600-1621 > On Dec 18, 2020, at 8:51 AM, Sharadha Krishna <sha...@hi...> wrote: > > Hi, > > I am using the s3cmd sync command to sync a backup folder to the S3 bucket. I am testing this using a new s3 bucket. > > When I execute s3cmd sync backups/ $S3_PATH/backups/ > > I get the following error message that delays my sync of just a few files to 3-5 mins. > > WARNING: Unable to remote copy files 's3://<bucket-name>/backups/mc-13-big-Filter.db' -> 's3://<bucket-name>/backups/mc-17-big-Filter.db' > > Why is it trying to sync a different file to another one? Can someone help me with this? > > Thanks, > Sharadha > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general |
From: Sharadha K. <sha...@hi...> - 2020-12-18 13:51:32
|
Hi, I am using the s3cmd sync command to sync a backup folder to the S3 bucket. I am testing this using a new s3 bucket. When I execute *s3cmd sync backups/ $S3_PATH/backups/* I get the following error message that delays my sync of just a few files to 3-5 mins. WARNING: Unable to remote copy files 's3://<bucket-name>/backups/mc-13-big-Filter.db' -> 's3://<bucket-name>/backups/mc-17-big-Filter.db' Why is it trying to sync a different file to another one? Can someone help me with this? Thanks, Sharadha |
From: Sharadha K. <sha...@hi...> - 2020-12-16 10:46:28
|
Hi Florent, Yes, I have a bunch of files under a subfolder. Alright got the point. On Wed, Dec 16, 2020 at 2:34 PM Florent Viard (Sodria) <fl...@so...> wrote: > Hi Sharadha, > > Maybe it is because you just have this single file in folder1/folder2? > > In s3 you don't have the concept of 'files' and 'folders'. There are just > files with a filename that can contains '/'. > So we give the illusion of a file hierarchy based on the filename prefix > but the folders in themselves doesn't exist. > > So, if you delete the single file in this part of the tree, you will not > have any other files with the prefix folder1/folder2 and so you will think > that the folders were deleted but it is just that they 'ever existed in the > first place. > > If you had another file in folder1, it would still appear after the > command. > > Regards, > > Florent > > > Le mer. 16 déc. 2020 à 07:42, Sharadha Krishna < > sha...@hi...> a écrit : > >> Hi, >> >> I am trying to delete files inside a particular folder in the s3 bucket. >> For this I am using the command *s3cmd del >> s3://bucket_name/folder1/folder2/file_name*. >> >> When I execute this, the entire directory structure gets deleted, meaning >> folder1 and folder2 also get deleted along with the filename. >> >> So can someone let me know how to solve this? >> >> Thanks, >> Sharadha >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > |
From: Florent V. (Sodria) <fl...@so...> - 2020-12-16 09:04:27
|
Hi Sharadha, Maybe it is because you just have this single file in folder1/folder2? In s3 you don't have the concept of 'files' and 'folders'. There are just files with a filename that can contains '/'. So we give the illusion of a file hierarchy based on the filename prefix but the folders in themselves doesn't exist. So, if you delete the single file in this part of the tree, you will not have any other files with the prefix folder1/folder2 and so you will think that the folders were deleted but it is just that they 'ever existed in the first place. If you had another file in folder1, it would still appear after the command. Regards, Florent Le mer. 16 déc. 2020 à 07:42, Sharadha Krishna <sha...@hi...> a écrit : > Hi, > > I am trying to delete files inside a particular folder in the s3 bucket. > For this I am using the command *s3cmd del > s3://bucket_name/folder1/folder2/file_name*. > > When I execute this, the entire directory structure gets deleted, meaning > folder1 and folder2 also get deleted along with the filename. > > So can someone let me know how to solve this? > > Thanks, > Sharadha > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > |
From: Sharadha K. <sha...@hi...> - 2020-12-16 06:42:33
|
Hi, I am trying to delete files inside a particular folder in the s3 bucket. For this I am using the command *s3cmd del s3://bucket_name/folder1/folder2/file_name*. When I execute this, the entire directory structure gets deleted, meaning folder1 and folder2 also get deleted along with the filename. So can someone let me know how to solve this? Thanks, Sharadha |
From: Luca P. <pi...@cr...> - 2020-11-23 10:02:30
|
Hi, with s3cmd, can I make anonymous requests, without specifying an access key? I have a Ceph S3 object store with some buckets configured for read-only anonymous, unauthenticated access. I can access these with the aws client: ``` $ aws --endpoint-url http://$AWS_S3_ENDPOINT --no-verify-ssl --region "" --no-sign-request s3 ls s3://test-policy 2020-11-20 14:50:55 298 policy_anon_read_bucket.json ``` I haven't been able to do the same operation with s3cmd. If I provide an empty key on the command line I get an InvalidArgument error: ``` $ s3cmd --host-bucket= --no-ssl -v --host $AWS_S3_ENDPOINT --access_key "" --secret_key "" ls s3://test-policy ERROR: S3 error: 400 (InvalidArgument) ``` If I don't specify one on the command line, it picks up whatever key is in $HOME/.s3cfg (even if its invalid), and if I don't have a key in the configuration or if I remove the configuration altogether it gives me another error. Thanks in advance for any suggestions. Cheers, Luca |
From: Frank C. <fr...@cr...> - 2020-10-13 05:57:42
|
Florent, Thanks for the response. A few weeks or even a month or so is not an issue, it would only be if you were planning to stretch it to years, or were just unresponsive when I would think of taking it straight from the repo. Thanks for all you work keeping it working. RegardsFrank On Mon, 2020-10-12 at 01:32 +0200, Florent Viard (Sodria) wrote: > Hi Frank, > Sorry for the long delay of my reply. > > First, thank you for keeping the package up to date for the Fedora > and EPEL distributions. > > I'm still working on a few things, and so I plan to be able to do the > next release in 1 or 2 weeks maximum. > In general the git repo version is somehow stable and usable, but I > would prefer and advise to not take anything else than release for > distribution. > Release are more tested and it might be harder to investigate > reported issues on random non release versions. > > Also, especially for the next release, I have a few bugs in mind and > at least something to cover python 3.9, so it will definitively be > better if you could wait for the coming release. > > Don't hesitate to ask if you need further details. > > Regards, > > Florent > > On Tue, Oct 6, 2020 at 12:59 PM Frank Crawford < > fr...@cr...> wrote: > > Folks, > > > > I'm currently maintaining of s3cmd for Fedora and EPEL, and the > > latest release is s3cmd-2.1.0, however there are a few bugs that > > have been reported that have been addressed in the current git > > repo. > > > > So, I am wondering what is your timeline for the next release, or > > if I am better shipping what is currently in the git repo. Related > > to that, how stable do you see the git repo? Is it suitable for > > distribution, or what would you suggest? > > > > Regards > > Frank > > _______________________________________________ > > > > S3tools-general mailing list > > > > S3t...@li... > > > > https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > > _______________________________________________S3tools-general > mailing lis...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general |
From: Florent V. (Sodria) <fl...@so...> - 2020-10-11 23:55:28
|
Hi Frank, Sorry for the long delay of my reply. First, thank you for keeping the package up to date for the Fedora and EPEL distributions. I'm still working on a few things, and so I plan to be able to do the next release in 1 or 2 weeks maximum. In general the git repo version is somehow stable and usable, but I would prefer and advise to not take anything else than release for distribution. Release are more tested and it might be harder to investigate reported issues on random non release versions. Also, especially for the next release, I have a few bugs in mind and at least something to cover python 3.9, so it will definitively be better if you could wait for the coming release. Don't hesitate to ask if you need further details. Regards, Florent On Tue, Oct 6, 2020 at 12:59 PM Frank Crawford <fr...@cr...> wrote: > Folks, > > I'm currently maintaining of s3cmd for Fedora and EPEL, and the latest > release is s3cmd-2.1.0, however there are a few bugs that have been > reported that have been addressed in the current git repo. > > So, I am wondering what is your timeline for the next release, or if I am > better shipping what is currently in the git repo. Related to that, how > stable do you see the git repo? Is it suitable for distribution, or what > would you suggest? > > Regards > Frank > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > |
From: Frank C. <fr...@cr...> - 2020-10-06 10:59:33
|
Folks, I'm currently maintaining of s3cmd for Fedora and EPEL, and the latest release is s3cmd-2.1.0, however there are a few bugs that have been reported that have been addressed in the current git repo. So, I am wondering what is your timeline for the next release, or if I am better shipping what is currently in the git repo. Related to that, how stable do you see the git repo? Is it suitable for distribution, or what would you suggest? Regards Frank |
From: Jeff W. <je...@je...> - 2020-07-12 17:53:58
|
Is this possible? or is admin authority always required? If it's possible, how do I install and run it? Thank you! |
From: Aaron Ni <Aaron.Ni@TymeDigital.com> - 2020-06-01 21:11:32
|
Hi S3Tools Development Team, We are testing the s3cmd tool and encountering the following error message: ERROR: S3 error: 400 (InvalidArgument): Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4. Our usage scenario is: * Copy file to S3 bucket * S3 bucket configured with default encryption which uses a customer managed KMS key * We are using s3cmd v2.1.0 Thanks in advance for your assistance Regards, Aaron. ************** Disclaimer ************************************ This email is intended solely for the use of the individual or entity named above as it may contain information that is confidential and privileged. If you are not the intended recipient, be advised that any dissemination, distribution or copying of this email is strictly prohibited. TymeBank cannot be held liable by any person other than the addressee in respect of any opinions, conclusions, advice or other information contained in this email. ************************************************************** |
From: Davide C. - ASB\\ <dav...@as...> - 2020-05-25 09:20:18
|
Hi, I try to install and configure s3cmd as always. I have to move a zip file from ec2 instance to related bucket (both EC2 and S3 bucket are in eu-south-1). After I launching the script this is the error message: "ERROR: S3 error: 400 (IllegalLocationConstraintException): The eu-south-1 location constraint is incompatible for the region specific endpoint this request was sent to." If I move the bucket from Milan to Frankfurt (eu-central-1) it works. If the bucket is in Milan (eu-south-1) it seems it doesn't work. Is there something like a access block or similar? Is there a way to use s3cmd on eu-south-1? Thank a lot Davide Carbonini |
From: Florent V. (Sodria) <fl...@so...> - 2020-04-29 22:44:59
|
There are some third parties online estimator on internet, but in your case, the calculus should not be too hard. In the following page, you have all the fees for all the cases: https://aws.amazon.com/fr/s3/pricing/ Basically, there are only 2 kind of fees that will be charged to you for this operation: 1) (GET,LIST, ....) fee required for the requests needed to do the operation 2) A data transfer out fee that is dependent of the size of the object that you want to download. For 1), you can just ignore it is it will probably be insignificant. You just download a single file, so the overall operation will just requires a dozen of requests. You pay proportionally to your usage with something like 0.005 or 0.0004 per thousand of request. So we can easily assume that you will probably be charged a lot less than 0.0005. For 2), this is where the cost will be significant. Here, I guess that you will transfer to "Internet" (ie you download to your computer). (Other case are transfer/copy bucket to bucket) For this it is a price per GB, with different levels. So, if we take the example of the 100 GB file: You will pay 0$ for the 1st GB +0.09 * 99 = 8.91$ for the remaining 99 GB. You can get the size of the file by using the "info" command instead of "get". A last point to take care of, prices are not the same depending of the "Region" where is located the file. In the amazon page, there is a select box to select the region to see the correct price. If you can't get the region info from the source, you can do 2 things: - Do your estimate with the most expensive region that is probable for your case, and so you know the upper bound of the cost. - When you call the "info" request, use the -d flag. It will show all the debug logs of s3cmd, and you should be able to see in the first exchange in which region is the file located, as we have to redirect the requests to the correct region. As a complement, you can find another example of calculating costs in the readme of s3cmd: https://github.com/s3tools/s3cmd/blob/master/README.md (Just look at the idea, but refer to amazon page for the actual pricing) Let me know if it is not clear enough, Regards, Florent On Wed, Apr 29, 2020 at 10:58 PM Andrew Davidson via S3tools-general < s3t...@li...> wrote: > I need to estimate how the following transfer will cost. Any idea how I > can go about this? Seem like AWS would have some sort of tool for this > however I could not find it > > > s3cmd get --requester-pays > s3://cgl-rnaseq-recompute-fixed/cwl-example.tar.gz > > Kind regards > > Andy Davidson > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > |
From: Andrew D. <aed...@uc...> - 2020-04-29 20:57:56
|
I need to estimate how the following transfer will cost. Any idea how I can go about this? Seem like AWS would have some sort of tool for this however I could not find it s3cmd get --requester-pays s3://cgl-rnaseq-recompute-fixed/cwl-example.tar.gz Kind regards Andy Davidson |
From: Florent V. (Sodria) <fl...@so...> - 2020-04-08 00:26:02
|
Dear s3cmd users and contributors, I'm glad to announce that s3cmd v2.1.0 has been released on github, sourceforge and PyPi. This release is mostly made of bug fixes and should not contain any breaking change. Thanks again to all the contributors, the testers and the issue reporters. s3cmd-2.1.0 - 2020-04-07 =============== * Changed size reporting using k instead of K as it a multiple of 1024 (#956) * Added "public_url_use_https" config to generate public url using https (#551, #666) (Jukka Nousiainen) * Added option to make connection pooling configurable and improvements (Arto Jantunen) * Added support for path-style bucket access to signurl (Zac Medico) * Added docker configuration and help to run test cases with multiple Python versions (Doug Crozier) * Relaxed limitation on special chars for --add-header key names (#1054) * Fixed all regions that were automatically converted to lower case (Harshavardhana) * Fixed size and alignment of DU and LS output reporting (#956) * Fixes for SignatureDoesNotMatch error when host port 80 or 443 is specified, due to stupid servers (#1059) * Fixed the useless retries of requests that fail because of ssl cert checks * Fixed a possible crash when a file disappears during cache generation (#377) * Fixed unicode issues with IAM (#987) * Fixed unicode errors with bucked Policy/CORS requests (#847) (Alex Offshore) * Fixed unicode issues when loading aws_credential_file (#989) * Fixed an issue with the tenant feature of CephRGW. Url encode bucket_name for path-style requests (#1080) * Fixed signature v2 always used when bucket_name had special chars (#1081) * Allow to use signature v4 only, even for commands without buckets specified (#1082) * Fixed small open file descriptor leaks. * Py3: Fixed hash-bang in headers to not force using python2 when setup/s3cmd/run-test scripts are executed directly. * Py3: Fixed unicode issues with Cloudfront (#1006) * Py3: Fixed http.client.RemoteDisconnected errors (#1014) (Ryan Huddleston) * Py3: Fixed 'dictionary changed size during iteration' error when using a cache-file (#945) (Doug Crozier) * Py3: Fixed the display of file sizes (Vlad Presnyak) * Py3: Python 3.8 compatibility fixes (Konstantin Shalygin) * Py2: Fixed unicode errors sometimes crashing remote2remote sync (#847) * Added s3cmd.egg-info to .gitignore (Philip Dubé) * Improved run-test script to not use hard-coded bucket names(#1066) (Doug Crozier) * Renamed INSTALL to INSTALL.md and improvements (Nitro, Prabhakar Gupta) * Improved the restore command help (Hrchu) * Updated the storage-class command help with the recent aws s3 classes (#1020) * Fixed typo in the --continue-put help message (Pengyu Chen) * Fixed typo (#1062) (Tim Gates) * Improvements for setup and build configurations * Many other bug fixes Shortlog below. Thanks, Florent Alex Offshore (1): Return decoded response in get_policy and get_cors Arto Jantunen (2): Add a close method to the ConnMan api Add option to disable connection pooling Doug Crozier (4): Changed "Buckets list" test to use pbucket rather than hardcoded names. This fixes issue #1066. Allow use of Docker to run test cases with multiple Python versions Fix for 'dictionary changed size during iteration' error on Python 3 with --cache-file (Issue #945) Test cases added for --cache-file EC2 Default User (1): .gitignore: add s3cmd.egg-info Florent Viard (36): Fixes #987 - Fixes using IAM with python3. Config options should be unicode. Fixes #987 - Before py36, py3 json was only able to load unicode str Complete the fix #986 - To satisfy py2 and p3, there should be float before and after the division Issue #956 - Fixes size reporting using k instead of K as it a multiple of 1024 Fixes #956 - fix size and alignement of du and ls output reporting Convert anything given to getTreeFromXml to utf8 bytes if needed Fixes #1006 - fixes multiple python3 encoding issues with cloudfront functions Fixes #1006 - missing return in previous commit 222af5d8ce90bbb2f7784360c019f5a6af2ef8bd Change version to development 2.0.2+ setup.cfg: set universal bdist_wheel flag setup.py: update meta files with support for recent python versions travis-ci: Disable tests for python 2.6 as not supported anymore by travis-ci default ubuntu distribution Complete the previous change to rename INSTALL file to INSTALL.md run-tests.py: Don't fail ACL tests if public_url_use_https is used svnignore: update to match latest gitignore content Little cleanup. TODO is out of date and obsolete Add a .dockerignore because of this stupid docker that does not know how to exclude files he has no business with Update run-tests.docker.README.md with a small display fix Create FUNDING.yml Factorize the region detection inner request and stop retrying requests that fail the ssl certificate checks Fixes #1059 - Hack for SignatureDoesNotMatch error when host port 80 or 443 is specified, due to stupid servers run-test-*: Minor fixes Small cleanup Fixes #377 - Avoid crashing when a file disappeared during cache generation Fixes #1080 - url encode bucket_name for path-style requests. Fixes an issue with CephRGW and the tenant feature. Fixes #1081 - Don't force signature v4 if bucket_name uses special chars Fixes #1082 - Allow to use signature v4 only, even for a command with no target bucket specified Fixes #1054 - Relax limitation on special chars for --add-header key names Fixes #1020 - Update storage-class cmd help msg with new aws s3 classes Fixes open file descriptors leaking by not being closed on time Added a comment about how to escape the comment char in exclude files Fixes #989 - in aws_credential_file, fixes read_string only available with py3 + refactor for possible unicode issues Fixes #847 - set a few str as unicode to fix UnicodeDecodeError with python2 when doing a sync remote 2 remote Fixes py headers to use system default python and not force python2 anymore setup.py: Update authors and cleanup Update version to 2.1.0 Harshavardhana (1): Do not convert all regions to lower case Jukka Nousiainen (2): Set public_url() protocol based on use_https Parameter for public URL protocol Konstantin Shalygin (1): s3cmd: python 3.8 compat Pengyu Chen (1): Fixed: Minor typo in help message for `--continue-put` Prabhakar Gupta (1): Fix INSTALL.md formatting issues Ryan Huddleston (1): With this commit 4 years ago OSError was caught: https://github.com/s3tools/s3cmd/commit/889c5a7ba1ef8fc48f0c1be593e7ddf1e518a408 Tim Gates (1): Fix simple typo: unecessarily -> unnecessarily Vlad Presnyak (1): Fixing proper handling of file sizes. Will correctly return floating point or integer as appropriate. Zac Medico (1): Fix signurl to support path-style bucket access via check_bucket_name_dns_support hrchu (1): Update restore cmd help matteobar (1): Copyright year updated nitro (1): README: brew install and renamed to INSTALL.md |
From: David R. <dav...@gm...> - 2019-12-30 12:31:52
|
I read someplace that the Linux file permissions and dates could be stored in the AWS S3 metadata, and that s3cmd could accomplish this. However, I can't find examples of this being done or in the documentation. Perhaps I overlooked it. Could someone please clarify if s3cmd can do this? If so, either provide an example or documentation/article that covers this? Thanks! David Roth |