s3tools-general — Both developers and users should use this list

[S3tools-general] Backblaze B2 S3 compatible pre-signed URLs

[<sil...@sr...>]

Hello group,

having problem with pre-signed URLs on B2.

Setup:

- application key with read access to the bucket and `listAllBucketNames` enabled
- private bucket
- file in that bucket
	- visible with `s3cmd ls s3://bucket-name/`
	- downloadable with `s3cmd get s3://bucket-name/file-name`
- no special characters in bucket-name nor file-name. Just plain alphanumeric ASCII

Now I try to get that file with a pre-signed URL:

$ s3cmd signurl s3://bucket-name/file-name `gdate -d 'now + 1 year' +%s`
https://bucket-name.s3.us-west-001.backblazeb2.com/file-name?AWSAccessKeyId=<correct-access-key-id-here>&Expires=1757021192&Signature=aJtG2JFU3xrWIMZlD%2F0wwWYhTIs%3D

But trying to use this URL returns always

<Error>
<Code>UnauthorizedAccess</Code>
<Message>bucket is not authorized: bucket-name</Message>
</Error>

Any clues as to what I might be doing wrong?

-- 
SD!

[S3tools-general] s3cmd v2.3.0

[Florent V. (Sodria) <fl...@so...>]

Dear s3cmd users and contributors,

I'm glad to announce that

s3cmd v2.3.0 has been released on github, sourceforge and PyPi.

This release should not contain any breaking change.

Thanks again to all the contributors, the testers and the issue reporters.


s3cmd-2.3.0      - 2022-10-03
===============
* Added "getnotification", "setnotification", and "delnotification"
commands for notification policies (hrchu)
* Added support for AWS_STS_REGIONAL_ENDPOINTS (#1218, #1228) (Johan Lanzrein)
* Added ConnectionRefused [111] exit code to handle connection errors
(Salar Nosrati-Ershad)
* Added support for IMDSv2. Should work automatically on ec2 (Anthony Foiani)
* Added --list-allow-unordered to list objects unordered. Only
supported by Ceph based s3-compatible services (#1269) (Salar
Nosrati-Ershad)
* Fixed --exclude dir behavior for python >= 3.6 (Daniil Tararukhin)
* Fixed Cloudfront invalidate retry issue (Yuan-Hsiang Lee)
* Fixed 0 byte cache files crashing s3cmd (#1234) (Carlos Laviola)
* Fixed --continue behavior for the "get" command (#1009) (Anton Ustyugov)
* Fixed unicode issue with fixbucket (#1259)
* Fixed CannotSendRequest and ConnectionRefusedError errors at startup (#1261)
* Fixed error reporting for object info when the object does not exist
* Fixed "setup.py test" to do nothing to avoid failure that could be
problematic for distribution packaging (#996)
* Improved expire command to use Rule/Filter/Prefix for
LifecycleConfiguration (#1247)
* Improved PASS/CHECK/INCLUDE/EXCLUDE debug log messages
* Improved setup.py with python 3.9 and 3.10 support info(Ori Avtalion)
* Many other bug fixes


Shortlog below.


Thanks,
Florent


Anthony Foiani (2):
      use IMDSv2 to retrieve instance metadata
      make IMDSv2 optional

Carlos Laviola (1):
      treat 0 byte cache files as nonexistent

Daniil Tararukhin (1):
      fix os walk when excluding directories

Florent Viard (13):
      Update CI test.yml to run on PR
      Fixes #1261 - Always read a conn after a request to be able to reuse it
      Fixes #1259 - Fixes unicode issue with fixbucket
      Fixes #1259 - Correct typo when retrieving response from IMDSv2
      Ensure 404 exc content is filled even for object_info
      Fixes #1009 - Fixes --continue behavior for the "get" command
      Fixes #996 - Avoid crashing when setup.py test is called
      Fixes #1247 - Update expire command to use Rule/Filter/Prefix
for LifecycleConfiguration
      Fixes #1261 - Additional fix for CannotSendRequest at startup
      Fixes ConnectionRefusedError not found error for python 2
      Use str instead of repr in FileList debug logs
      Use quotes around files in FileLists for debug logs
      Update version to 2.3.0

Johan Lanzrein (1):
      Take into account the AWS_STS_REGIONAL_ENDPOINTS

Ori Avtalion (1):
      List support of Python 3.9, 3.10

Salar Nosrati-Ershad (3):
      excepting/handling tcp connection refused error
      feat: implement list objects allow-onordered option as a config
      feat: add `--list-allow-unorderd` command-line switch

Yuan-Hsiang Lee (1):
      fixes cloudfront invalidate retry

hrchu (1):
      Add notification commands

[S3tools-general] How to create a bucket in a different region using the config file?

[Felipe G. <fel...@gm...>]

Hello,

I am trying to use the `s3cmd`  tool to create a bucket in a different
region than the default US. And I am using a config file for that.

The docs [1] say that there are other regions available.
```
--region=REGION, --bucket-location=REGION
                        Region to create bucket in. As of now the regions are:
                        us-east-1, us-west-1, us-west-2, eu-west-1, eu-
                        central-1, ap-northeast-1, ap-southeast-1, ap-
                        southeast-2, sa-east-1
```
But when I try to use a region different than `US` I receive an error
message. Here is the line of my config file to use another region. I based
this file also on the official doc [2].
```
[default]
access_key = ******
access_token =
add_encoding_exts =
add_headers =
bucket_location = eu-central-1
```
The error seems to be the same of [3] but I dont understand how they solve
to create the bucket in a different region. I also used the `
--bucket-location=":default-placement" ` and the bucket is still created at
the US region.
```

ERROR: S3 error: 400 (InvalidLocationConstraint)

```
Could someone help me?

[1] https://s3tools.org/usage
[2] https://s3tools.org/kb/item14.htm
[3] https://github.com/rook/rook/issues/2945

Thanks,
Felipe

[S3tools-general] s3cmd v2.2.0

[Florent V. (Sodria) <fl...@so...>]

Dear s3cmd users and contributors,

I'm glad to announce that
s3cmd v2.2.0 has been released on github, sourceforge and PyPi.

This release is mostly made of bug fixes and should not contain any
breaking change.

Thanks again to all the contributors, the testers and the issue reporters.


s3cmd-2.2.0      - 2021-09-27
===============
* Added support for metadata modification of files bigger than 5 GiB
* Added support for remote copy of files bigger than 5 GiB using
MultiPart copy (Damian Martinez, Florent Viard)
* Added progress info output for multipart copy and current-total info
in output for cp, mv and modify
* Added support for all special/foreign character names in object
names to cp/mv/modify
* Added support for SSL authentication (Aleksandr Chazov)
* Added the http error 429 to the list of retryable errors (#1096)
* Added support for listing and resuming of multipart uploads of more
than 1000 parts (#346)
* Added time based expiration for idle pool connections in order to
avoid random broken pipe errors (#1114)
* Added support for STS webidentity authentication (ie AssumeRole and
AssumeRoleWithWebIdentity) (Samskeyti, Florent Viard)
* Added support for custom headers to the mb command (#1197) (Sébastien Vajda)
* Improved MultiPart copy to preserve acl and metadata of objects
* Improved the server errors catching and reporting for cp/mv/modify commands
* Improved resiliency against servers sending garbage responses
(#1088, #1090, #1093)
* Improved remote copy to have consistent copy of metadata in all
cases: multipart or not, aws or not
* Improved security by revoking public-write acl when private acl is
set (#1151) (ruanzitao)
* Improved speed when running on an EC2 instance (#1117) (Patrick Allain)
* Reduced connection_max_age to 5s to avoid broken pipes as AWS closes
https conns after around 6s (#1114)
* Ensure that KeyboardInterrupt are always properly raised (#1089)
* Changed sized of multipart copy chunks to 1 GiB
* Fixed ValueError when using more than one ":" inside add_header in
config file (#1087)
* Fixed extra label issue when stdin used as source of a MultiPart upload
* Fixed remote copy to allow changing the mime-type (ie content-type)
of the new object
* Fixed remote_copy to ensure that meta-s3cmd-attrs will be set based
on the real source and not on the copy source
* Fixed deprecation warnings due to invalid escape sequences
(Karthikeyan Singaravelan)
* Fixed getbucketinfo that was broken when the bucket lifecycle uses
the filter element (Liu Lan)
* Fixed RestoreRequest XML namespace URL (#1203) (Akete)
* Fixed PARTIAL exit code that was not properly set when needed for
object_get (#1190)
* Fixed a possible inifinite loop when a file is truncated during
hashsum or upload (#1125) (Matthew Krokosz, Florent Viard)
* Fixed report_exception wrong error when LANG env var was not set (#1113)
* Fixed wrong wiki url in error messages (Alec Barrett)
* Py3: Fixed an AttributeError when using the "files-from" option
* Py3: Fixed compatibility issues due to the removal of getchildren()
from ElementTree in python 3.9 (#1146, #1157, #1162, # 1182, #1210)
(Ondřej Budai)
* Py3: Fixed compatibility issues due to the removal of encodestring()
in python 3.9 (#1161, #1174) (Kentaro Kaneki)
* Fixed a crash when the AWS_ACCESS_KEY env var is set but not
AWS_SECRET_KEY (#1201)
* Cleanup of check_md5 (Riccardo Magliocchetti)
* Removed legacy code for dreamhost that should be necessary anymore
* Migrated CI tests to use github actions (Arnaud J)
* Improved README with a link to INSTALL.md (Sia Karamalegos)
* Improved help content (Dmitrii Korostelev, Roland Van Laar)
* Improvements for setup and build configurations
* Many other bug fixes


Shortlog below.


Thanks,
Florent


AJP (1):
      Fix printing error in `report_exception(e)`

Aaron Maxwell (17):
      Branch for Aaron / --follow-symlinks
      Provide --bucket-prefix for run-tests.py
      Test case for following symlinks (simplest case) (currently fails)
      Implementation of --follow-symlinks option (nonrecursive case)
      Test for s3cmd put --follow-symlinks
      Negative test for s3cmd put --follow-symlinks
      Manual page documentation for --follow-symlinks
      More thorough and precise tests for --follow-symlinks
      Guard against common error when using --bucket-prefix
      correct error in test
      use Autotest-3, not autotest-3
      Compartmentalized symlink-safe directory walk, with all tests
passing on python2.5 and python2.6
      Fix leak in recursive-symlink-filtering algorithm
      document fswalk function
      name refactor to indicate _fswalk_follow_symlinks is an internal
helper function
      Update Changelog
      Option to follow local symlinks for sync and put
(--follow-symlinks option) including tests and documentation.  Also
addition to run-tests.py: --bucket-prefix option, to allow different
developers to run tests in their own sandbox

Abhishek Varshney (1):
      fix: handle empty md5 in find_md5_one

Albert Bendicho (4):
      Off-by-one on MAX_CHUNKS verification
      max_chunks now a config option
      MAX_CHUNKS constant removed, now a config option
      Added multipart_max_chunks

Alec Barrett (1):
      Update wiki URL in error message

Aleksandr Chazov (4):
      Allow setting SSL authentication
      Update man pages
      make names more understandable
      allow skip key file config option for embedded keys case

Alex Harford (1):
      Ensure testsuite-out is created before running dd

Alex Offshore (1):
      Return decoded response in get_policy and get_cors

Alexander Allakhverdiyev (1):
      Fix Python 3 bytes string encoding when getting IAM credentials

Alexander Todorov (2):
      Disable hostname checks for SSL connections on RHEL 7.2
      In case check_hostname isn't defined catch the error

Andrew Gaul (7):
      Sort s3cmd ls output by bucket name
      Capitalize command descriptions for consistency
      Horizontally align object policy info
      Emit user metadata in object info
      Support signed URL content disposition type
      Emit storage class in info subcommand
      Handle missing ETag in subcmd_bucket_list

Anton I. Sipos (1):
      Fix crash when cloudfront operations receive 500 errors. Retries
parameter was not being sent to send_request as a keyword arg.

Arnaud J (1):
      Migrate CI tests to use github actions

Arto Jantunen (2):
      Add a close method to the ConnMan api
      Add option to disable connection pooling

Arturas Moskvinas (1):
      Do not close stdout, it might be used when file is downloaded
and destination is "-"

Billy Crook (2):
      Added support for listing StorageClass attribute. Disabled by
default.  Enable with list_storageclass = True in .s3cfg
      Adding -l option for long_listing list_storageclass now provided
by long_listing and -l. Minor whitespace cleanup.

Bobby Impollonia (1):
      Add missing import of errno in S3.py

Brady (2):
      Install instructions for PIP
      Correct typo in instructions

Brandon Galbraith (1):
      Update s3cmd.1

Brendan O'Connor (1):
      Bugfix: use extra headers -- allows Requester Pays for buckets
you don't own. Patch from http://arxiv.org/help/bulk_data_s3

Brian C. Lane (1):
      Fix handling of config tri-state bool values (like acl_public)

Børge Nerdal Kjelsrud (1):
      Add option to ignore error on copy when key no longer exists

Carlo Cabanilla (1):
      fix bug in datetime tuple concatenation

Castedo Ellerman (1):
      content_encoding != charset, bug #153

Charlie Schluting (2):
      s3cmd du can gobble gigs of RAM on a bucket with millions of
objects. Re-worked 'du' to traverse the structure and store only the
sum at each level, allowing python to free memory at each folder. Went
from 4GB consumption (and being killed) on a test bucket with ~50M
objects in thousands of directories, to a max of 80M usage.
      that's not necessary any more.

Chris Lamb (1):
      Support relative expiry times in signurl.

Christian Rodriguez (2):
      Fix #849 by adding requester pays header to HEAD requests
      Merge branch 'bug/849'

Christopher Noyes (3):
      allow adding encoding to content type in s3
      don't add charset if it's already there, make param a comma
delimited str, not multiple
      remove comments, add space after ;

Christopher Porter (1):
      Fixing bug 3091912 - KeyError when copying multiple keys

Cisco Guillaume (1):
      Merge pull request #1 from GuillaumeCisco/gcisco-fix-appdata-directory

Craig Ringer (1):
      Add support for creating signed S3 URLs with the new signurl command

Dale Lovelace (1):
      Allow --acl-grant on AWS groups e.g.

Damian Gerow (2):
      Support proxying SSL connections
      Only support proxied SSL with Python >= 2.7

Damian Martinez (1):
      Adding MultiPartCopy class and enable for object_copy when
remote file is greater than 5gb.

Dan Carroll (1):
      Fixed typo: paramters -> parameters

Daniel Gryniewicz (1):
      Add getlifecycle command

Daniel Harris (1):
      Fix potential divide by zero errors

Dave Cole (1):
      Apply include/exclude to destinations too, via
mdomsch/s3cmd@268e7c3bd37cf60920025d556449fc1b93a1c590 Refs
s3tools/s3cmd#109.

David Eckelkamp (1):
      Let Makefile determine version number from PkgInfo.py file

David Kohen (2):
      Add support for IAM roles and temp tokens
      IAM role typo fixes

David Liu (1):
      Fail silently on SIGPIPE

Devon Jones (3):
      added ability to set home based on environment variable
      added S3CMD_CONFIG documentation to the man page
      Merge remote-tracking branch 'upstream/master'

Dmitry Kamenka (1):
      Copy uri_params in bucket_list

Dongmao Zhang (2):
      add new option --limit-rate to throttle download/upload speed
      only gettimeofday when limitrate is required

Doug Crozier (4):
      Changed "Buckets list" test to use pbucket rather than hardcoded
names. This fixes issue #1066.
      Allow use of Docker to run test cases with multiple Python versions
      Fix for 'dictionary changed size during iteration' error on
Python 3 with --cache-file (Issue #945)
      Test cases added for --cache-file

EC2 Default User (1):
      .gitignore: add s3cmd.egg-info

Emmanuel (1):
      managed files that was not remote copy to upload them directly

EnlytonJ (1):
      Windows fix re / vs os.path.sep

Eric Connell (1):
      added the ability to upload files by reading from stdin

Eric Drechsel (1):
      Fix for ignored --default-mime-type option

Eric Mill (1):
      rewrite of README into markdown

Etienne Michon (1):
      Typo in manual

Eugene Brevdo (4):
      Added --continue-put and --upload-id, allowing skipping of files
or part files during upload when their size and md5sum match the file
being uploaded.
      Minor bugfix to --upload-id check
      Fixed a small bug in MultiPart upload
      Fixed a upload corruption bug in MultiPart continue.

Florent Viard (403):
      BugFix: Fix dateS3toPython to gracefully handle returned date
that are not in GMT to respect iso8601. Ex.: 2014-03-05T18:45:50+00:00
instead of 2014-03-05T18:45:50Z
      Fixes date functions to return timestamps based on UTC time
value. + so simplified formatDateTime().
      Minor changes in README to fix points 1 and 2 in my comment at
https://github.com/s3tools/s3cmd/pull/369.
      Cosmetic: removed one empty space in excess of end of run summary.
      Fix MD5 comparison with s3 compatible servers that are not Amazon S3.
      Small cleanup of code, using "in" instead of "find" for strings
when possible.
      Fix an error in my previous commit:
b5b61abc8eb159f5932b936c7ca47e618615f6a3. (.find() forgotten to be
removed)
      Improve the exit code of s3cmd in the case of an error from S3
to give a clearer status of the reason of the failure. Some existing
exit codes were modified when they were not already in use.
      Disable dns support for buckets when "host_bucket" hostname
doesn't have a %(bucket)s item in it.
      Backported changes from the following pull request:
https://github.com/s3tools/s3cmd/pull/369 regarding aws pricing.
      Updated README.md with latest changes of S3 prices:
https://aws.amazon.com/fr/blogs/aws/aws-data-transfer-price-reduction/?sc_ichannel=EM&sc_icountry=global&sc_icampaigntype=launch&sc_icampaign=EM_128506540&sc_idetail=em_1778261631&ref_=pe_411040_128506540_12
      Improvement of error reporting in the case of an issue when
importing S3 sub-files. Now correctly cover cases like syntax errors
in these files.
      Fixes #423 (https://github.com/s3tools/s3cmd/issues/423).
Autodetected locale default should be set in default config for only
being used when this field is not overriden, neither by command line
or by a config file.
      Fixes #448 : Fixes S3UriS3 regexp to also recognize "s3://" as a
valid S3UriS3 entity with no bucket and no object.
      Renaming some functions for the code to be more understandable.
      Fixes #424 : In python 2.7, parsing invalid xml now raise a
ParserError(that didn't exist in python <2.7) instead of a ExpatError
+ cleanup in Utils.py (see issue #424 for more details)
      Improved handling for errors 501 and 504: Error 501: It means
that a feature is not implemented in the server, so no need for
useless retries. Error 504: Gateway timeout situation, exit code
should indicate a server error.
      Cleanup: Add prologue to files to state that they are UTF-8 encoded.
      Fix run-test script to take into account the config file that is
specified on the command line.
      Fixes encoding/decoding issues and so support for all strange
characters (chinese, japanese, ...) in file names, folder or path
arguments (Should fix bug like #417i) when UTF-8 is not the system
def$ Now, in theory, everything that gets in should be converted to
unicoded and then things would be converted back to string (user
defined encoding) at the outmost parts when dealing with the system.
When talking to S3, convert unicode to strings in utf-8 encoding.
      Remove useless error import
      Pass2 of "fixes encoding/decoding issues": os.walk replaced to
handle unicodise conversion + fixed the removal of most of useless or
invalid unicodise/decodise in the code.
      Fix for issue #432. Read the response from httplib even if there
is a 0 byte to read. So httplib doesn't think that there is a response
waiting to be read when the next request come. Straightforward to fix
thanks to the detailed analysis of Edwintorok.
      Use deunicodise consistentely in object_copy and object_modify
      fix encoding : Added encode_to_s3 and decode_from_s3 function +
switched to "unicode" call instead of "decode" as it is supposed to be
more efficient.
      Fix wrong exception used for EOF from S3 error during recv_file.
(S3Error expect an http response, not a text message)
      For copy and modify, take care to check if there is an error
after the start of the operation (reported by a status 200 but an
"Error" root entry in the xml body) In addition, for the move, It is
good to check for the CopyObject xml reply to see if it is a success.
But, if there is a status of 200 and no "data" in response, don't
fail. Other S3 server implementations use only status code and don't
return any body when there is no error.
      Another fix related to fix encoding issues. (file.name /
stream.name are bytes, so unicodise them also)
      Fix invalid file transfer progress info in case of local2remote
and remote2remote: If there was failed copy, seq number was preserved
but previously uploaded file number was forgotten, thus user could
have faced an invalid indication like: 14/8 when there is 6 upload
following 6 remote copy failures.
      Add a link to the known issues page (on github) within the
header and footer of s3cmd generated error reports. The goal is to
reduce the number of duplicated "bug reports" for known issues.
      Fixes #528 (https://github.com/s3tools/s3cmd/issues/528).
There is a lot of different "python-magic" libraries/versions, all
doing different things, so all cases need to be covered. file-5.11 has
be switched with "filemagic" as it should be more commonly used.
Introspection in each python magic lib has been done to ensure to
handle encoding or not only when it is needed.
      Fixes issue #576 (https://github.com/s3tools/s3cmd/issues/576).
EX_SOMEFAILED was still used in a PR that was merged instead of
EX_PARTIAL.
      Merge pull request #572 from fviard/fix_encoding_magic
      Merge pull request #574 from koenrh/https-by-default
      Don't slurp whole file while computing SHA256 checksum. Rework
of the PR of petr-tichy. When using large --multipart-chunk-size-mb or
--disable-multipart, the whole file was read into memory.
      Merge pull request #580 from Tim-Anderson/error_stop
      Merge pull request #583 from Tim-Anderson/issue577
      Merge pull request #586 from Tim-Anderson/remove_ignore_failed_copy
      Merge pull request #587 from taz77/master
      Merge pull request #589 from jcwilson/master
      Merge pull request #560 from Flipboard/iter
      Merge pull request #593 from jrschumacher/set-cors
      Merge pull request #603 from mdomsch/bug/testsuite
      Following #603 : WARNING OUTPUT FORMAT CHANGE. Changed the
format of output strings to be more consistant. Warning, the format of
output for a lot of actions was changed. Thus, any automatic string
for parsing the output could have to be fixed when using s3cmd after
this checkin. Ex: "File hello/world.txt stored as
s3://mybucket/world.txt" will now be: "upload: 'hello/world.txt' ->
's3://mybucket/world.txt'
      Fixes #602 (and complete #603) Fix run-tests.py to have all
tests passing against HEAD
      If s3error message is none, don't print an useless ": None"
      Return NotFound error with message s3Error: NoSuchBucket when
there is no bucket found to delete
      Raise standard S3Error in S3.py instead of simply exiting. So
the normal error logic can't operate on any unexpected error. Avoid to
have empty errors in output like: "ERROR: S3 error:" (Because of an
400 InvalidBucketName error returned by a non AWS3 server)
      Merge pull request #590 from andrewgaul/content-disposition
      Fix stop_on_error config type
      Attempt to fix the first bug of issue #600 : When a subfolder
"xxx" was selected as source, the sync comparison was done against the
destination base whereas it had to be done based on the "xxx"
subfolder of the destination.
      Fixes run-tests following the fix for
https://github.com/s3tools/s3cmd/issues/600
      Merge pull request #616 from mdomsch/bug/612
      Fix bad help info for the "--no-check-certificate" switch.
Thanks to @psi29a for proposing the fix.
      Merge pull request #618 from mdomsch/bug/futurewarning
      Fixes #592 : Ignore any leading slash before bucket_name in an
S3Uri. Original issue: "Running s3cmd ls s3:///foo lists owned buckets
instead of emitting an error."
      Fix for error reporting in case of upload error. (Following the
fix for #577)
      Fixes #581 regarding the reporting of SSL verification issues.
Other problems of this issues should already be fixed.
      Merge pull request #621 from mdomsch/bug/ssl-checking
      Merge pull request #619 from fviard/second_fix_for_577
      Merge pull request #617 from fviard/fix_subfolder_source
      Merge pull request #623 from mdomsch/bug/testsuite2
      Reworked Upload and Download errors to be more consistent. Add
detection and handling of different OSErrors like "File name too
long."
      Merge pull request #628 from mdomsch/bug/curl
      Merge pull request #627 from fviard/upload_download_errors_rework
      Merge pull request #630 from mdomsch/feature/ia-class
      Merge pull request #611 from bsapiro/master
      Added --server-side-encryption-kms-id config option. Following PR #611.
      Fixes bug in PR #611
      Fixes #636. Also remove the temporary file when encountering an S3Error.
      Merge pull request #599 from shigezobriand/master
      Fix "info" command for servers not supporting the
"request_payer" feature. Based on PR #640 of Qiankunzheng.
      Fixes #644 : Minor fix to not output "u quoted" invalid
parameter in the case of an invalid command.
      Reworked source and destination input of "remote2local" to cover
gracefully more cases.
      Block "-" (stdin / stdout) to be used in parameters for sync as
it has no sense.
      Only warn for the non regular files that are skipped.
      In the case of local2remote, raise an error if a source arg is missing.
      Fix automatic tests for recent changes.
      Merge pull request #650 from fviard/fix-remote2local-input
      Don't fail the copy if a non AWS s3 server doesn't support setting ACLs.
      Have extra labels coherent when progress is enabled or not and
for multipart uploads.
      Added the --stats option that is similar to the one of rsync.
      Added "--stats" info for sync commands (remote2local,
remote2remote, local2remote)
      Merge pull request #652 from s3tools/bug/598
      Merge pull request #655 from fviard/rework_progress
      Fix duplicated code for the error alert message. Based on
@edunham PR: https://github.com/s3tools/s3cmd/pull/609
      Fixes #661 : Don't try to perform a delete if there is nothing
to delete. + Restore warning that was in 1.5.2.
      Fixes #659 : Don't fail if "content-type" field is missing in
info like for eucalyptus walrus.
      Fixes #646: number of "files to update" are not reported in the
summary (there was only the new files to be added)
      Fixes #646 : remote2remote sync -  "Updated" files were not
reported as to be remotely copied when in dry-run mode.
      Merge pull request #667 from kobake/skip-restore-already-in-progress
      When using --no-preserve, still add 'md5' custom attribute if
md5 check is enabled.
      Fixes #674 : When using AWS_CREDENTIAL_FILE, crash in case of
invalid key in .aws/credentials file + updated the code for actual
name of the keys.
      Merge pull request #711 from Shamanoid/patch-2
      Merge pull request #559 from jmif/master
      Merge pull request #701 from makery/master
      Don't ask to set access_token in run_configure as it should not
be used very often.
      Merge pull request #720 from harshavardhana/etag
      Merge pull request #684 from harshavardhana/try-fail
      Fix for local2remote in the case of "." as a source. Fixes #706
      Fixes #709 : Fixes sync command with windows + cleanup
      Fix the error message to avoid confusion in a special cases like
"no space left on device".
      Merge pull request #702 from artms/quiet
      Fixes #744 - Ensure to check non None or empty bucket names
before calling check_bucket_name.
      Merge pull request #755 from andrewgaul/list-missing-etag
      Merge pull request #714 from
alces-software/fix/generalize-wildcard-certificate-forgiveness
      Minor optimization and cosmetic fix in ConnMan.
      Fix Issue #672 - Patch based on PR #729 proposed by MLopez-Ibanez.
      Merge pull request #763 from dang/getlifecycle
      Fixes getBucketFromHostname for when host_bucket config has no
"bucket" element.
      Fix the infamous "UnboundLocalError" for conn exception hidding
meaningful exceptions
      Added an explicit error message when the servers hostname are
wrong or unresolvable.
      Avoid having an error when user uses "https://" prefix to host_base...
      Fixes #741 Issues validating ssl certs when using a custom port
for host_bucket address.
      Real fix for #772. Use urlparse to extract hostnames in connman
      Merge pull request #779 from cybojanek/master
      Follow up on #779, not using a CONNECT "tunnel" for http proxies
as it is not the most common case.
      Fixes #777 : Fix __repr__ returning unicode and sign_v2 not encoded to s3.
      Merge pull request #757 from mozawa/max-ls-keys
      Merge pull request #781 from kwarunek/master
      Fixes recursive delete that was broken by the new "truncated" tag.
      Fixes orphan var in "du" exception handler.
      run-tests.py Update expirations tests dates to the future.
      Merge pull request #784 from mozawa/configure-endpoint
      Merge pull request #785 from shaform/py3k
      Fix failure of copy or rename for s3-like servers that don't
have support for acl.
      Fix error of "info" command when s3-like servers don't implement
"acl" or another feature.
      Merge pull request #789 from shaform/py3k
      Merge pull request #787 from Abhishekvrshny/fix-empty-md5
      Avoid errors with modify command when s3-like servers don't support acl.
      Fix possible bug in previous change for info command and get_lifecycle.
      Fixes #722 : basename fonction was broken for local files for
windows as the os path separator is not the same on linux.
      Aws sometimes uses 301 code for redirects + redirect to aws base host.
      Fix signature v4 when aws redirect to aws base host.
      Fixes #778 : Workaround for aws sending early errors before
requests are completed + support for http expect continue feature.
      Updated maintainer in setup.py.
      Fixes #797 : unicodisation missing for exclude patterns coming from files.
      Fixes #800 (second part): "cfg.include" was not checked like is
cfg.exclude to use the non memory optimal del subcmd.
      Fix for part of #660: fix detection for "sync single-file.txt
s3://mybucket/single-file.txt"
      Fix to supporting python 1.6. Use "List" for http methods and
not a kind of dict.
      Backport more code from httplib for compatibility with python 2.6
      Fix previous patch on custom httpconnection, it is _send_output
and not send_output.
      Attemps to fix last issues of #769 .
      Initial version of a .travis.yml and it's .travis.s3cfg
configuration for CI with travis-ci.
      custom_httplib27 - Missing symbols in python 2.6
      Missing bit in previous commit.
      [travis] Fixes allow_failures not being strings.
      [travis] Added run-test-minio.py + have travis run it.
      Added travis-ci build status of master in the github displayed readme.
      Merge pull request #806 from shaform/py3k
      Merge pull request #808 from robd003/master
      Merge pull request #809 from shaform/py3k
      Fixes #810 : Also handle redirects for path styles syntax.
      Fixes #816 : Fix recv_file 3xx,4xx, handlers + add region
extraction of 3xx redirections.
      Fixes #798 : HEAD request can't parse xml response to get redirection url.
      Fix missing imports in custom_httplib27.
      Move urlencode func into Utils and fix encoding   issues when
having uid groupid with special chars.
      Small cleanup in S3.py
      Py3 support: remove usage of deprecated has_key.
      Py3 support: fix SortedDict (keys() are now views and iterators
need __next__)
      Update README to indicate python versions supported by s3cmd.
      Changed unicodise/deunicodise wrappers to fit with python3 string model.
      Python3 support: Fixed stderrr, stdout unicode output.
      Remove some code that was there for supporting py version < 2.6
      Use the stdout/stderr buffer directly instead of detaching.
      Python3 support: fix error reporting at import time and unicode.
      Python3 support WIP: Added Custom_httplib3x to patch python3
version of httplib as it is done for py2.
      [Py3 WIP] Fix for compatibility.
      PY3 support: fixed S3Uri and Exceptions regarding unicode/str
compatibility.
      Missing change that was needed for custom_httplib functions...
      PY3 support: more unicode related things.
      PY3 support: long was removed in py3. Replace by int
      PY3 support: ensure to use the adequate / sign for division with python3.
      PY3 support: Added a warning at runtime to warn of WIP
      PY3 support: Added silent versions for unicodise/deunicodise funcs
      Small cosmetic changes in Custom_httplib(s)
      Remove legacy code. s3cmd is now only compatible py >2.6
      Py3 support: Reworked request/signature regarding unicode/bytes mix.
      Py3 support: signature v2 - limit list of subresources url that are signed
      Py3 support: ensure that request body are encoded when "unicode" strings
      Signature v2: add "cors" to list of subresources to include in signature
      Improve get_bucket_location to be able to work all the time with aws
      Use pretty print everywhere in debug for request responses
      Getlocation for each type of request when region is not set and
signature v4 used
      PY3 support: more fixes for str/unicode related issues.
      Typo fix in debug log message
      Merge pull request #837 from alexjh/allow-skip-tests
      Merge pull request #851 from burstworks/master
      PY3 support: use getitem instead of getslices for slices
      PY3 support: Fix for regexp/unicode in runtest-minio
      Use io buffers everywhere for files or stdin/stdout + Use custom
stream_name attribute.
      Fixes signurl (v2) for python3.
      run-tests-minio: re-activated a test that is now working.
      Urlencode string only when formatting the url and not earlier.
      Get ride of the "extra" parameter for requests and use a clean
"params" dict for all the query strings.
      Ensure to not have "{}" in functions parameters to be safe.
      In the v4 canonical query string, parameter without values
should still have an equal sign.
      Fixes signature v2 invalid call to format_param_str.
      PY3 support: Fixes acl.py use of repr.
      Remove bad duplicated code.
      Little performance + memory improvement. + Fixes randomless in auto tests.
      Typo fixes.
      Sort headers in requests to always have the same order.
      Fixes failing tests with special chars for PY3
      Small changes on run-tests.py
      Change .travis.yml now all tests are passing with minio.
      Merge pull request #870 from yossi-stratoscale/issue-869
      Merge pull request #858 from chauffer/patch-1
      Documentation: Fixes #872 . Updated following the proposal of @cthorley.
      Merge pull request #857 from guygurari/cp-mv-bug
      Fix error message to be accurate with all cases + Fixed auto
tests for last change.
      Update version to 2.0.0
      Update setup.py classifiers to indicate that Python3 is supported.
      Changed version to 2.0.0+ for github nightly versions
      Fixes #887 - Invalid code inside cf __str__ breaking cf invalidation.
      Fixes #889 - info output cosmetic fix.
      Fixes #885 - Specific error message when getting policy by non owner.
      Small cleanup of legacy checks in Config.py
      Fixes #888 - Fixes cfinvalinfo broken in 2.0.0
      Fixes #886 - Don't force to have a / on last parameter for
"modify" command.
      Fixes #890 - Missing json import after cleanup patch.
      Fixes #892 - Ensure to add "LICENSE" file to release tarballs.
      Fixes #894 - Detect and report error 403 in getpolicy for info command
      Fixes #895 - Refactored py3 source to be byte-compilable with
py2 to fix warnings during "setup.py install"
      Remove python3 support warning.
      [PY3 support] Fixes --dump-config broken because of "verbosity"
      Cosmetic - use "with" instead of an open/close
      Fix encoding issues with config files + PY3 support for --configure
      Use "unicode" for config attributes
      Small rework for pre existing hash cache file detection
      PY3 fix: Open/Save hash cache file in "binary" mode
      Fix for non string cmdline parameters being ignored
      Cleanup: Don't have anymore cfg as a global
      Fixes missing local cfg in copy/move
      Fixes last missing local cfg
      Fixes #905 - Fixes https with proxy when no port defined for server.
      Fixes #908 - Fixes unicode in req headers for py2 and py3
      Merge pull request #917 from thelan/master
      Fixes #921 - fixes gpg_command for python3
      Fixed #922 - getenv return unicode in py3 and bytes in py2
      Merge pull request #923 from harshavardhana/fix-etag
      Fix unicode issue in report in the case of an error
      Fixes #920 Always treat hostnames in lower case. Signature issues.
      Merge pull request #925 from jmaitrehenry/marker
      Fix #922 - Handle libmagic missing on windows
      Fixes getBucketFromHostname not using "lowercase" host_bucket.
      Avoid crashing python when encountering a CertificateError
      Update version to 2.0.1
      Fixes sourceforge mailing list addr in RELEASE_INSTRUCTIONS file.
      Update version to 2.0.1+ for MASTER dev.
      Fixes #929 - strip namespace well even without xml declaration
      Fixes #933 Fix dst copy when the destination file was modified
      Merge pull request #938 from Nossnevs/feature/fix_nr_of_paths
      Fixes #525 - Fixed no invalidate of remotely copied files during sync
      Fixes error reporting for import error and utf8 characters as
cmdline arguments
      Added clarification comment for the socket_timeout configuration
value OS limit
      Partial fix for #941 - Retry for all kind of requests in case of
OS network timeout events
      Fixes #941 - Restore retries for some 400 errors like RequestTimeout
      Partial fix for #913 - Only throttle retries for s3 slowdown errors
      Fixes #913 - Fix limitrate to recover of sudden changes in
bandwidth and to not slow down too much the transfers
      Merge pull request #959 from mouchar/2.0.1
      Little help clarification for --delete-after. Fixes #958
      Merge pull request #947 from alexallah/master
      Addition to #947 - ensure it works with py2. aws requests always
returns utf-8 encoded data
      Merge pull request #940 from bcl/master-acl_public
      Fixes bug created by #940 - cmdline set values are real booleans
      Complete #940 - upload_id, content_disposition and content_type
default value is now "" and not anymore None
      Merge pull request #968 from LocutusOfBorg/avoid-distutils
      Merge pull request #970 from radeksimko/b-exclude-connection-header
      Merge pull request #975 from cybojanek/add_restore_subresource
      Merge pull request #966 from Trane9991/master
      Merge pull request #973 from mcv21/nulldisplay
      Fixes #960 - Do a retry if exc has errno but errno is None.
Python3 support.
      Fixes auto-test following recent aws changes regarding bucket
names. See: #957
      Fixes #967 - In addition with patch for #960, ensure that error
logging with not crash
      Update version to 2.0.2
      Fixes #987 - Fixes using IAM with python3. Config options should
be unicode.
      Fixes #987 - Before py36, py3 json was only able to load unicode str
      Merge pull request #986 from vpresnyak/output_alignment_fix
      Complete the fix #986 - To satisfy py2 and p3, there should be
float before and after the division
      Issue #956 - Fixes size reporting using k instead of K as it a
multiple of 1024
      Fixes #956 - fix size and alignement of du and ls output reporting
      Merge changes of release 2.0.2
      Merge pull request #994 from harshavardhana/no-need
      Merge pull request #1007 from hrchu/patch-1
      Merge pull request #1010 from starrify/fix-minor-help-message-typo
      Convert anything given to getTreeFromXml to utf8 bytes if needed
      Fixes #1006 - fixes multiple python3 encoding issues with
cloudfront functions
      Merge pull request #1021 from offshore/patch-1
      Merge pull request #1033 from junousi/public_url_https
      Fixes #1006 - missing return in previous commit
222af5d8ce90bbb2f7784360c019f5a6af2ef8bd
      Merge pull request #1042 from serprex/master
      Merge pull request #1061 from k0ste/help
      Merge pull request #1063 from timgates42/bugfix/typo_unnecessarily
      Change version to development 2.0.2+
      setup.cfg: set universal bdist_wheel flag
      setup.py: update meta files with support for recent python versions
      Merge pull request #1011 from zmedico/signurl_use_dns_based_on_bucket_host
      travis-ci: Disable tests for python 2.6 as not supported anymore
by travis-ci default ubuntu distribution
      Merge pull request #1071 from nitrocode/patch-1
      Complete the previous change to rename INSTALL file to INSTALL.md
      Merge pull request #1067 from NotTheDr01ds/bug/1066
      Merge pull request #1068 from NotTheDr01ds/feature/run-tests-in-docker
      run-tests.py: Don't fail ACL tests if public_url_use_https is used
      svnignore: update to match latest gitignore content
      Little cleanup. TODO is out of date and obsolete
      Add a .dockerignore because of this stupid docker that does not
know how to exclude files he has no business with
      Update run-tests.docker.README.md with a small display fix
      Create FUNDING.yml
      Factorize the region detection inner request and stop retrying
requests that fail the ssl certificate checks
      Fixes #1059 - Hack for SignatureDoesNotMatch error when host
port 80 or 443 is specified, due to stupid servers
      Merge pull request #1014 from Quiq/master
      Merge pull request #1032 from
vainu-arto/make-connection-pooling-configurable
      run-test-*: Minor fixes
      Small cleanup
      Fixes #377 - Avoid crashing when a file disappeared during cache
generation
      Fixes #1080 - url encode bucket_name for path-style requests.
Fixes an issue with CephRGW and the tenant feature.
      Fixes #1081 - Don't force signature v4 if bucket_name uses special chars
      Fixes #1082 - Allow to use signature v4 only, even for a command
with no target bucket specified
      Fixes #1054 - Relax limitation on special chars for --add-header key names
      Fixes #1020 - Update storage-class cmd help msg with new aws s3 classes
      Fixes open file descriptors leaking by not being closed on time
      Added a comment about how to escape the comment char in exclude files
      Fixes #989 - in aws_credential_file, fixes read_string only
available with py3 + refactor for possible unicode issues
      Fixes #847 - set a few str as unicode to fix UnicodeDecodeError
with python2 when doing a sync remote 2 remote
      Merge pull request #1084 from prabhakar267/master
      Fixes py headers to use system default python and not force
python2 anymore
      setup.py: Update authors and cleanup
      Update version to 2.1.0
      Change version to development 2.1.0+
      Fixes #1087 - fixes ValueError when using more than one ":"
inside add_header in config file
      Fixes #1089 - Ensure that KeyboardInterrupt are always properly raised
      Update RELEASE_INSTRUCTIONS with the new procedure to upload to Pypi
      Rework and cleanup of MultiPart and MultiPartCopy (following the
merge of PR #74)
      Add progress info output for multipart copy and current-total
info in output for cp, mv and modify
      Move s3_quote to Utils module
      Multipart copy - lower multipart copy chunk size to 1 GiB
      Move cmd - take care in that copy s3 xml result can be a
multipart copy result
      Url encode source for copy/mv/modify to support all
special/foreign character names in object names
      Cosmetic cleanup
      MultipartCopy avoid retrieving again object info when we already
know its size
      Ensure to catch and display server errors for cmd cp,mv,modify errors
      Cleanup - no message was set in 2 cases for S3UploadError
      MultiPart - Fixes extra label issue with stdin as source
      MultiPartCopy - Also copy acl like direct copy
      MultipartCopy - Preserve metadata of the object that is copied
      Fixes #1091 - Ugly hack to have modify work with files bigger than 5Gb
      Allows to set mime-type (ie content-type) also for remote copies also
      Allows to set public ACL with "modify" to match "copy"
      Remove legacy code for dreamhost. I guess that they are now supporting acl
      Small cleanup of _build_attr_header
      Fixes typo/failure in multipart cp when progress meter is off
      Refactored object_copy to avoid duplicated code in object_modify
      Fixes #685 - Grammatical correction a debug message
      Metadata for copy should  now be consistent for multipart or not
and for aws or not cases
      Fix remote_copy to ensure that meta-s3cmd-attrs will be set
based on the real source and not on the copy source
      Fixes _build_attr_header for case of wrong old value being use
for wrong attribute
      Simplify the code to use _build_attr_header in remote_copy
      Fixes #1088 - Handle servers indicating (lying) that listObject
result is truncated, when it is not the case
      Fixes #1090 - Be more resilient to a server sending random data
in ?lifecycle response
      Fixes #1093 - Fixes error encountered with Minio send empty
content instead of a valid ?requestPayment response.
      Merge pull request #1051 from chazer/ssl-auth
      Fixes #1094 - Fixes typo bug + mini cleanup
      Fixes #1096 - Add http error 429 to the retryable errors
      Fixes #1101 - "files-from" option was resulting in an
AttributeError with Python3
      Cleanup of a few lines ending with a ";"
      Cleanup - Remove unreachable code and an useless test
      Fixes #1113 - Fixes report_exception wrong error when LANG env
var was not set
      Fixes #346 - Add support for listing and resume of multipart
uploads of more than 1000 parts
      Utils/parseNodes: cleanup - avoid returning empty dicts inside
the result list
      Ref #1114 - Add a time based expiration for idle pool
connections in order to avoid random broken pipe errors
      Fixes #1114 - Reduce connection_max_age to 5s as it looks like
that aws closes https conns after around 6s
      Merge pull request #1118 from Patouche/issues/1117
      Merge pull request #1119 from Patouche/fix/datetime-tz
      Merge pull request #1120 from anbnyc/patch-1
      Merge pull request #1124 from tirkarthi/fix-warnings
      Merge pull request #1137 from ondrejbudai/python39-fixes
      Merge pull request #1141 from onagurna/master
      Merge pull request #1145 from xrmx/cleanup_check_md5
      travis.yml : Add python 3.9 to the ci test matrix
      Merge pull request #1144 from k3ntako/master
      Change import to have the py3 import first
      Merge pull request #1160 from siakaramalegos/patch-1
      Merge pull request #1165 from FFKL/master
      Merge pull request #1163 from Rjerk/wip-fix-empty-lc-prefix
      Merge pull request #1152 from orztt/wip-private-acl
      Change dev version to 2.1.0-dev to respect PEP 440 that is now
enforced by pip
      Fixes #1125 - avoid infinite loops when files are truncated
during hashsum or upload
      Config: Get token expiration from AssumeRoleWithWebIdentityResult response
      Minor fixes in s3cmd
      Create a new BaseUtils module for helper functions that don't
need access to Config
      Ensure that "https" url namespaces are also stripped in xml
      Complete work for AssumeRole And AssumeRoleWithWebIdentity support
      Use explicit raw strings for regexp patterns in Config
      Cleanup: Factor unicodise/deunicodise to be reused inside Config
      Fixes #1190 - Set properly PARTIAL exit code for object_get when needed
      Merge pull request #1199 from arnaudj/dev/migrateTravisToGitHubActions
      CI: Add python 3.10 to test matrix
      Merge pull request #1193 from rvanlaar/patch-1
      Merge pull request #1204 from akete/issue-1203
      Merge pull request #1198 from sebvajda/master
      Fixes #1201 - Raise an explicit error when only env var
AWS_ACCESS_KEY is set
      Update version to 2.2.0

Francois Gaudin (2):
      Fix custom header support for cp
      Modify command equivalent to a self copy to modify metadata

Francois Lebel (1):
      Fix the example for recursive listing.

François Wirion (1):
      Fix missing slash in directory semantics documentation

George Hickman (2):
      Don't include python files in the manifest.
      Add a makefile for releasing

George Melika (1):
      Added a --disable-content-encoding to disable adding the
content-encoding header to the pushed objects

Gianfranco Costamagna (2):
      Fix typo (overriden -> overridden)
      Add fallback code to make it compatible with Python2
implementation (we could simplify by adding whichcraft, but a new
dependency just for one which command seems an overkill)

Gianni Chiappetta (1):
      Always attempt to set the content-type

Giorgos Logiotatidis (1):
      If response['headers'] is empty, then don't update dict.

GuillaumeCisco (3):
      Fix app data directory
      Add .idea directory to gitignore
      Add fallback for old windows versions

Guy Gur-Ari (4):
      run-tests.py: use lowercase bucket name
      don't allow mv/cp of multiple files to single file
      raise exception for missing '/' during mv, cp
      remove unreachable code

Harshavardhana (5):
      s3cmd: Fix s3 path style signature v4 handling.
      Missing etag in HTTP headers, handle gracefully.
      build: Do not throw exceptions if no arguments were passed for setup.py
      Don't need to compute ETag md5sum match if it is not a real md5sum
      Do not convert all regions to lower case

Haruka Iwao (1):
      * S3/Config.py   Get user input aws access key, secret and gpg
passphrase out of debug format string

James Brown (1):
      Respect the $TZ environment variable to show local times if requested

Jan Kasiak (2):
      Change proxy SSL URI to relative
      Add restore subresource to V2 signature

Jason Dalton (8):
      Replaced mktemp with alternative temp file generation based on
secure randoms in order to avoid owner-restricted permission issues
with mktemp files
      Added cleanup of temp file if error or keyboard interrupt occurs
during transfer.
      cleanup
      Revert "cleanup"
      Revert "Replaced mktemp with alternative temp file generation
based on secure randoms in order to avoid owner-restricted permission
issues with mktemp files"
      Set permission on copied file based on umask.
      Updated references to temp file in exception handlers
      Relaxed timeout for http connection during IAM role lookup

Jason Godfrey (1):
      Quick fix for cfcreate and cf modify not working. These commands
would generate a KeyError for 'Origin'. (Bug 3536626)

Jens Braeuer (2):
      Added support for "Static Websites"
      Remove recursion detection for symlinks.

Jerome Leclanche (11):
      Avoid catching SystemExit in a try block
      Add __repr__ for S3Uri class
      Add S3.MultiPart module with basic functionality
      Make use of S3.MultiPart functionality by default
      Rename MultiPartUpload.bucket into MultiPartUpload.s3
      Implement multipart threading
      Use the ThreadPool interface to thread multipart uploads and
return a proper response
      Dynamically increase the chunk size depending on the file size
      Add the --enable-multipart option to s3cmd
      Add multipart_num_threads and multipart_chunk_size to Config
      Properly pass multipart enabling, default enabled on > 100MB

Joe Fiorini (1):
      Added command to set bucket access policy

Joe Mifsud (2):
      Add support for invalidating multiple CF distributions with
--cf-inval when the bucket is the origin for multiple distributions
      Switching to set()

Jose Fernandez (magec) (2):
      Make use of TMP variable to change tempdir
      Document Environment variables

Josep del Rio (4):
      When using default index, invalidate path
      Corrected the new flags for CF index invalidation
      Removed debug messages, make sure default index is defined
      Fixed exception for root document

Josh C. Wilson (1):
      move progress_meter default setting down to Config class

Jukka Nousiainen (2):
      Set public_url() protocol based on use_https
      Parameter for public URL protocol

Julien Maitrehenry (1):
      Change the marker value for NextMarker in bucket_list_streaming

Julien Recurt (1):
      Adding option signurl_use_https for signurl to use https prefix

Ka-Hing Cheung (1):
      enlarge write and read buffers

Karl Matthias (1):
      Add support for putting arbitrary additional headers in s3cfg file.

Karsten Sperling (15):
      Use python-magic for guessing MIME types if available
      Catch the right exception, doh.
      Added example /etc/magic file with better support for web file types
      Add support for setting Content-Encoding header where it can be
guessed, in particular Content-Encoding: gzip is handled.
      Flush stdout after logging messages, in case buffered pipes are involved
      Merge branch 'master' of git://github.com/s3tools/s3cmd
      Added example /etc/magic file with better support for web file types
      Add support for setting Content-Encoding header where it can be
guessed, in particular Content-Encoding: gzip is handled.
      Flush stdout after logging messages, in case buffered pipes are involved
      Fix up code that was merged incorrectly
      Merge branch 'mime-guessing'
      Add -M short option for --guess-mime-type
      Fix ValueError when magic module is not present
      Add support for http://pypi.python.org/pypi/filemagic version of
python-magic
      Merge branch 'mime-guessing'

Karthikeyan Singaravelan (1):
      Fix deprecation warnings due to invalid escape sequences.

Kelly McLaughlin (2):
      Handle empty return bodies when processing S3 errors.
      Reuse a single connection when doing a bucket list

Kentaro (2):
      Update base64.encodestring for Python 3.9 compatibility
encodestring() method was removed from base64 in Python 3.9.
      Update base64 import to support Python 2 and 3

Kevin Daub (5):
      Fixed typo in output messages.
      Added --server-side-encryption as an option and adds the header
to put and put-copy.
      Reverted out manpage update
      Added Server-Side Encryption (SSE) to output of --info
      Fixed formatting

Kiana McNellis (2):
      Fixed bug where the error report will not show if there was an
import error due to dependency on unicodise which was not successfully
imported
      Changed the exit code for import errors to not rely on
dependency of EX_GENERAL being defined when not imported

Koen Rouwhorst (1):
      Use HTTPS by default

Konstantin Shalygin (1):
      s3cmd: python 3.8 compat

Krzysztof Warunek (1):
      support endpoint with custom path

Lance Batson (1):
      fix issue with error display after config modify

Leandro Ostera (1):
      Update s3cmd --configure

Liu Lan (1):
      fix get bucket info error when LC use the filter element

Luke Winslow (1):
      Fixed issue with mixed path separators with s3cmd get
--recursive on Windows.

Marco Jakob (7):
      Get the access token from the environment
      Do not refresh role
      Do not refresh role
      Add option for token-refresh
      Add option for token-refresh
      Add option for token-refresh
      Add option for token-refresh

Mark Titorenko (2):
      Generalize wildcard certificate forgiveness
      Updated based on feedback.

Masashi Ozawa (2):
      add "limit" option to "ls" and "la" commands to return the
specified number of objects instead of returning all objects in the
bucket.
      Enable to configure "host_base" as "S3 Endpoint" in interactive
configuration.

Matt Domsch (485):
      Apply excludes/includes at local os.walk() time
      add --delete-after option for sync
      add more --delete-after to sync variations
      Merge remote-tracking branch 'origin/master' into merge
      Merge branch 'delete-after' into merge
      add Config.delete_after
      Merge branch 'delete-after' into merge
      fix os.walk() exclusions for new upstream code
      Merge branch 'master' into merge
      Merge branch 'master' of git://github.com/s3tools/s3cmd
      add --delay-updates option
      finish merge
      Handle hardlinks and duplicate files
      hardlink/copy fix
      remote_copy() doesn't need to know dst_list anymore
      handle remote->local transfers with local hardlink/copy if possible
      sync: add --add-destination, parallelize uploads to multiple destinations
      add local tree MD5 caching
      fix getting uid
      Apply excludes/includes at local os.walk() time
      add --delete-after option for sync
      add more --delete-after to sync variations
      add Config.delete_after
      fix os.walk() exclusions for new upstream code
      add --delay-updates option
      Handle hardlinks and duplicate files
      hardlink/copy fix
      remote_copy() doesn't need to know dst_list anymore
      handle remote->local transfers with local hardlink/copy if possible
      sync: add --add-destination, parallelize uploads to multiple destinations
      add local tree MD5 caching
      fix getting uid
      HashCache: add missing break during purge
      merge manpage conflict]
      Merge branch 'cache-local-md5' into hardlink-fixes
      sync: refactor parent/child and single process code
      Merge branch 'econnell/master' into econnell-merge
      merge relistan/master
      Merge remote-tracking branch 'brendano/master' into brendano-merge
      Merge remote-tracking branch 'ksperling/master' into ksperling-merge
      Merge remote-tracking branch 'michaeltyson/master' into michaeltyson-merge
      Merge remote-tracking branch 'joseprio/master' into joseprio-merge
      Merge remote-tracking branch 'ghickman/packaging' into ghickman-merge
      Merge remote-tracking branch 'mattswe/patch-1' into mattswe-merge
      Merge remote-tracking branch 'joefiorini/access-policy' into
joefiorini-merge
      Merge remote-tracking branch 'Roguelazer/tz_support' into Roguelazer-merge
      Merge remote-tracking branch 'godfreja/fix_cf_keyerror' into
godfreja-merge
      Merge remote-tracking branch 'manos/master' into manos-merge
      Merge remote-tracking branch 'ghickman/manifest-cleanup' into merge
      fix mime detection for Fedora python-mime
      fix multipart uploads
      fix --cf-invalidate after merge
      cannot use both --cf-invalidate and --add-destination
      note mdomsch merge branch in version string
      Merge remote-tracking branch 'ejnoyes/master' into test-merge
      Merge remote-tracking branch 'spikegronim/fix-hang-bug' into merge
      Merge remote-tracking branch 'tycho/master' into merge
      finish merge
      fix up failed merge of --delete-after-fetch
      update mdomsch-merge version string
      fixup trivial merge conflict
      remove unused and incorrect function hash_file()
      wrap get_md5() with try/except IOError
      remove debug output that mixes unicode and ascii strings poorly
      be verbose when both hardlinking and copying of local files fails
      update_acl(): fix output message
      Handle local hardlink and copy failures by downloading the objects
      add output on remote copy, fixes test cases 27 and 28
      add --files-from=FILE to allow transfer of select files only
      accept --files-from=- to read from stdin
      add --files-from to manpage
      use --files-from only on local source (not remote, not dest)
      fix sync --dry-run traceback, bug #124
      fix single file sync local2remote traceback, bug #128
      sync remote2remote: set target_uri on updated files too, bug 131
      Split md5/hardlink stuff out of SortedDict into FileDict
      fix sync single remote file to existing local file traceback, bug 88
      add --max-delete for del and sync commands, fixes bug #155
      sync: ignore --delete-removed if no source files, bug #155
      provide a better exception / error reporting message, bug #158
      Merge branch 'files-from' into merge, pull request #116
      Merge remote-tracking branch 'borgenk/master' into merge, PR #143
      Merge remote-tracking branch 'yar/master' into merge, PR #147
      Merge remote-tracking branch 'aisipos/master' into merge, PR #151
      Merge remote-tracking branch 'developmentseed/master' into merge, PR #152
      Merge branch 'bug/155' into merge, PR #156
      Merge branch 'bug/153' into merge, PR #157
      Merge remote-tracking branch 'abuchanan-grio/master' into merge, PR #162
      Merge remote-tracking branch 'tjasond/master' into merge, PR #163
      add --xattr option referring to file system extended attribute
containing md5sums
      Merge branch 'xattr' into merge
      fix conditional import xattr module
      Merge branch 'xattr' into merge
      push md5_xattr to Config only if xattr module was imported
      Merge branch 'xattr' into merge
      Don't emit "cowardly" message unless we have source files & delete files
      Merge branch 'bug/155' into merge
      Get md5 from remote attribute header in recursive syncs too
      Merge branch 'bug/remote-md5sum' into merge
      FileLists: get md5 from headers
      Merge branch 'bug/remote-md5sum' into merge
      move x-amz-meta-s3cmd-attrs handling into S3.py
      use metadata-stored md5 in bucket list, put, and info
      Merge branch 'bug/remote-md5sum' into merge
      look at metadata md5 for get as well
      Merge branch 'bug/remote-md5sum' into merge
      s3cmd get to stdout: set file_exists to avoid traceback, bug #175
      don't purge the HashCache if using --files-from
      HashCache: catch KeyError in unmark_for_purge(), bug #194
      HashCache and FileDict: ignore files with device = 0 or inode = 0
      Merge branch 'files-from' into merge
      Merge branch 'bug/194' into merge
      fixup merge
      Merge remote-tracking branch 'ivebeenlinuxed/master' into merge
      Merge remote-tracking branch 'reed-sandberg/master' into merge
      Merge remote-tracking branch 'kuenishi/more_multipart' into merge
      Merge remote-tracking branch 'scharissis/fix_help_typo' into merge
      finish merge of #186
      Merge remote-tracking branch 'orenhe/simplify_wildcard_handling'
into merge
      Merge remote-tracking branch 'robysath/master' into merge
      Merge remote-tracking branch 'spartman/remote-copy-fix' into merge
      remote_copy(): fix value in failed_copy_list
      Merge branch 'no-mime-magic-option' of
git://github.com/radomirml/s3cmd into merge
      Merge branch 'master' of git://github.com/brandongalbraith/s3cmd
into merge
      Merge branch 'listallmybuckets-error' of
git://github.com/leostera/s3cmd into merge
      Fix for ignored --default-mime-type option
      Merge branch 'bug/default-mime-type' into merge
      really merge @edrex https://github.com/s3tools/s3cmd/pull/244
      include python version in error dump
      remove python-2.5-ism when guessing mimetype (bug #218)
      don't hardlink 0-length files
      encode filenames to system encoding before calling shutils.copy2()
      getting the file system encoding is loop invariant
      Merge pull request #250 from mdomsch/bug/python-version-on-error
      Merge pull request #251 from mdomsch/bug/218
      Merge pull request #253 from mdomsch/bug/unicode-stat
      Print a useful error message when MemoryError is raised
      Merge pull request #259 from 36degrees/patch-1
      fixes for Windows not including a pwd module
      fix divide-by-zero when no files were transferred
      handle OSError: No such file or directory in get_local_filelist
      Merge branch 'bug/memoryerror' into merge
      Merge branch 'bug/divide-by-zero' into merge
      Merge branch 'bug/lstat-removed-file' into merge
      Merge branch 'bug/windows-no-pwd' into merge
      Merge git://github.com/robd003/s3cmd into merge
      --no-check-md5 should disable md5sum calculation, including
preserved attributes
      Merge branch 'bug/217' into merge
      fix put - s3://  (upload from stdin, #270)
      Merge branch 'bug/put-from-stdin' into merge
      os.gete[ug]id() doesn't exist on non-Unix
      Merge branch 'bug/put-from-stdin' into merge
      Merge git://github.com/flebel/s3cmd into merge
      add 'make tarball' and 'make rpm' to Makefile
      remove useless import of S3.Exceptions
      reduce "Process files that was not remote copied" to debug()
      add s3cmd.spec to .gitignore
      add a friendly error message on UnicodeEncodeError exception
      report LANG= environment variable on crash
      handle empty directory on S3 sync remote2local
      _download: Exception != OSError
      _download: handle S3Error
      Merge pull request #92 from
ksauzz/support-get-operation-to-multibyte-objectname
      Ignore S3 objects with empty names
      Merge pull request #284 from swiftserve/fix_config_file_verbosity
      set content-type, content-charset, and not content-encoding
      use ~/.s3cfg in run-tests too
      move empty object name detection where it belongs
      delete unused keys_to_remove logic
      s/0o775/0755/ for python 2.4 compatibility
      Merge pull request #286 from swiftserve/fix_verbosity_config
      Merge pull request #291 from Knewton/master
      Merge pull request #293...
 
[truncated message content]

[S3tools-general] Python breakage

[Russell G. <rus...@gm...>]

I had this series of error messages:
Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 4, in <module>
    __import__('pkg_resources').run_script('s3cmd==2.1.0', 's3cmd')
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line
3088, in <module>
    @_call_aside
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line
3072, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line
3101, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line
574, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line
892, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line
778, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 's3cmd==2.1.0' distribution was not
found and is required by the application

This is Linux Mint 19.3. I noticed a Python 2.7 update come through so
perhaps this caused the issue. Alternatively I have been messing with
Python 3.6 which I guess I would prefer to use if there is a choice. At one
time recently I changed my python link to python3 from python2 but found it
borked another app I use so I changed it back. Maybe that was the issue,

Please suggest what the issue is - any suggestions gratefully received.
Does s3cmd work on Python3?

Russell

Re: [S3tools-general] Can s3cmd preserve the original Linux file time stamp on restore?

[Florent V. (Sodria) <fl...@so...>]

Hi David,

Yes, the "sync" command will preserve users, creation and modification
dates if you use the following flag:
-p, --preserve

Take care that you need to use it both when uploading to the server and
downloading.
And any way, it would be better for you to give a try with test files.

Florent


On Sat, Feb 6, 2021 at 11:34 AM David Roth <dav...@gm...> wrote:

> Is there a way for s3cmd to restore the original Linux file time
> stamp when it is restored to a Linux server? I'm concerned about using
> s3cmd for legacy files, and losing the original time stamp on the files.
> Thanks.
>
> David Roth
>
> _______________________________________________
> S3tools-general mailing list
> S3t...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>

[S3tools-general] Can s3cmd preserve the original Linux file time stamp on restore?

[David R. <dav...@gm...>]

Is there a way for s3cmd to restore the original Linux file time stamp when
it is restored to a Linux server? I'm concerned about using s3cmd for
legacy files, and losing the original time stamp on the files. Thanks.

David Roth

Re: [S3tools-general] s3cmd sync issue

[Sharadha K. <sha...@hi...>]

Hi,

These are database files. So I am not sure about its content.

So it means that the files are the same but with different names and it is
trying to upload it only once? But that fails and so both the files are
copied individually. Is my understanding correct?

Regards,
Sharadha

On Fri, Dec 18, 2020 at 8:40 PM Florent Viard (Sodria) <fl...@so...>
wrote:

> Hi,
>
> Indeed there is no worry to have with this warning.
>
> To be smart, s3cmd identify files that have the same content, and in that
> case try to upload it only once and do remote copies to create the other
> files more efficiently.
> But some servers do not support remote copies, so, in that case there is
> this warning and we fallback to upload all the files individually.
>
> Regards,
> Florent
>
>
> Le ven. 18 déc. 2020 à 15:59, William Smith <w_...@co...> a
> écrit :
>
>> Just a WAG:  Are the two files the same?  Maybe it's detected the fact
>> that they are the same file (MD5 or such) and decided to make a local copy
>> rather than upload a new one...
>>
>> William P. N. Smith
>> ComputerSmiths Consulting, Inc.
>> 978-600-1621
>>
>>
>>
>>
>> On Dec 18, 2020, at 8:51 AM, Sharadha Krishna <
>> sha...@hi...> wrote:
>>
>> Hi,
>>
>> I am using the s3cmd sync command to sync a backup folder to the S3
>> bucket. I am testing this using a new s3 bucket.
>>
>> When I execute *s3cmd sync backups/ $S3_PATH/backups/*
>>
>> I get the following error message that delays my sync of just a few files
>> to 3-5 mins.
>>
>> WARNING: Unable to remote copy files
>> 's3://<bucket-name>/backups/mc-13-big-Filter.db' ->
>> 's3://<bucket-name>/backups/mc-17-big-Filter.db'
>>
>> Why is it trying to sync a different file to another one? Can someone
>> help me with this?
>>
>> Thanks,
>> Sharadha
>> _______________________________________________
>> S3tools-general mailing list
>> S3t...@li...
>> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>>
>>
>> _______________________________________________
>> S3tools-general mailing list
>> S3t...@li...
>> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>>
> _______________________________________________
> S3tools-general mailing list
> S3t...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>

Re: [S3tools-general] s3cmd sync issue

[Florent V. (Sodria) <fl...@so...>]

Hi,

Indeed there is no worry to have with this warning.

To be smart, s3cmd identify files that have the same content, and in that
case try to upload it only once and do remote copies to create the other
files more efficiently.
But some servers do not support remote copies, so, in that case there is
this warning and we fallback to upload all the files individually.

Regards,
Florent


Le ven. 18 déc. 2020 à 15:59, William Smith <w_...@co...> a
écrit :

> Just a WAG:  Are the two files the same?  Maybe it's detected the fact
> that they are the same file (MD5 or such) and decided to make a local copy
> rather than upload a new one...
>
> William P. N. Smith
> ComputerSmiths Consulting, Inc.
> 978-600-1621
>
>
>
>
> On Dec 18, 2020, at 8:51 AM, Sharadha Krishna <sha...@hi...>
> wrote:
>
> Hi,
>
> I am using the s3cmd sync command to sync a backup folder to the S3
> bucket. I am testing this using a new s3 bucket.
>
> When I execute *s3cmd sync backups/ $S3_PATH/backups/*
>
> I get the following error message that delays my sync of just a few files
> to 3-5 mins.
>
> WARNING: Unable to remote copy files
> 's3://<bucket-name>/backups/mc-13-big-Filter.db' ->
> 's3://<bucket-name>/backups/mc-17-big-Filter.db'
>
> Why is it trying to sync a different file to another one? Can someone help
> me with this?
>
> Thanks,
> Sharadha
> _______________________________________________
> S3tools-general mailing list
> S3t...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>
>
> _______________________________________________
> S3tools-general mailing list
> S3t...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>

Re: [S3tools-general] s3cmd sync issue

[William S. <w_...@co...>]

Just a WAG:  Are the two files the same?  Maybe it's detected the fact that they are the same file (MD5 or such) and decided to make a local copy rather than upload a new one...

William P. N. Smith
ComputerSmiths Consulting, Inc.
978-600-1621




> On Dec 18, 2020, at 8:51 AM, Sharadha Krishna <sha...@hi...> wrote:
> 
> Hi,
> 
> I am using the s3cmd sync command to sync a backup folder to the S3 bucket. I am testing this using a new s3 bucket. 
> 
> When I execute s3cmd sync backups/ $S3_PATH/backups/
> 
> I get the following error message that delays my sync of just a few files to 3-5 mins.
> 
> WARNING: Unable to remote copy files 's3://<bucket-name>/backups/mc-13-big-Filter.db' -> 's3://<bucket-name>/backups/mc-17-big-Filter.db'
> 
> Why is it trying to sync a different file to another one? Can someone help me with this?  
> 
> Thanks,
> Sharadha
> _______________________________________________
> S3tools-general mailing list
> S3t...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general

[S3tools-general] s3cmd sync issue

[Sharadha K. <sha...@hi...>]

Hi,

I am using the s3cmd sync command to sync a backup folder to the S3 bucket.
I am testing this using a new s3 bucket.

When I execute *s3cmd sync backups/ $S3_PATH/backups/*

I get the following error message that delays my sync of just a few files
to 3-5 mins.

WARNING: Unable to remote copy files
's3://<bucket-name>/backups/mc-13-big-Filter.db' ->
's3://<bucket-name>/backups/mc-17-big-Filter.db'

Why is it trying to sync a different file to another one? Can someone help
me with this?

Thanks,
Sharadha

Re: [S3tools-general] Delete files in S3 bcukets

[Sharadha K. <sha...@hi...>]

Hi Florent,

Yes, I have a bunch of files under a subfolder.

Alright got the point.



On Wed, Dec 16, 2020 at 2:34 PM Florent Viard (Sodria) <fl...@so...>
wrote:

> Hi Sharadha,
>
> Maybe it is because you just have this single file in folder1/folder2?
>
> In s3 you don't have the concept of 'files' and 'folders'. There are just
> files with a filename that can contains '/'.
> So we give the illusion of a file hierarchy based on the filename prefix
> but the folders in themselves doesn't exist.
>
> So, if you delete the single file in this part of the tree, you will not
> have any other files with the prefix folder1/folder2 and so you will think
> that the folders were deleted but it is just that they 'ever existed in the
> first place.
>
> If you had another file in folder1, it would still appear after the
> command.
>
> Regards,
>
> Florent
>
>
> Le mer. 16 déc. 2020 à 07:42, Sharadha Krishna <
> sha...@hi...> a écrit :
>
>> Hi,
>>
>> I am trying to delete files inside a particular folder in the s3 bucket.
>> For this I am using the command *s3cmd del
>> s3://bucket_name/folder1/folder2/file_name*.
>>
>> When I execute this, the entire directory structure gets deleted, meaning
>> folder1 and folder2 also get deleted along with the filename.
>>
>> So can someone let me know how to solve this?
>>
>> Thanks,
>> Sharadha
>> _______________________________________________
>> S3tools-general mailing list
>> S3t...@li...
>> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>>
> _______________________________________________
> S3tools-general mailing list
> S3t...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>

Re: [S3tools-general] Delete files in S3 bcukets

[Florent V. (Sodria) <fl...@so...>]

Hi Sharadha,

Maybe it is because you just have this single file in folder1/folder2?

In s3 you don't have the concept of 'files' and 'folders'. There are just
files with a filename that can contains '/'.
So we give the illusion of a file hierarchy based on the filename prefix
but the folders in themselves doesn't exist.

So, if you delete the single file in this part of the tree, you will not
have any other files with the prefix folder1/folder2 and so you will think
that the folders were deleted but it is just that they 'ever existed in the
first place.

If you had another file in folder1, it would still appear after the command.

Regards,

Florent


Le mer. 16 déc. 2020 à 07:42, Sharadha Krishna <sha...@hi...>
a écrit :

> Hi,
>
> I am trying to delete files inside a particular folder in the s3 bucket.
> For this I am using the command *s3cmd del
> s3://bucket_name/folder1/folder2/file_name*.
>
> When I execute this, the entire directory structure gets deleted, meaning
> folder1 and folder2 also get deleted along with the filename.
>
> So can someone let me know how to solve this?
>
> Thanks,
> Sharadha
> _______________________________________________
> S3tools-general mailing list
> S3t...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>

[S3tools-general] Delete files in S3 bcukets

[Sharadha K. <sha...@hi...>]

Hi,

I am trying to delete files inside a particular folder in the s3 bucket.
For this I am using the command *s3cmd del
s3://bucket_name/folder1/folder2/file_name*.

When I execute this, the entire directory structure gets deleted, meaning
folder1 and folder2 also get deleted along with the filename.

So can someone let me know how to solve this?

Thanks,
Sharadha

[S3tools-general] anonymous requests with s3cmd

[Luca P. <pi...@cr...>]

Hi,

with s3cmd, can I make anonymous requests, without specifying an access key?

I have a Ceph S3 object store with some buckets configured for read-only 
anonymous, unauthenticated access.  I can access these with the aws client:

```
$ aws --endpoint-url http://$AWS_S3_ENDPOINT --no-verify-ssl --region "" 
--no-sign-request s3 ls s3://test-policy
2020-11-20 14:50:55        298 policy_anon_read_bucket.json
```

I haven't been able to do the same operation with s3cmd.  If I provide 
an empty key on the command line I get an InvalidArgument error:

```
$ s3cmd --host-bucket= --no-ssl -v --host $AWS_S3_ENDPOINT --access_key 
"" --secret_key "" ls  s3://test-policy
ERROR: S3 error: 400 (InvalidArgument)
```

If I don't specify one on the command line, it picks up whatever key is 
in $HOME/.s3cfg (even if its invalid), and if I don't have a key in the 
configuration or if I remove the configuration altogether it gives me 
another error.

Thanks in advance for any suggestions.

Cheers,

Luca

Re: [S3tools-general] When Is The Next Release

[Frank C. <fr...@cr...>]

Florent,
Thanks for the response.  A few weeks or even a month or so is not an
issue, it would only be if you were planning to stretch it to years, or
were just unresponsive when I would think of taking it straight from
the repo.
Thanks for all you work keeping it working.
RegardsFrank
On Mon, 2020-10-12 at 01:32 +0200, Florent Viard (Sodria) wrote:
> Hi Frank,
> Sorry for the long delay of my reply.
> 
> First, thank you for keeping the package up to date for the Fedora
> and EPEL distributions.
> 
> I'm still working on a few things, and so I plan to be able to do the
> next release in 1 or 2 weeks maximum.
> In general the git repo version is somehow stable and usable, but I
> would prefer and advise to not take anything else than release for
> distribution.
> Release are more tested and it might be harder to investigate
> reported issues on random non release versions.
> 
> Also, especially for the next release, I have a few bugs in mind and
> at least something to cover python 3.9, so it will definitively be
> better if you could wait for the coming release.
> 
> Don't hesitate to ask if you need further details.
> 
> Regards,
> 
> Florent
> 
> On Tue, Oct 6, 2020 at 12:59 PM Frank Crawford <
> fr...@cr...> wrote:
> > Folks,
> > 
> > I'm currently maintaining of s3cmd for Fedora and EPEL, and the
> > latest release is s3cmd-2.1.0, however there are a few bugs that
> > have been reported that have been addressed in the current git
> > repo.
> > 
> > So, I am wondering what is your timeline for the next release, or
> > if I am better shipping what is currently in the git repo.  Related
> > to that, how stable do you see the git repo?  Is it suitable for
> > distribution, or what would you suggest?
> > 
> > Regards
> > Frank
> > _______________________________________________
> > 
> > S3tools-general mailing list
> > 
> > S3t...@li...
> > 
> > https://lists.sourceforge.net/lists/listinfo/s3tools-general
> > 
> 
> _______________________________________________S3tools-general
> mailing lis...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general

Re: [S3tools-general] When Is The Next Release

[Florent V. (Sodria) <fl...@so...>]

Hi Frank,

Sorry for the long delay of my reply.

First, thank you for keeping the package up to date for the Fedora and EPEL
distributions.

I'm still working on a few things, and so I plan to be able to do the next
release in 1 or 2 weeks maximum.
In general the git repo version is somehow stable and usable, but I would
prefer and advise to not take anything else than release for distribution.
Release are more tested and it might be harder to investigate reported
issues on random non release versions.

Also, especially for the next release, I have a few bugs in mind and at
least something to cover python 3.9, so it will definitively be better if
you could wait for the coming release.

Don't hesitate to ask if you need further details.

Regards,

Florent

On Tue, Oct 6, 2020 at 12:59 PM Frank Crawford <fr...@cr...>
wrote:

> Folks,
>
> I'm currently maintaining of s3cmd for Fedora and EPEL, and the latest
> release is s3cmd-2.1.0, however there are a few bugs that have been
> reported that have been addressed in the current git repo.
>
> So, I am wondering what is your timeline for the next release, or if I am
> better shipping what is currently in the git repo.  Related to that, how
> stable do you see the git repo?  Is it suitable for distribution, or what
> would you suggest?
>
> Regards
> Frank
> _______________________________________________
> S3tools-general mailing list
> S3t...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>

[S3tools-general] When Is The Next Release

[Frank C. <fr...@cr...>]

Folks,

I'm currently maintaining of s3cmd for Fedora and EPEL, and the latest
release is s3cmd-2.1.0, however there are a few bugs that have been
reported that have been addressed in the current git repo.

So, I am wondering what is your timeline for the next release, or if I
am better shipping what is currently in the git repo.  Related to that,
how stable do you see the git repo?  Is it suitable for distribution,
or what would you suggest?

Regards
Frank

[S3tools-general] Running s3cmd on a mac without sudo?

[Jeff W. <je...@je...>]

Is this possible? or is admin authority always required? If it's possible,
how do I install and run it? Thank you!

[S3tools-general] s3cmd - support for Signature Version 4

[Aaron Ni <Aar...@Ty...>]

Hi S3Tools Development Team,

We are testing the s3cmd tool and encountering the following error message:

ERROR: S3 error: 400 (InvalidArgument): Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.

Our usage scenario is:

  *   Copy file to S3 bucket
  *   S3 bucket configured with default encryption which uses a customer managed KMS key
  *   We are using s3cmd v2.1.0


Thanks in advance for your assistance

Regards,
Aaron.
************** Disclaimer ************************************ This email is intended solely for the use of the individual or entity named above as it may contain information that is confidential and privileged. If you are not the intended recipient, be advised that any dissemination, distribution or copying of this email is strictly prohibited. TymeBank cannot be held liable by any person other than the addressee in respect of any opinions, conclusions, advice or other information contained in this email. **************************************************************

[S3tools-general] eu-south-1 region problems

[Davide C. - ASB\\ <dav...@as...>]

Hi,
I try to install and configure s3cmd as always.
I have to move a zip file from ec2 instance to related bucket (both EC2 and S3 bucket are in eu-south-1).
After I launching the script this is the error message:

"ERROR: S3 error: 400 (IllegalLocationConstraintException): The eu-south-1 location constraint is incompatible for the region specific endpoint this request was sent to."

If I move the bucket from Milan to Frankfurt (eu-central-1) it works.
If the bucket is in Milan (eu-south-1) it seems it doesn't work.

Is there something like a access block or similar?
Is there a way to use s3cmd on eu-south-1?

Thank a lot
Davide Carbonini

Re: [S3tools-general] newbie, how to estimate s3 --requester-pays costs

[Florent V. (Sodria) <fl...@so...>]

There are some third parties online estimator on internet, but in your
case, the calculus should not be too hard.

In the following page, you have all the fees for all the cases:
https://aws.amazon.com/fr/s3/pricing/

Basically, there are only 2 kind of fees that will be charged to you for
this operation:
1) (GET,LIST, ....) fee required for the requests needed to do the operation
2) A data transfer out fee that is dependent of the size of the object that
you want to download.

For 1), you can just ignore it is it will probably be insignificant.
You just download a single file, so the overall operation will just
requires a dozen of requests.
You pay proportionally to your usage with something like 0.005 or 0.0004
per thousand of request.
So we can easily assume that you will probably be charged a lot less than
0.0005.

For 2), this is where the cost will be significant.
Here, I guess that you will transfer to "Internet" (ie you download to your
computer).
(Other case are transfer/copy bucket to bucket)
For this it is a price per GB, with different levels.
So, if  we take the example of the 100 GB file:
You will pay 0$ for the 1st GB +0.09 * 99 = 8.91$ for the remaining 99 GB.

You can get the size of the file by using the "info" command instead of
"get".

A last point to take care of, prices are not the same depending of the
"Region" where is located the file.
In the amazon page, there is a select box to select the region to see the
correct price.

If you can't get the region info from the source, you can do 2 things:
- Do your estimate with the most expensive region that is probable for your
case, and so you know the upper bound of the cost.
- When you call the "info" request, use the -d flag. It will show all the
debug logs of s3cmd, and you should be able to see in the first exchange in
which region is the file located, as we have to redirect the requests to
the correct region.

As a complement, you can find another example of calculating costs in the
readme of s3cmd:
https://github.com/s3tools/s3cmd/blob/master/README.md
(Just look at the idea, but refer to amazon page for the actual pricing)

Let me know if it is not clear enough,

Regards,

Florent

On Wed, Apr 29, 2020 at 10:58 PM Andrew Davidson via S3tools-general <
s3t...@li...> wrote:

> I need to estimate how the following transfer will cost. Any idea how I
> can go about this? Seem like AWS would have some sort of tool for this
> however I could not find it
>
>
> s3cmd get --requester-pays
> s3://cgl-rnaseq-recompute-fixed/cwl-example.tar.gz
>
> Kind regards
>
> Andy Davidson
> _______________________________________________
> S3tools-general mailing list
> S3t...@li...
> https://lists.sourceforge.net/lists/listinfo/s3tools-general
>

[S3tools-general] newbie, how to estimate s3 --requester-pays costs

[Andrew D. <aed...@uc...>]

I need to estimate how the following transfer will cost. Any idea how I can
go about this? Seem like AWS would have some sort of tool for this however
I could not find it


s3cmd get --requester-pays
s3://cgl-rnaseq-recompute-fixed/cwl-example.tar.gz

Kind regards

Andy Davidson

[S3tools-general] s3cmd v2.1.0

[Florent V. (Sodria) <fl...@so...>]

Dear s3cmd users and contributors,

I'm glad to announce that

s3cmd v2.1.0 has been released on github, sourceforge and PyPi.

This release is mostly made of bug fixes and should not contain any
breaking change.

Thanks again to all the contributors, the testers and the issue reporters.


s3cmd-2.1.0      - 2020-04-07
===============
* Changed size reporting using k instead of K as it a multiple of 1024 (#956)
* Added "public_url_use_https" config to generate public url using
https (#551, #666) (Jukka Nousiainen)
* Added option to make connection pooling configurable and
improvements (Arto Jantunen)
* Added support for path-style bucket access to signurl (Zac Medico)
* Added docker configuration and help to run test cases with multiple
Python versions (Doug Crozier)
* Relaxed limitation on special chars for --add-header key names (#1054)
* Fixed all regions that were automatically converted to lower case
(Harshavardhana)
* Fixed size and alignment of DU and LS output reporting (#956)
* Fixes for SignatureDoesNotMatch error when host port 80 or 443 is
specified, due to stupid servers (#1059)
* Fixed the useless retries of requests that fail because of ssl cert checks
* Fixed a possible crash when a file disappears during cache generation (#377)
* Fixed unicode issues with IAM (#987)
* Fixed unicode errors with bucked Policy/CORS requests (#847) (Alex Offshore)
* Fixed unicode issues when loading aws_credential_file (#989)
* Fixed an issue with the tenant feature of CephRGW. Url encode
bucket_name for path-style requests (#1080)
* Fixed signature v2 always used when bucket_name had special chars (#1081)
* Allow to use signature v4 only, even for commands without buckets
specified (#1082)
* Fixed small open file descriptor leaks.
* Py3: Fixed hash-bang in headers to not force using python2 when
setup/s3cmd/run-test scripts are executed directly.
* Py3: Fixed unicode issues with Cloudfront (#1006)
* Py3: Fixed http.client.RemoteDisconnected errors (#1014) (Ryan Huddleston)
* Py3: Fixed 'dictionary changed size during iteration' error when
using a cache-file (#945) (Doug Crozier)
* Py3: Fixed the display of file sizes (Vlad Presnyak)
* Py3: Python 3.8 compatibility fixes (Konstantin Shalygin)
* Py2: Fixed unicode errors sometimes crashing remote2remote sync (#847)
* Added s3cmd.egg-info to .gitignore (Philip Dubé)
* Improved run-test script to not use hard-coded bucket names(#1066)
(Doug Crozier)
* Renamed INSTALL to INSTALL.md and improvements (Nitro, Prabhakar Gupta)
* Improved the restore command help (Hrchu)
* Updated the storage-class command help with the recent aws s3 classes (#1020)
* Fixed typo in the --continue-put help message (Pengyu Chen)
* Fixed typo (#1062) (Tim Gates)
* Improvements for setup and build configurations
* Many other bug fixes


Shortlog below.


Thanks,
Florent


Alex Offshore (1):
      Return decoded response in get_policy and get_cors

Arto Jantunen (2):
      Add a close method to the ConnMan api
      Add option to disable connection pooling

Doug Crozier (4):
      Changed "Buckets list" test to use pbucket rather than hardcoded
names. This fixes issue #1066.
      Allow use of Docker to run test cases with multiple Python versions
      Fix for 'dictionary changed size during iteration' error on
Python 3 with --cache-file (Issue #945)
      Test cases added for --cache-file

EC2 Default User (1):
      .gitignore: add s3cmd.egg-info

Florent Viard (36):
      Fixes #987 - Fixes using IAM with python3. Config options should
be unicode.
      Fixes #987 - Before py36, py3 json was only able to load unicode str
      Complete the fix #986 - To satisfy py2 and p3, there should be
float before and after the division
      Issue #956 - Fixes size reporting using k instead of K as it a
multiple of 1024
      Fixes #956 - fix size and alignement of du and ls output reporting
      Convert anything given to getTreeFromXml to utf8 bytes if needed
      Fixes #1006 - fixes multiple python3 encoding issues with
cloudfront functions
      Fixes #1006 - missing return in previous commit
222af5d8ce90bbb2f7784360c019f5a6af2ef8bd
      Change version to development 2.0.2+
      setup.cfg: set universal bdist_wheel flag
      setup.py: update meta files with support for recent python versions
      travis-ci: Disable tests for python 2.6 as not supported anymore
by travis-ci default ubuntu distribution
      Complete the previous change to rename INSTALL file to INSTALL.md
      run-tests.py: Don't fail ACL tests if public_url_use_https is used
      svnignore: update to match latest gitignore content
      Little cleanup. TODO is out of date and obsolete
      Add a .dockerignore because of this stupid docker that does not
know how to exclude files he has no business with
      Update run-tests.docker.README.md with a small display fix
      Create FUNDING.yml
      Factorize the region detection inner request and stop retrying
requests that fail the ssl certificate checks
      Fixes #1059 - Hack for SignatureDoesNotMatch error when host
port 80 or 443 is specified, due to stupid servers
      run-test-*: Minor fixes
      Small cleanup
      Fixes #377 - Avoid crashing when a file disappeared during cache
generation
      Fixes #1080 - url encode bucket_name for path-style requests.
Fixes an issue with CephRGW and the tenant feature.
      Fixes #1081 - Don't force signature v4 if bucket_name uses special chars
      Fixes #1082 - Allow to use signature v4 only, even for a command
with no target bucket specified
      Fixes #1054 - Relax limitation on special chars for --add-header key names
      Fixes #1020 - Update storage-class cmd help msg with new aws s3 classes
      Fixes open file descriptors leaking by not being closed on time
      Added a comment about how to escape the comment char in exclude files
      Fixes #989 - in aws_credential_file, fixes read_string only
available with py3 + refactor for possible unicode issues
      Fixes #847 - set a few str as unicode to fix UnicodeDecodeError
with python2 when doing a sync remote 2 remote
      Fixes py headers to use system default python and not force
python2 anymore
      setup.py: Update authors and cleanup
      Update version to 2.1.0

Harshavardhana (1):
      Do not convert all regions to lower case

Jukka Nousiainen (2):
      Set public_url() protocol based on use_https
      Parameter for public URL protocol

Konstantin Shalygin (1):
      s3cmd: python 3.8 compat

Pengyu Chen (1):
      Fixed: Minor typo in help message for `--continue-put`

Prabhakar Gupta (1):
      Fix INSTALL.md formatting issues

Ryan Huddleston (1):
      With this commit 4 years ago OSError was caught:
https://github.com/s3tools/s3cmd/commit/889c5a7ba1ef8fc48f0c1be593e7ddf1e518a408

Tim Gates (1):
      Fix simple typo: unecessarily -> unnecessarily

Vlad Presnyak (1):
      Fixing proper handling of file sizes. Will correctly return
floating point or integer as appropriate.

Zac Medico (1):
      Fix signurl to support path-style bucket access via
check_bucket_name_dns_support

hrchu (1):
      Update restore cmd help

matteobar (1):
      Copyright year updated

nitro (1):
      README: brew install and renamed to INSTALL.md

[S3tools-general] S3CMD: Preserving Linux file permissions and dates in AWS S3 metadata?

[David R. <dav...@gm...>]

I read someplace that the Linux file permissions and dates could be stored
in the AWS S3 metadata, and that s3cmd could accomplish this. However, I
can't find examples of this being done or in the documentation. Perhaps I
overlooked it. Could someone please clarify if s3cmd can do this? If so,
either provide an example or documentation/article that covers this? Thanks!

David Roth