the s3cmd not support Asia china,To increase support for the Asia-related address,for Singapore,China
I am a Chinese Amazon user, use the S3cmd tool, but does not support China!
thanks
Thank you for the debug messages. That helps. We are using signature v4, so that's not the problem. It's not a trivial problem to solve. By changing the format of the host name (e.g. from <bucket>.s3.amazonaws.com to <bucket>.s3.<region>.amazonaws.com.cn), this breaks several hard-coded assumptions as to hostname to bucket mappings. In this case, it is manifest in the bucket name not appearing in the canonical_uri in the v4 signature, where it should be if it were to use the old-style s3://host/bucket/object path, rather than the new-style s3://bucket.host/object path. But it's not using the new-style path either.</region></bucket></bucket>
this is going to take quite a lot of thinking to fix properly. It would best be done in conjunction with directing requests at region-specific endpoints for all AWS regions. I had a start at this a while back, but it gets harder to also handle all the non-AWS S3-like systems out there - you can't make too many AWS hostname-based decisions and have it work for non-AWS systems too.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi, I got the same issue when using AWS for China region. How's it going? I know it's harder to handle all non-AWS s3-like systems, but can I ask why not handle all AWS systems for all regions first?
Last edit: Liang Sun 2015-07-23
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I don't know how to create access keys for China. But if you can do that, you can put these values into your ~/.s3cfg and direct requests there.
bucket_location = cn-north-1
host_base = s3.cn-north-1.amazonaws.com.cn
host_bucket = %(bucket).s3.cn-north-1.amazonaws.com.cn
Please let us know if that works for you, or if not, results with --debug.
ok,I test it!
DEBUG: ConfigParser: Reading file '/root/.s3cfg'
DEBUG: ConfigParser: access_key->AK...17_chars...A
DEBUG: ConfigParser: access_token->
DEBUG: ConfigParser: add_encoding_exts->
DEBUG: ConfigParser: add_headers->
DEBUG: ConfigParser: bucket_location->cn-north-1
DEBUG: ConfigParser: ca_certs_file->
DEBUG: ConfigParser: cache_file->
DEBUG: ConfigParser: check_ssl_certificate->True
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delay_updates->False
DEBUG: ConfigParser: delete_after->False
DEBUG: ConfigParser: delete_after_fetch->False
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: enable_multipart->True
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: expiry_date->
DEBUG: ConfigParser: expiry_days->
DEBUG: ConfigParser: expiry_prefix->
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->on...8_chars...4
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.cn-north-1.amazonaws.com.cn
DEBUG: ConfigParser: host_bucket->%(bucket).s3.cn-north-1.amazonaws.com.cn
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: ignore_failed_copy->False
DEBUG: ConfigParser: invalidate_default_index_on_cf->False
DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True
DEBUG: ConfigParser: invalidate_on_cf->False
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: max_delete->-1
DEBUG: ConfigParser: mime_type->
DEBUG: ConfigParser: multipart_chunk_size_mb->15
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: put_continue->False
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->4096
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: restore_days->1
DEBUG: ConfigParser: secret_key->5w...38_chars...
DEBUG: ConfigParser: send_chunk->4096
DEBUG: ConfigParser: server_side_encryption->False
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: use_mime_magic->True
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
DEBUG: ConfigParser: website_error->
DEBUG: ConfigParser: website_index->index.html
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config human_readable_sizes -> True
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'du' using UTF-8
DEBUG: Unicodising 's3://vpns3' using UTF-8
DEBUG: Command: du
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(vpns3): s3.cn-north-1.amazonaws.com.cn
DEBUG: canonical_headers = host:s3.cn-north-1.amazonaws.com.cn
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20150305T035200Z
DEBUG: Canonical Request:
GET
/
delimiter=%2F
host:s3.cn-north-1.amazonaws.com.cn
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20150305T035200Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxxxx/20150305/cn-north-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=8165b10bf824239bab9cdbebf03fe722ee1ef2e518b5baf7e82a49cebf49647c', 'x-amz-date': '20150305T035200Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(vpns3): s3.cn-north-1.amazonaws.com.cn
DEBUG: ConnMan.get(): creating new connection: http://s3.cn-north-1.amazonaws.com.cn
DEBUG: non-proxied HTTPConnection(s3.cn-north-1.amazonaws.com.cn)
DEBUG: format_uri(): /vpns3/?delimiter=/
DEBUG: Sending request method_string='GET', uri='/vpns3/?delimiter=/', headers={'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxxxx/20150305/cn-north-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=8165b10bf824239bab9cdbebf03fe722ee1ef2e518b5baf7e82a49cebf49647c', 'x-amz-date': '20150305T035200Z'}, body=(0 bytes)
DEBUG: Response: {'status': 403, 'headers': {'x-amz-id-2': '0r+1Ok5H0TuMC0+w22l7vxuplQQCQzvsQTxfUQtkwL9CjJYynjgHpck4eco7FsAa', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': 'EC1A35231D46E234', 'date': 'Thu, 05 Mar 2015 03:52:01 GMT', 'content-type': 'application/xml'}, 'reason': 'Forbidden', 'data': '\n<error>
SignatureDoesNotMatch<message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</message><awsaccesskeyid>xxxxxxxxxxxxxxxxxxx</awsaccesskeyid><stringtosign>AWS4-HMAC-SHA256\n20150305T035200Z\n20150305/cn-north-1/s3/aws4_request\nc22a3e92ee5cd0a45f7e8215d227f1091e0088a5c781c369636f8364f60f7c79</stringtosign><signatureprovided>8165b10bf824239bab9cdbebf03fe722ee1ef2e518b5baf7e82a49cebf49647c</signatureprovided><stringtosignbytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 30 33 30 35 54 30 33 35 32 30 30 5a 0a 32 30 31 35 30 33 30 35 2f 63 6e 2d 6e 6f 72 74 68 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 63 32 32 61 33 65 39 32 65 65 35 63 64 30 61 34 35 66 37 65 38 32 31 35 64 32 32 37 66 31 30 39 31 65 30 30 38 38 61 35 63 37 38 31 63 33 36 39 36 33 36 66 38 33 36 34 66 36 30 66 37 63 37 39</stringtosignbytes><canonicalrequest>GET\n/vpns3/\ndelimiter=%2F\nhost:s3.cn-north-1.amazonaws.com.cn\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20150305T035200Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</canonicalrequest><canonicalrequestbytes>47 45 54 0a 2f 76 70 6e 73 33 2f 0a 64 65 6c 69 6d 69 74 65 72 3d 25 32 46 0a 68 6f 73 74 3a 73 33 2e 63 6e 2d 6e 6f 72 74 68 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 2e 63 6e 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 35 30 33 30 35 54 30 33 35 32 30 30 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</canonicalrequestbytes><requestid>EC1A35231D46E234</requestid><hostid>0r+1Ok5H0TuMC0+w22l7vxuplQQCQzvsQTxfUQtkwL9CjJYynjgHpck4eco7FsAa</hostid></error>'}DEBUG: ConnMan.put(): connection put back to pool (http://s3.cn-north-1.amazonaws.com.cn#1)
ERROR: S3 error: The request signature we calculated does not match the signature you provided. Check your key and signing method.
hi, this is debug messages!
I asked AWS technical support engineer, he said China area just for V4 signature!
thank you!
Thank you for the debug messages. That helps. We are using signature v4, so that's not the problem. It's not a trivial problem to solve. By changing the format of the host name (e.g. from <bucket>.s3.amazonaws.com to <bucket>.s3.<region>.amazonaws.com.cn), this breaks several hard-coded assumptions as to hostname to bucket mappings. In this case, it is manifest in the bucket name not appearing in the canonical_uri in the v4 signature, where it should be if it were to use the old-style s3://host/bucket/object path, rather than the new-style s3://bucket.host/object path. But it's not using the new-style path either.</region></bucket></bucket>
this is going to take quite a lot of thinking to fix properly. It would best be done in conjunction with directing requests at region-specific endpoints for all AWS regions. I had a start at this a while back, but it gets harder to also handle all the non-AWS S3-like systems out there - you can't make too many AWS hostname-based decisions and have it work for non-AWS systems too.
Hi, I got the same issue when using AWS for China region. How's it going? I know it's harder to handle all non-AWS s3-like systems, but can I ask why not handle all AWS systems for all regions first?
Last edit: Liang Sun 2015-07-23