Hi Moorthy,

Your fix is now in s3cmd 1.0.0-rc1, thanks for the contribution!

See http://s3tools.org/s3cmd-100rc1-released for a list of all new features.


On 10/18/2010 04:59 AM, Srinivasa Moorthy wrote:

I faced this bug and got this resolved. Here is the problem and the fix:

Problem: If there are non-ascii characters in the S3 Key, "fixbucket" issues the error "UnicodeEncodeError: 'ascii' codec can't encode characters in position 188-189: ordinal not in range(128)"


1. First the S3 key (filename) obtained from the incoming XML is stored as "ascii" -- as a string, not as a unicode string. But the data itself can contain non-ascii characters. Marked bold is the change specified.

def cmd_fixbucket(args):
    for arg in args:
        culprit = S3Uri(arg)
        if culprit.type != "s3":
            raise ParameterError("Expecting S3Uri instead of: %s" % arg)
        response = s3.bucket_list_noparse(culprit.bucket(), culprit.object(), recursive = True)
        r_xent = re.compile("&#x[\da-fA-F]+;")
        response['data'] = unicode(response['data'], 'UTF-8')
        keys = re.findall("<Key>(.*?)</Key>", response['data'], re.MULTILINE)
        debug("Keys: %r" % keys)

2. Now the Key is saved into a unicode variable, we need to do appropriate encoding to convert to ascii. This is done in the same function, as below.

def cmd_fixbucket(args):
    def _unescape(text):
        def _unescape_fixup(m):
            return text # leave as is
        text = text.encode('ascii', 'xmlcharrefreplace')
        return re.sub("&#?\w+;", _unescape_fixup, text)

Hope this is helpful.


On 17/10/10 18:41, Srinivasa Moorthy wrote:
 Problem: UnicodeEncodeError: ('ascii', u's3://isnap/&amp;#039;&amp;#039;,&amp;#039;&amp;#039;,&amp;#039;res&amp;#039;,&amp;#039;1&amp;#039;,&amp;#039;&amp;#039;)&amp;quot;&amp;gt;&amp;lt;em&amp;gt;blessed herbs&amp;lt;/em&amp;gt;\xc2\xae reviews 2009&amp;lt;/a&amp;gt;&amp;lt&#x3;', 188, 190, 'ordinal not in range(128)')

Traceback (most recent call last):
  File "./s3cmd", line 1736, in <module>
  File "./s3cmd", line 1681, in main
  File "./s3cmd", line 496, in cmd_object_del
  File "./s3cmd", line 504, in subcmd_object_del_uri
    remote_list = fetch_remote_list(uri_str, require_attribs = False, recursive = recursive)
  File "./s3cmd", line 245, in fetch_remote_list
    uri_str = str(uri)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 188-189: ordinal not in range(128)

------------------------------------------------------------------------------ Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev