[S3tools-general] unicode, wildcards

[Joe H. <joe...@gm...>]

Hello, the s3cmd 'mv' command fails if the source contains non-ASCII characters, for example:

$ s3cmd mv s3://bucket1/Thörn.wav s3://bucket2/

Problem: UnicodeEncodeError: ('ascii', u's3://bucket1/Th\xf6rn.wav', 15, 16, 'ordinal not in range(128)')
S3cmd:   0.9.9.91

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 1736, in <module>
    main()
  File "/usr/local/bin/s3cmd", line 1681, in main
    cmd_func(args)
  File "/usr/local/bin/s3cmd", line 582, in cmd_mv
    subcmd_cp_mv(args, s3.object_move, "move", "File %(src)s moved to %(dst)s")
  File "/usr/local/bin/s3cmd", line 533, in subcmd_cp_mv
    remote_list = fetch_remote_list(args, require_attribs = False)
  File "/usr/local/bin/s3cmd", line 245, in fetch_remote_list
    uri_str = str(uri)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 15: ordinal not in range(128)

The problem isn't with the locale, but with the wildcard handling code here:

		for uri in remote_uris:
			uri_str = str(uri)
			## Wildcards used in remote URI?
			## If yes we'll need a bucket listing...
			if uri_str.find('*') > -1 or uri_str.find('?') > -1:

str(uri) fails with a non-ASCII uri. I fixed it by changing this to unicode(uri). I'm a newcomer to python and there might be a better fix. I also added a command-line option to make the wildcard parsing optional. Otherwise you'll get into trouble 'mv'ing files with '?' and '*' in the name. Here are the total diffs (0.9.9.1):

+++ ./S3/Config.py	2010-04-08 12:40:30.000000000 -0400
@@ -71,0 +72 @@
+	wildcard = True

+++ ./s3cmd	2010-04-08 12:41:16.000000000 -0400
@@ -245 +245 @@
-			uri_str = str(uri)
+			uri_str = unicode(uri)
@@ -248 +248 @@
-			if uri_str.find('*') > -1 or uri_str.find('?') > -1:
+			if cfg.wildcard and (uri_str.find('*') > -1 or uri_str.find('?') > -1):
@@ -1515,0 +1516,2 @@
+	optparser.add_option(      "--wildcard", dest="wildcard", action="store_true", help="Handle wildcards in remote file lists")
+	optparser.add_option(      "--no-wildcard", dest="wildcard", action="store_false", help="Disable handling wildcards in remote file lists")

Thanks for an otherwise excellent tool. We use s3cmd at Bandcamp to move thousands of files each day to and from S3 without trouble.

Joe