#106 s3cmd does not work properly with non-ascii filenames

Malfunction
closed
nobody
s3cmd (119)
5
2011-11-07
2011-11-05
Anonymous
No

Bug 3194827 and bug 3424181 is the same error, I believe.

My filesystem is set to nb_no,utf-8
This means I have files with the letters æ,ø and å in them.

Files with these letters in the filenames will upload to S3 without a problem.
But the next sync will first delete them, and then re-upload the same files again.
Kind of a pita with hundreds of rather large files affected :)

"Fix" for SYNC (only)
---------------------
I changed line 1356 in s3cmd (1.0.1):
Original: text = text.encode('ascii', 'xmlcharrefreplace')
New: text = text.decode("utf-8").encode('ascii', 'xmlcharrefreplace')

and now it works... :)
That is, I've tested it, and I can't find any errors.

This is an entirely unsupported fix!!!
I use it, but I take NO responsibility for anyone else!
This fix might cause other parts of s3cmd to break!
If you modify s3cmd, you're on your own!

I only uploaded the "code" to help the developers with a starting point for their eventual enquiries..

---------------------------------------------------------------------------

This fix does not help with the fact that GET fails...
---------------------------------------------------------------------------

$ s3cmd-1.0.1/s3cmd get s3://<bucket>/Test/teestæææ.txt /private/Test/teestæææ.txt

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the following lines to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Problem: UnicodeEncodeErr: 'ascii' codec can't encode characters in position 34-36: ordinal not in range(128)
S3cmd: 1.0.1

Traceback (most recent call last):
File ".../s3cmd-1.0.1/s3cmd", line 2006, in ?
main()
File ".../s3cmd-1.0.1/s3cmd", line 1950, in main
cmd_func(args)
File ".../s3cmd-1.0.1/s3cmd", line 439, in cmd_object_get
remote_list = fetch_remote_list(args, require_attribs = False)
File ".../s3cmd-1.0.1/s3cmd", line 281, in fetch_remote_list
uri_str = str(uri)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 34-36: ordinal not in range(128)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the above lines to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

---------------------------------------------------------------------------
Or that PUT also fails...
---------------------------------------------------------------------------

$ s3cmd-1.0.1/s3cmd-tl put /private/Test/testøøå.txt s3://<bucket>/Test/testøøå.txt
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the following lines to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Problem: UnicodeEncodeErr: 'ascii' codec can't encode characters in position 33-35: ordinal not in range(128)
S3cmd: 1.0.1

Traceback (most recent call last):
File ".../s3cmd-1.0.1/s3cmd-tl", line 2006, in ?
main()
File ".../s3cmd-1.0.1/s3cmd-tl", line 1950, in main
cmd_func(args)
File ".../s3cmd-1.0.1/s3cmd-tl", line 330, in cmd_object_put
destination_base = str(destination_base_uri)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 33-35: ordinal not in range(128)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the above lines to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Discussion

  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-11-05

    Note: forget the fact that the PUT command is executed through /s3cmd-tl, that's just a debug thing.
    I tried with both original and edited s3cmd.

     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-11-07
    • status: open --> closed
     
  • Comment has been marked as spam. 
    Undo

    You can see all pending comments posted by this user  here

    Anonymous - 2011-11-07

    It seems I was mistaken.
    s3cmd works as advertised when i run it interactively, but not via a cron shell script.
    I must assume there's something wring with my environment variables

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks