Re: [S3tools-bugs] KeyError: 'size' failure

[Matt D. <ma...@do...>]

Thanks much for the debug output. This helps.

Your basic problem is that your local file system contains a file name '-'
which is highly unusual, and unexpected by s3cmd.   The 'stat -' output you
sent doesn't actually seem to correspond to the file on disk - stat
recognizes that a file named '-' is special too.  What does 'ls -alZ' on
the directory with that filename yield as information on the file '-' ?
 Can you delete it?

I have tried reproducing by creating a file named '-' in several manners,
but none cause s3cmd to fail in the manner you describe.






On Tue, Apr 1, 2014 at 12:18 PM, Moshe Yudkowsky <mo...@po...> wrote:

> Matt Domsch wrote, On 04/01/2014 10:11 AM:
>
>  can you run with --debug and determine which file name it's failing on,
>> and provide the stat() and the XML output from the debug on the file
>> information about that file?  I can't see yet why it's not setting
>> 'size' on one of those two.
>>
>
> Hi,
>
> First of all, I see to have a basic error:
>
> DEBUG: Unicodising "Bucket name 'aaaBBB' contains disallowed character
> 'B'. The only supported ones are: lowercase us-asc
> ii letters (a-z), digits (0-9), dot (.) and hyphen (-)." using UTF-8
>
> As for the problem at hand, I don't see any XML output.
>
> I ran a pass with --debug, and I don't see any XML output. What appears to
> be the file it chokes on is one called "-", or perhaps the one right after
> it, but all I see is the "-" followed by a failure.
>
> I do see
>
> DEBUG: Unicodising '-' using UTF-8
> DEBUG: Unicodising '/home/moshe/-' using UTF-8
>
> I see
>
> DEBUG: CHECK: -
> DEBUG: PASS: u'-'
>
>
> And
>
> DEBUG: Applying --exclude/--include
> DEBUG: CHECK: -
> DEBUG: PASS: '-'
>
> So apparently that file -- it's a typical filename goof when writing to
> stdout -- is unicoded properly.
>
>
> Assuming the problem is the file "-", I have the stat:
>
> #: stat \-
>   File: ‘-’
>   Size: 0               Blocks: 0          IO Block: 1024   character
> special file
> Device: bh/11d  Inode: 23          Links: 1     Device type: 88,14
> Access: (0620/crw--w----)  Uid: ( 1000/   moshe)   Gid: (    5/     tty)
> Access: 2014-04-01 12:17:04.339945557 -0500
> Modify: 2014-04-01 12:17:04.339945557 -0500
> Change: 2014-04-01 07:18:06.339945557 -0500
>  Birth: -
>
>
>
> Log fragment below:
>
> DEBUG: CHECK: yyy
> DEBUG: PASS: 'yyy'
> INFO: Found 64952 local files, 64950 remote files
> INFO: Verifying attributes...
> DEBUG: Comparing filelists (direction: local -> remote)
> DEBUG: CHECK: -
>
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>     An unexpected error has occurred.
>   Please try reproducing the error using
>   the latest s3cmd code from the git master
>   branch found at:
>     https://github.com/s3tools/s3cmd
>   If the error persists, please report the
>   following lines (removing any private
>
>   info as necessary) to:
>    s3t...@li...
> a
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
> DEBUG: Unicodising '/home/moshe/bin/s3cmd' using UTF-8
> DEBUG: Unicodising 'sync' using UTF-8
> DEBUG: Unicodising '--delete-removed' using UTF-8
> DEBUG: Unicodising '--delete-after' using UTF-8
> DEBUG: Unicodising '--human-readable-sizes' using UTF-8
> DEBUG: Unicodising '--debug' using UTF-8
> DEBUG: Unicodising '--exclude-from=/home/moshe/.s3backup/exclude-from'
> using UTF-8
> DEBUG: Unicodising '/home/moshe/' using UTF-8
> DEBUG: Unicodising 's3://aaaBBB/computers/machinename/moshe/' using UTF-8
> Invoked as: /home/moshe/bin/s3cmd sync --delete-removed --delete-after
> --human-readable-sizes --debug --exclude-from=/home/moshe/.s3backup/exclude-from
> /home/moshe/ s3://aaaBBB/computers/machinename/moshe/
>
> Problem: KeyError: 'size'
> S3cmd:   1.5.0-beta1
> python:   2.7.6 (default, Feb 26 2014, 00:34:35)
> [GCC 4.8.2]
> environment LANG=en_US.UTF-8
>
> Traceback (most recent call last):
>   File "/home/moshe/bin/s3cmd", line 2422, in <module>
>     main()
>   File "/home/moshe/bin/s3cmd", line 2344, in main
>     cmd_func(args)
>   File "/home/moshe/bin/s3cmd", line 1414, in cmd_sync
>     return cmd_sync_local2remote(args)
>   File "/home/moshe/bin/s3cmd", line 1398, in cmd_sync_local2remote
>     destination_base_uri = _single_process(local_list)
>   File "/home/moshe/bin/s3cmd", line 1209, in _single_process
>     _child(destination_base, local_list)
>   File "/home/moshe/bin/s3cmd", line 1295, in _child
>     local_list, remote_list, update_list, copy_pairs =
> compare_filelists(local_list, remote_list, src_remote = False, dst_remote =
> True, delay_updates = cfg.delay_updates)
>   File "/home/moshe/unbacked/code/s3cmd/S3/FileLists.py", line 525, in
> compare_filelists
>     same_file = _compare(src_list, dst_list, src_remote, dst_remote,
> relative_file)
>   File "/home/moshe/unbacked/code/s3cmd/S3/FileLists.py", line 471, in
> _compare
>     if 'size' in cfg.sync_checks and dst_list[file]['size'] !=
> src_list[file]['size']:
> KeyError: 'size'
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>     An unexpected error has occurred.
>   Please try reproducing the error using
>   the latest s3cmd code from the git master
>   branch found at:
>     https://github.com/s3tools/s3cmd
>   If the error persists, please report the
>   above lines (removing any private
>   info as necessary) to:
>    s3t...@li...
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>
> --
> Moshe Yudkowsky * mo...@po... * www.pobox.com/~moshe
>  "If you pay peanuts, you get monkeys."
>   Edward Yourdon, _The Decline and Fall of the American Progammer_
>
>