Re: [S3tools-bugs] s3cmd: An unexpected error has occurred.

[Jonathan D. <jon...@na...>]

Matt,

Thanks for the response. The interesting thing about this error is that the directory that it says it is having problems with is not empty. In fact, there are no empty directories on the source filesystem (S3 in this case). And the first time I ran the command it actually transferred most of the files, which I have since removed on the local system.

I decided to start playing around with syntax and came up with some more interesting results.

*******************************************
Source has a trailing slash.

# s3cmd sync s3://bucket/folder/ .
s3://bucket/folder/ -> <fdopen>  [1 of 23]
 0 of 0     0% in    0s     0.00 B/s  done

Problem: OSError: [Errno 16] Device or resource busy

*******************************************
Source has no trailing slash.

# s3cmd sync s3://bucket/folder .
s3://bucket/folder/ -> <fdopen>  [1 of 23]
 0 of 0     0% in    0s     0.00 B/s  done

Problem: OSError: [Errno 20] Not a directory

*******************************************
Source has a slash star.

# s3cmd sync s3://bucket/folder/* .
#

*******************************************
Source is now a different bucket and the specified folder does not exist remotely or locally. There was no error, and it created a new folder locally.

# s3cmd sync s3://bucket2/folder/ .
# ll
drwxr-xr-x 2 root root 4096 Jul 15 21:41 folder/  <-- created a new local folder

*******************************************
Source has obviously bogus folder specification. No error, no creation of new folder locally.

# s3cmd sync s3://bucket2/adsfadsfafsd/ .
#

*******************************************
Source now has a sub-folder with contents. It downloaded the file fine, but when it hit the directory (not empty) it died.

# s3cmd sync s3://bucket2 .
s3://bucket2/README -> <fdopen>  [1 of 4]
 13130 of 13130   100% in    0s   102.03 kB/s  done
s3://bucket2/folder2/ -> <fdopen>  [2 of 4]
 0 of 0     0% in    0s     0.00 B/s  done

Problem: OSError: [Errno 20] Not a directory

*******************************************
I explicitly specified the sub-folder and it died.

s3cmd sync s3://bucket2/folder2 .
s3://bucket2/folder2/ -> <fdopen>  [1 of 3]
 0 of 0     0% in    0s     0.00 B/s  done

Problem: OSError: [Errno 20] Not a directory

*******************************************

The sync command seems to have issues with directories in general, empty or not.

Does s3cmd have a command cache? Some of the results seem weird.

Jonathan


On 15 Jul 2013, at 15:19, Matt Domsch wrote:

> We are getting this error report quite a lot lately. It's because s3cmd
> doesn't properly handle empty directories within buckets.  The original
> idea was that one could not have an empty directory, because the only
> things that could be referenced were objects (perhaps objects in a
> directory).  No object, no directory.  But then Amazon does have a way to
> create empty directories, and s3cmd doesn't know what to do with such.
> 
> I started a patch (https://github.com/mdomsch/s3cmd/tree/bug/174) but I
> don't think it's complete...  You're welcome to try and see if it works.
> 
> 
> On Mon, Jul 15, 2013 at 3:45 PM, Jonathan Duncan wrote:
> 
>> Problem: OSError: [Errno 20] Not a directory
>> S3cmd:   1.5.0-alpha3
>> 
>> Traceback (most recent call last):
>>  File "/usr/local/bin/s3cmd", line 2094, in <module>
>>    main()
>>  File "/usr/local/bin/s3cmd", line 2034, in main
>>    cmd_func(args)
>>  File "/usr/local/bin/s3cmd", line 1193, in cmd_sync
>>    return cmd_sync_remote2local(args)
>>  File "/usr/local/bin/s3cmd", line 891, in cmd_sync_remote2local
>>    seq, total_size = _download(remote_list, seq, remote_count +
>> update_count, total_size, dir_cache)
>>  File "/usr/local/bin/s3cmd", line 850, in _download
>>    raise e
>> OSError: [Errno 20] Not a directory
>>