Didn’t CC the list, apparently sourceforge’s list manager isn’t terribly smart about Reply-To: …


From: Andrew Kesterson
Sent: Friday, May 17, 2013 6:32 AM
To: 'Matt Domsch'
Subject: RE: [S3tools-bugs] OSError [Errno 21] when syncing remote bucket with directory to local directory


                No dice. To begin with, that merge branch’s current HEAD is broken in general:


                Invoked as: /usr/bin/s3cmd sync s3://[redacted] .Problem: AttributeErr: Values instance has no attribute 'md5_xattr'


                … I ‘git checkout’ to revision 12cb7e0b7633d30a78afaa9b93484e4614a6b340 (before the xattr merge) and I get the same error that the current alpha head has:


                Invoked as: /usr/bin/s3cmd sync s3://[redacted] .Problem: OSErr: [Errno 20] Not a directory



                For what it’s worth, the current 0.9 build I have will work if I do a recursive get instead of a sync.  The current alpha head, however, has other issues on recursive gets (constantly retrying failed downloads?...)


From: Matt Domsch [mailto:matt@domsch.com]
Sent: Thursday, May 16, 2013 1:25 PM
To: Andrew Kesterson
Subject: Re: [S3tools-bugs] OSError [Errno 21] when syncing remote bucket with directory to local directory


Please try github.com/mdomsch/s3tools merge branch.  I need to push a new version into EPEL.

On May 16, 2013 12:15 PM, "Andrew Kesterson" <Andrew.Kesterson@vocalocity.com> wrote:

This occurs on s3cmd installed from epel on a CentOS 5 Linux system using python 2.4.3.


I have an s3 bucket with a ruby gems repository in it, like this:


[akesterson@puppet rubygems]$ tree


|-- Marshal.4.8

|-- Marshal.4.8.Z

|-- gems

|   |-- (redacted )

|-- latest_specs.4.8

|-- latest_specs.4.8.gz

|-- prerelease_specs.4.8

|-- prerelease_specs.4.8.gz

|-- quick

|   `-- Marshal.4.8

|       |-- ( redacted )

|-- specs.4.8

`-- specs.4.8.gz


3 directories, 22 files


And when I try to synchronize it down from s3, I get this error:


[akesterson@puppet rubygems]$ s3cmd sync s3://[redacted] .



    An unexpected error has occurred.

  Please report the following lines to:




Problem: OSErr: [Errno 21] Is a directory: './gems/'



Traceback (most recent call last):

  File "/usr/bin/s3cmd", line 1736, in ?


  File "/usr/bin/s3cmd", line 1681, in main


  File "/usr/bin/s3cmd", line 1070, in cmd_sync

    return cmd_sync_remote2local(args)

  File "/usr/bin/s3cmd", line 889, in cmd_sync_remote2local

    raise e

OSError: [Errno 21] Is a directory: './gems/'



    An unexpected error has occurred.

    Please report the above lines to:




This occurs whether the destination directory is empty or not.


I fired it up in pdb, and apparently this is due to s3 giving out wacky path names, some that end in the path separator:


> /usr/bin/s3cmd(868)cmd_sync_remote2local()

-> os.close(os.open(dst_file, open_flags))

(Pdb) item

{'object_uri_str': u's3://[redacted]/gems/', 'base_uri': <S3.S3Uri.S3UriS3 object at 0xcdf3990>, 'timestamp': 1362443115.0, 'local_filename': './gems/', 'object_key': 'gems/', 'size': 0, 'md5': 'd41d8cd98f00b204e9800998ecf8427e'}


… notice that ‘local_filename’ ends with the path separator; this apparently confuses os.path.dirname(), and causes it to think “./gems” is a directory and that there is a file on the right side of the / with no name. So “./gems” becomes dst_dir and “./gems/” becomes dst_file. Naturally, chaos ensues.


I started working on a patch for this, but got pulled off for other things. I’ll send it up to the list shortly if I beat you to the punch on a solution.



AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.