From: Andrew K. <And...@vo...> - 2013-05-16 17:15:35
|
This occurs on s3cmd 0.9.9.91-1 installed from epel on a CentOS 5 Linux system using python 2.4.3. I have an s3 bucket with a ruby gems repository in it, like this: [akesterson@puppet rubygems]$ tree . |-- Marshal.4.8 |-- Marshal.4.8.Z |-- gems | |-- (redacted ) |-- latest_specs.4.8 |-- latest_specs.4.8.gz |-- prerelease_specs.4.8 |-- prerelease_specs.4.8.gz |-- quick | `-- Marshal.4.8 | |-- ( redacted ) |-- specs.4.8 `-- specs.4.8.gz 3 directories, 22 files And when I try to synchronize it down from s3, I get this error: [akesterson@puppet rubygems]$ s3cmd sync s3://[redacted] . !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please report the following lines to: s3t...@li... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Problem: OSErr: [Errno 21] Is a directory: './gems/' S3cmd: 0.9.9.91 Traceback (most recent call last): File "/usr/bin/s3cmd", line 1736, in ? main() File "/usr/bin/s3cmd", line 1681, in main cmd_func(args) File "/usr/bin/s3cmd", line 1070, in cmd_sync return cmd_sync_remote2local(args) File "/usr/bin/s3cmd", line 889, in cmd_sync_remote2local raise e OSError: [Errno 21] Is a directory: './gems/' !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please report the above lines to: s3t...@li... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! This occurs whether the destination directory is empty or not. I fired it up in pdb, and apparently this is due to s3 giving out wacky path names, some that end in the path separator: > /usr/bin/s3cmd(868)cmd_sync_remote2local() -> os.close(os.open(dst_file, open_flags)) (Pdb) item {'object_uri_str': u's3://[redacted]/gems/', 'base_uri': <S3.S3Uri.S3UriS3 object at 0xcdf3990>, 'timestamp': 1362443115.0, 'local_filename': './gems/', 'object_key': 'gems/', 'size': 0, 'md5': 'd41d8cd98f00b204e9800998ecf8427e'} ... notice that 'local_filename' ends with the path separator; this apparently confuses os.path.dirname(), and causes it to think "./gems" is a directory and that there is a file on the right side of the / with no name. So "./gems" becomes dst_dir and "./gems/" becomes dst_file. Naturally, chaos ensues. I started working on a patch for this, but got pulled off for other things. I'll send it up to the list shortly if I beat you to the punch on a solution. Thanks. |
From: Michael W. <esi...@gm...> - 2013-05-16 19:21:38
|
Hi. Before patching it, try out 1.5.0 alpha 3. On 16 May 2013 19:00, Andrew Kesterson <And...@vo...>wrote: > This occurs on s3cmd 0.9.9.91-1 installed from epel on a CentOS 5 Linux > system using python 2.4.3. **** > > ** ** > > I have an s3 bucket with a ruby gems repository in it, like this:**** > > ** ** > > [akesterson@puppet rubygems]$ tree**** > > .**** > > |-- Marshal.4.8**** > > |-- Marshal.4.8.Z**** > > |-- gems**** > > | |-- (redacted )**** > > |-- latest_specs.4.8**** > > |-- latest_specs.4.8.gz**** > > |-- prerelease_specs.4.8**** > > |-- prerelease_specs.4.8.gz**** > > |-- quick**** > > | `-- Marshal.4.8**** > > | |-- ( redacted )**** > > |-- specs.4.8**** > > `-- specs.4.8.gz**** > > ** ** > > 3 directories, 22 files**** > > ** ** > > And when I try to synchronize it down from s3, I get this error:**** > > ** ** > > [akesterson@puppet rubygems]$ s3cmd sync s3://[redacted] .**** > > ** ** > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!**** > > An unexpected error has occurred.**** > > Please report the following lines to:**** > > s3t...@li...**** > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!**** > > ** ** > > Problem: OSErr: [Errno 21] Is a directory: './gems/'**** > > S3cmd: 0.9.9.91**** > > ** ** > > Traceback (most recent call last):**** > > File "/usr/bin/s3cmd", line 1736, in ?**** > > main()**** > > File "/usr/bin/s3cmd", line 1681, in main**** > > cmd_func(args)**** > > File "/usr/bin/s3cmd", line 1070, in cmd_sync**** > > return cmd_sync_remote2local(args)**** > > File "/usr/bin/s3cmd", line 889, in cmd_sync_remote2local**** > > raise e**** > > OSError: [Errno 21] Is a directory: './gems/'**** > > ** ** > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!**** > > An unexpected error has occurred.**** > > Please report the above lines to:**** > > s3t...@li...**** > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!**** > > ** ** > > This occurs whether the destination directory is empty or not.**** > > ** ** > > I fired it up in pdb, and apparently this is due to s3 giving out wacky > path names, some that end in the path separator:**** > > ** ** > > > /usr/bin/s3cmd(868)cmd_sync_remote2local()**** > > -> os.close(os.open(dst_file, open_flags))**** > > (Pdb) item**** > > {'object_uri_str': u's3://[redacted]/gems/', 'base_uri': <S3.S3Uri.S3UriS3 > object at 0xcdf3990>, 'timestamp': 1362443115.0, 'local_filename': > './gems/', 'object_key': 'gems/', 'size': 0, 'md5': > 'd41d8cd98f00b204e9800998ecf8427e'}**** > > ** ** > > … notice that ‘local_filename’ ends with the path separator; this > apparently confuses os.path.dirname(), and causes it to think “./gems” is a > directory and that there is a file on the right side of the / with no name. > So “./gems” becomes dst_dir and “./gems/” becomes dst_file. Naturally, > chaos ensues.**** > > ** ** > > I started working on a patch for this, but got pulled off for other > things. I’ll send it up to the list shortly if I beat you to the punch on a > solution.**** > > ** ** > > Thanks.**** > > > ------------------------------------------------------------------------------ > AlienVault Unified Security Management (USM) platform delivers complete > security visibility with the essential security capabilities. Easily and > efficiently configure, manage, and operate all of your security controls > from a single console and one unified framework. Download a free trial. > http://p.sf.net/sfu/alienvault_d2d > -- Michael Wood <esi...@gm...> |
From: Andrew K. <And...@vo...> - 2013-05-17 10:53:16
|
Didn’t CC the list, apparently sourceforge’s list manager isn’t terribly smart about Reply-To: … From: Andrew Kesterson Sent: Friday, May 17, 2013 6:32 AM To: 'Matt Domsch' Subject: RE: [S3tools-bugs] OSError [Errno 21] when syncing remote bucket with directory to local directory No dice. To begin with, that merge branch’s current HEAD is broken in general: │Invoked as: /usr/bin/s3cmd sync s3://[redacted] .Problem: AttributeErr: Values instance has no attribute 'md5_xattr' … I ‘git checkout’ to revision 12cb7e0b7633d30a78afaa9b93484e4614a6b340 (before the xattr merge) and I get the same error that the current alpha head has: │Invoked as: /usr/bin/s3cmd sync s3://[redacted] .Problem: OSErr: [Errno 20] Not a directory For what it’s worth, the current 0.9 build I have will work if I do a recursive get instead of a sync. The current alpha head, however, has other issues on recursive gets (constantly retrying failed downloads?...) From: Matt Domsch [mailto:ma...@do...] Sent: Thursday, May 16, 2013 1:25 PM To: Andrew Kesterson Subject: Re: [S3tools-bugs] OSError [Errno 21] when syncing remote bucket with directory to local directory Please try github.com/mdomsch/s3tools<http://github.com/mdomsch/s3tools> merge branch. I need to push a new version into EPEL. On May 16, 2013 12:15 PM, "Andrew Kesterson" <And...@vo...<mailto:And...@vo...>> wrote: This occurs on s3cmd 0.9.9.91-1 installed from epel on a CentOS 5 Linux system using python 2.4.3. I have an s3 bucket with a ruby gems repository in it, like this: [akesterson@puppet rubygems]$ tree . |-- Marshal.4.8 |-- Marshal.4.8.Z |-- gems | |-- (redacted ) |-- latest_specs.4.8 |-- latest_specs.4.8.gz |-- prerelease_specs.4.8 |-- prerelease_specs.4.8.gz |-- quick | `-- Marshal.4.8 | |-- ( redacted ) |-- specs.4.8 `-- specs.4.8.gz 3 directories, 22 files And when I try to synchronize it down from s3, I get this error: [akesterson@puppet rubygems]$ s3cmd sync s3://[redacted] . !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please report the following lines to: s3t...@li...<mailto:s3t...@li...> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Problem: OSErr: [Errno 21] Is a directory: './gems/' S3cmd: 0.9.9.91 Traceback (most recent call last): File "/usr/bin/s3cmd", line 1736, in ? main() File "/usr/bin/s3cmd", line 1681, in main cmd_func(args) File "/usr/bin/s3cmd", line 1070, in cmd_sync return cmd_sync_remote2local(args) File "/usr/bin/s3cmd", line 889, in cmd_sync_remote2local raise e OSError: [Errno 21] Is a directory: './gems/' !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please report the above lines to: s3t...@li...<mailto:s3t...@li...> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! This occurs whether the destination directory is empty or not. I fired it up in pdb, and apparently this is due to s3 giving out wacky path names, some that end in the path separator: > /usr/bin/s3cmd(868)cmd_sync_remote2local() -> os.close(os.open(dst_file, open_flags)) (Pdb) item {'object_uri_str': u's3://[redacted]/gems/', 'base_uri': <S3.S3Uri.S3UriS3 object at 0xcdf3990>, 'timestamp': 1362443115.0, 'local_filename': './gems/', 'object_key': 'gems/', 'size': 0, 'md5': 'd41d8cd98f00b204e9800998ecf8427e'} … notice that ‘local_filename’ ends with the path separator; this apparently confuses os.path.dirname(), and causes it to think “./gems” is a directory and that there is a file on the right side of the / with no name. So “./gems” becomes dst_dir and “./gems/” becomes dst_file. Naturally, chaos ensues. I started working on a patch for this, but got pulled off for other things. I’ll send it up to the list shortly if I beat you to the punch on a solution. Thanks. ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d |