Re: [S3tools-bugs] Bug Report

[Matt D. <ma...@do...>]

MemoryError occurs when you have that many files in a single sync.
https://github.com/mdomsch/s3cmd/commit/6decc41b220a160e30099d2905829c2c2b146e8e
(pending
inclusion) adds this error message when such occurs:

+    except MemoryError:

+        msg = """

+MemoryError!  You have exceeded the amount of memory available for
this process.

+This usually occurs when syncing >750,000 files on a 32-bit python instance.

+The solutions to this are:

+1) sync several smaller subtrees; or

+2) use a 64-bit python on a 64-bit OS with >8GB RAM

+        """

+        sys.stderr.write(msg)

+        sys.exit(1)

+



On Fri, Jan 3, 2014 at 8:54 AM, Vinicius Ferriani <vin...@gm...>wrote:

> Hi there just wanted to report a bug… I can no longer use s3tools on a
> large dataset of files to sync
>
>
>
> s3cmd sync --recursive s3://xxx/image/filename s3://xxx-development/image/
>
>
>
> Files should be around ~1M in quantity and 25GB in size
>
>
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>     An unexpected error has occurred.
>
>   Please report the following lines to:
>
>    s3t...@li...
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>
>
> Problem: MemoryError:
>
> S3cmd:   1.0.0
>
>
>
> Traceback (most recent call last):
>
>   File "/usr/bin/s3cmd", line 2006, in <module>
>
>     main()
>
>   File "/usr/bin/s3cmd", line 1950, in main
>
>     cmd_func(args)
>
>   File "/usr/bin/s3cmd", line 1215, in cmd_sync
>
>     return cmd_sync_remote2remote(args)
>
>   File "/usr/bin/s3cmd", line 850, in cmd_sync_remote2remote
>
>     dst_list = fetch_remote_list(destination_base, recursive = True,
> require_attribs = True)
>
>   File "/usr/bin/s3cmd", line 277, in fetch_remote_list
>
>     for key in objectlist:
>
>   File "/usr/share/s3cmd/S3/SortedDict.py", line 46, in __iter__
>
>     return SortedDictIterator(self, self.keys())
>
>   File "/usr/share/s3cmd/S3/SortedDict.py", line 42, in keys
>
>     keys.sort()
>
> MemoryError
>
>
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>     An unexpected error has occurred.
>
>     Please report the above lines to:
>
>    s3t...@li...
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
> All the best,
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
>