I've run into this with later versions too.  32-bit python _will_ run out of memory like this if you have >800k files in your trees.  You can either split your sync runs into smaller chunks (e.g. sync fewer directories, one after the other), or use a 64-bit python.  64-bit python consumes roughly 2x the memory of 32-bit python (all those objects have lots of pointers, which dominates the runtime size), but if you have a 64-bit system and can spare sufficient memory or swap, that works.

The s3tools project won't be fixing python's memory hogging problems on 64-bit, and we'd be hard-pressed to use less memory per file - if anything I've made it worse by storing _more_ metadata about files recently to speed up runtime by eliminating more unnecessary file transfers.


On Wed, Apr 17, 2013 at 4:52 PM, Bob Tucker <Bob.Tucker@bazaarvoice.com> wrote:
s3cmd put --recursive <large file tree with lots and lots of files>  <empty s3 bucket>

Sending as requested

    An unexpected error has occurred.
  Please report the following lines to:

Problem: MemoryError: 
S3cmd:   1.0.1

Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 2006, in <module>
  File "/usr/bin/s3cmd", line 1950, in main
  File "/usr/bin/s3cmd", line 335, in cmd_object_put
    local_list, single_file_local = fetch_local_list(args)
  File "/usr/bin/s3cmd", line 245, in fetch_local_list
    list_for_uri, single_file = _get_filelist_local(uri)
  File "/usr/bin/s3cmd", line 689, in _get_filelist_local
    'mtime' : sr.st_mtime,

    An unexpected error has occurred.
    Please report the above lines to:

Bazaarvoice Robert Tucker 
Senior DevOps Engineer 
o: 415.489.6067  m: 415.269.5484 
e: bob.tucker@bazaarvoice.com 
Site  |  Blog  |  Twitter

Lots of planets have a North!

Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!