Hi there just wanted to report a bug… I can no longer use s3tools on a large dataset of files to sync

 

s3cmd sync --recursive s3://xxx/image/filename s3://xxx-development/image/

 

Files should be around ~1M in quantity and 25GB in size

 

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    An unexpected error has occurred.

  Please report the following lines to:

   s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

 

Problem: MemoryError:

S3cmd:   1.0.0

 

Traceback (most recent call last):

  File "/usr/bin/s3cmd", line 2006, in <module>

    main()

  File "/usr/bin/s3cmd", line 1950, in main

    cmd_func(args)

  File "/usr/bin/s3cmd", line 1215, in cmd_sync

    return cmd_sync_remote2remote(args)

  File "/usr/bin/s3cmd", line 850, in cmd_sync_remote2remote

    dst_list = fetch_remote_list(destination_base, recursive = True, require_attribs = True)

  File "/usr/bin/s3cmd", line 277, in fetch_remote_list

    for key in objectlist:

  File "/usr/share/s3cmd/S3/SortedDict.py", line 46, in __iter__

    return SortedDictIterator(self, self.keys())

  File "/usr/share/s3cmd/S3/SortedDict.py", line 42, in keys

    keys.sort()

MemoryError

 

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    An unexpected error has occurred.

    Please report the above lines to:

   s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


All the best,