Menu

#147 resuming a multipart upload only skips up to 1000 existing files/parts

Enhancement_request
open
nobody
None
5
2014-06-15
2014-06-15
anierman
No

A multipart upload failed after several thousand 15MB files/parts had been uploaded.

After restarting the upload with put and --upload-id only the first 1000 files/parts were skipped due to md5/size matches. Re-uploading began at file/part 1001.

The issue is that list_multipart in S3.py only returns 1000 records at most, see:
http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListParts.html

To skip all of the appropriate matches--past 1000--IsTruncated could be detected in the response and appropriate output retrieved in a loop (making use of part-number​-marker).

Discussion


Log in to post a comment.

MongoDB Logo MongoDB