hyphen and extra digits appended to md5sum?

garden
2011-09-26
2013-02-19
  • garden
    garden
    2011-09-26

    Why does 's3cmd ls -list-md5' sometimes append a hyphen and number(s) to the md5sum?

    For example:

    2011-09-21 01:31   9045790   18fa2e421a60fc7a972bf4368d12de29-2  s3://bucketname/filename.3gp
    2011-09-21 01:32  20593256   597feeeb303ae660b12515b568f28cb9-4  s3://bucketname/filename.3gp
    2011-09-21 01:35  87817116   9d362a529ae32bb0bca2506d5c916975-17  s3://bucketname/filename.3gp
    2011-09-21 01:48  11685234   924155ee5dd930b62b2fc0bbcbcf8d25-3  s3://bucketname/filename.3gp
    2011-09-21 01:49  11513702   3bbbbfce772e3da29bb9d169b7f22ae6-3  s3://bucketname/filename.3gp
    2011-09-21 01:51  17471966   ec07fd9275e7c66f65c532baa10d736a-4  s3://bucketname/filename.3gp
    2011-09-21 01:53  15766830   2c45dc2a8bbdcf2f043b59c2c10a2329-4  s3://bucketname/filename.3gp
    2011-09-21 01:23  16975359   878cb20c5ead81739a7b8143e711fc26-4  s3://bucketname/filename.3gp
    2011-09-21 01:31     61146   0761f52d189485087ff636a46a5633f9  s3://bucketname/filename.jpg
    2011-09-21 01:31     66022   f85c25b31de40f73e8ca95d672478b04  s3://bucketname/filename.jpg
    2011-09-21 01:31     66819   932e5cba5d1915d11b7b2be29184a4a4  s3://bucketname/filename.jpg
    2011-09-21 01:31     64667   1d0a0859dd0179e79e399977a766fa07  s3://bucketname/filename.jpg
    
     
  • garden
    garden
    2011-09-26

    The hyphens are related to multi-part uploads.

    https://forums.aws.amazon.com/thread.jspa?messageID=203510

    It would be nice if 's3cmd sync' supported multi-part uploads.  Currently 's3cmd sync' interprets ETAG's with hyphens as an md5 checksum mismatch and tries to re-upload the file.

     
  • Chris F
    Chris F
    2011-09-28

    I'm using s3cmd sync to sync an s3 bucket to a local machine. I keep getting md5 mismatch warnings which causes the files to be downloaded every time s3cmd sync is run. This must be the reason. Is there a way around getting the mismatch errors?

     
  • It must be possible to add this feature to s3cmd.  The S3 Sync feature in Cyberduck does not try to sync the multi-part uploaded files, even though it lists the hyphenated hashes in the "Get Info" view.  I have not researched alternative command line tools.