MD5 from S3 seems uncorrect

  • Anonymous - 2012-08-28

    I was puzzled by this warning which I receive for all my downloads with "s3cmd sync"

    WARNING: MD5 signatures do not match: computed=Y, received="X"

    I noticed that X is the number that is also displayed in the AWS console under Properties > ETag

    However Y is the MD5 that I can calculate on the download target system (my VM) with md5sum. But also on the source system.

    I believe Y is the correct MD5 and X is some different identifier from Amazon but not an MD5.

  • Anonymous - 2012-11-14

    Did you ever resolve this?  I have the same issue, using s3tools-1.0.1.

    Thanks, Hans

  • Anonymous - 2012-12-02

    Is the file very large? Or, was it uploaded in a multipart upload?

    Multipart uploads on S3 don't get normal MD5 ETags. Instead they get an incorrect ETag with a dash and a number on the end.

    Eg. a0c0bb4ec5338586c0fccdb45238ee62-3

    From what I've read, multipart uploads to S3 are only necessary on files larger than 5GB, so not everybody comes up against this problem.

    If your ETag at S3 has a dash and a number on the end, then you're probably having the same problem. Luckily, there is a simple fix: Move the S3 object (it doesn't matter where); this will cause S3 to recalculate the ETag and produce a correct MD5 value.

    The simplest way to do this is to move the object to its existing location:

    s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET1/OBJECT1

    Try that; it might just fix your problem.

  • greg bell

    greg bell - 2015-08-11

    s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET1/OBJECT1

    Do not do this unless you want your object deleted. I confirmed this with two tests.


Log in to post a comment.