SourceForge has been redesigned. Learn more.
Close

#123 Multi-part uploads corrupts md5checksum (etag)

Malfunction
closed-fixed
nobody
s3cmd (119)
5
2014-06-17
2012-08-08
Anonymous
No

Using 1.1.0-beta3

Uploading a large file (120Mb) is causing the md5 (etag) checksum to be a corrupted value..... (eg. 22e201a50836af721e8494198d0bc708-9 below)

First without multi part and everything is fine

-bash-3.2# s3cmd put --disable-multipart zzzz s3://bityota_test/test/test
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
zzzz -> s3://bityota_test/test/test [1 of 1]
126213121 of 126213121 100% in 10s 11.45 MB/s done
-bash-3.2# s3cmd get s3://bityota_test/test/test yyyy
s3://bityota_test/test/test -> yyyy [1 of 1]
126213121 of 126213121 100% in 5s 22.62 MB/s done

Now as multipart, and the md5sum has a dash (which is invalid for a hex-string)

-bash-3.2# s3cmd put zzzz s3://bityota_test/test/test
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
zzzz -> s3://bityota_test/test/test [part 1 of 9, 15MB]
15728640 of 15728640 100% in 1s 9.43 MB/s done
zzzz -> s3://bityota_test/test/test [part 2 of 9, 15MB]
15728640 of 15728640 100% in 1s 13.42 MB/s done
zzzz -> s3://bityota_test/test/test [part 3 of 9, 15MB]
15728640 of 15728640 100% in 1s 13.99 MB/s done
zzzz -> s3://bityota_test/test/test [part 4 of 9, 15MB]
15728640 of 15728640 100% in 4s 3.71 MB/s done
zzzz -> s3://bityota_test/test/test [part 5 of 9, 15MB]
15728640 of 15728640 100% in 3s 4.13 MB/s done
zzzz -> s3://bityota_test/test/test [part 6 of 9, 15MB]
15728640 of 15728640 100% in 0s 15.53 MB/s done
zzzz -> s3://bityota_test/test/test [part 7 of 9, 15MB]
15728640 of 15728640 100% in 1s 12.07 MB/s done
zzzz -> s3://bityota_test/test/test [part 8 of 9, 15MB]
15728640 of 15728640 100% in 1s 14.52 MB/s done
zzzz -> s3://bityota_test/test/test [part 9 of 9, 375kB]
384001 of 384001 100% in 0s 4.11 MB/s done
-bash-3.2# s3cmd get s3://bityota_test/test/test xxxx
s3://bityota_test/test/test -> xxxx [1 of 1]
126213121 of 126213121 100% in 6s 18.43 MB/s done
WARNING: MD5 signatures do not match: computed=9e15f4c02f44f8b439a838322793ebd1, received="22e201a50836af721e8494198d0bc708-9"

-bash-3.2# md5sum xxxx yyyy zzzz
9e15f4c02f44f8b439a838322793ebd1 xxxx
9e15f4c02f44f8b439a838322793ebd1 yyyy
9e15f4c02f44f8b439a838322793ebd1 zzzz

Discussion

  • reimerp

    reimerp - 2012-09-20

    Having the same issue with 1.1.0-beta2, and searched for documentation of this feature. When not using multipart the S3 Etag seems to reflect the md5 checksum. If using multipart the suffix -9 reflect the number of the chunks.
    See
    https://forums.aws.amazon.com/thread.jspa?messageID=203436&#203436
    for details.
    I think s3cmd should not compare the locally computed md5 (which is correct) against the received etag if the suffi is found.

     
  • Matt Domsch

    Matt Domsch - 2014-06-17
    • status: open --> closed-fixed
     
  • Matt Domsch

    Matt Domsch - 2014-06-17

    This is fixed in upstream github.com/s3tools/s3cmd master branch. Checksums for multipart files are not checked, unless there was also an s3cmd-generated checksum stored in the file metadata at upload time.

     

Log in to post a comment.