From: SourceForge.net <no...@so...> - 2007-09-22 15:15:57
|
Bugs item #1800209, was opened at 2007-09-22 17:15 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: s3cmd Group: Enhancement request Status: Open Resolution: None Priority: 5 Private: No Submitted By: Kim-Minh KAPLAN (kmkaplan) Assigned to: Nobody/Anonymous (nobody) Summary: [PATCH] Check that PUT of files are correct Initial Comment: Amazon S3 uses the Content-MD5 and ETag HTTP headers to provide for consistency checking. The attached patch makes sure that any upload error is reported. Please include this in your next release. PS: I do not know what kind of error I should raise in case of inconsistency so I just raise a descriptive string. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 |
From: SourceForge.net <no...@so...> - 2007-09-23 01:12:17
|
Bugs item #1800209, was opened at 2007-09-23 03:15 Message generated for change (Comment added) made by ludvigm You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: s3cmd Group: Enhancement request Status: Open Resolution: None Priority: 5 Private: No Submitted By: Kim-Minh KAPLAN (kmkaplan) >Assigned to: Michal Ludvig (ludvigm) Summary: [PATCH] Check that PUT of files are correct Initial Comment: Amazon S3 uses the Content-MD5 and ETag HTTP headers to provide for consistency checking. The attached patch makes sure that any upload error is reported. Please include this in your next release. PS: I do not know what kind of error I should raise in case of inconsistency so I just raise a descriptive string. ---------------------------------------------------------------------- >Comment By: Michal Ludvig (ludvigm) Date: 2007-09-23 13:12 Message: Logged In: YES user_id=344740 Originator: NO Hi, did it ever happen to you that the stored file had a different checksum from the one on your disk? Or is it just a safety check to be sure? Michal ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 |
From: SourceForge.net <no...@so...> - 2007-09-23 10:15:47
|
Bugs item #1800209, was opened at 2007-09-22 17:15 Message generated for change (Comment added) made by kmkaplan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: s3cmd Group: Enhancement request Status: Open Resolution: None Priority: 5 Private: No Submitted By: Kim-Minh KAPLAN (kmkaplan) Assigned to: Michal Ludvig (ludvigm) Summary: [PATCH] Check that PUT of files are correct Initial Comment: Amazon S3 uses the Content-MD5 and ETag HTTP headers to provide for consistency checking. The attached patch makes sure that any upload error is reported. Please include this in your next release. PS: I do not know what kind of error I should raise in case of inconsistency so I just raise a descriptive string. ---------------------------------------------------------------------- >Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-23 12:15 Message: Logged In: YES user_id=24992 Originator: YES I have never had any corruption on Amazon S3 yet. It is sanity check. TCP does not protect against transmission corruption (I have experienced this). ---------------------------------------------------------------------- Comment By: Michal Ludvig (ludvigm) Date: 2007-09-23 03:12 Message: Logged In: YES user_id=344740 Originator: NO Hi, did it ever happen to you that the stored file had a different checksum from the one on your disk? Or is it just a safety check to be sure? Michal ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 |
From: SourceForge.net <no...@so...> - 2007-09-26 00:47:09
|
Bugs item #1800209, was opened at 2007-09-23 03:15 Message generated for change (Comment added) made by ludvigm You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: s3cmd Group: Enhancement request Status: Open >Resolution: Accepted Priority: 5 Private: No Submitted By: Kim-Minh KAPLAN (kmkaplan) Assigned to: Michal Ludvig (ludvigm) Summary: [PATCH] Check that PUT of files are correct Initial Comment: Amazon S3 uses the Content-MD5 and ETag HTTP headers to provide for consistency checking. The attached patch makes sure that any upload error is reported. Please include this in your next release. PS: I do not know what kind of error I should raise in case of inconsistency so I just raise a descriptive string. ---------------------------------------------------------------------- >Comment By: Michal Ludvig (ludvigm) Date: 2007-09-26 12:47 Message: Logged In: YES user_id=344740 Originator: NO I like the idea, but will need a better patch: - this behaviour should be optional (i.e. need a new command line switch for it) because it's not that important in case of HTTPS and some users may opt for a faster operation without precomputing MD5. - the response for PUT-Object contains ETag with MD5 of the uploaded file as Amazon stored it. IMO it's better to compute the MD5 in the upload loop so the file is read from the disk only once. After the whole file is uploaded we'll compare our MD5 with ETag from response and see if they match. If not, re-upload. - with 'sync' command we do some MD5 checksums as well. It may be worth to cache them and reuse for upload. Are you keen to rework the patch or should I do it? Michal ---------------------------------------------------------------------- Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-23 22:15 Message: Logged In: YES user_id=24992 Originator: YES I have never had any corruption on Amazon S3 yet. It is sanity check. TCP does not protect against transmission corruption (I have experienced this). ---------------------------------------------------------------------- Comment By: Michal Ludvig (ludvigm) Date: 2007-09-23 13:12 Message: Logged In: YES user_id=344740 Originator: NO Hi, did it ever happen to you that the stored file had a different checksum from the one on your disk? Or is it just a safety check to be sure? Michal ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 |
From: SourceForge.net <no...@so...> - 2007-09-29 18:06:58
|
Bugs item #1800209, was opened at 2007-09-22 17:15 Message generated for change (Comment added) made by kmkaplan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: s3cmd Group: Enhancement request Status: Open Resolution: Accepted Priority: 5 Private: No Submitted By: Kim-Minh KAPLAN (kmkaplan) Assigned to: Michal Ludvig (ludvigm) Summary: [PATCH] Check that PUT of files are correct Initial Comment: Amazon S3 uses the Content-MD5 and ETag HTTP headers to provide for consistency checking. The attached patch makes sure that any upload error is reported. Please include this in your next release. PS: I do not know what kind of error I should raise in case of inconsistency so I just raise a descriptive string. ---------------------------------------------------------------------- >Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-29 20:06 Message: Logged In: YES user_id=24992 Originator: YES Second try... There is now -5, --no-early-md5. Unless it is specified the MD5 is computed. Note that it also affects sync command. Sync computation is used. File Added: s3cmd-md5.patch ---------------------------------------------------------------------- Comment By: Michal Ludvig (ludvigm) Date: 2007-09-26 02:47 Message: Logged In: YES user_id=344740 Originator: NO I like the idea, but will need a better patch: - this behaviour should be optional (i.e. need a new command line switch for it) because it's not that important in case of HTTPS and some users may opt for a faster operation without precomputing MD5. - the response for PUT-Object contains ETag with MD5 of the uploaded file as Amazon stored it. IMO it's better to compute the MD5 in the upload loop so the file is read from the disk only once. After the whole file is uploaded we'll compare our MD5 with ETag from response and see if they match. If not, re-upload. - with 'sync' command we do some MD5 checksums as well. It may be worth to cache them and reuse for upload. Are you keen to rework the patch or should I do it? Michal ---------------------------------------------------------------------- Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-23 12:15 Message: Logged In: YES user_id=24992 Originator: YES I have never had any corruption on Amazon S3 yet. It is sanity check. TCP does not protect against transmission corruption (I have experienced this). ---------------------------------------------------------------------- Comment By: Michal Ludvig (ludvigm) Date: 2007-09-23 03:12 Message: Logged In: YES user_id=344740 Originator: NO Hi, did it ever happen to you that the stored file had a different checksum from the one on your disk? Or is it just a safety check to be sure? Michal ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 |
From: SourceForge.net <no...@so...> - 2007-09-30 07:43:21
|
Bugs item #1800209, was opened at 2007-09-22 17:15 Message generated for change (Comment added) made by kmkaplan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: s3cmd Group: Enhancement request Status: Open Resolution: Accepted Priority: 5 Private: No Submitted By: Kim-Minh KAPLAN (kmkaplan) Assigned to: Michal Ludvig (ludvigm) Summary: [PATCH] Check that PUT of files are correct Initial Comment: Amazon S3 uses the Content-MD5 and ETag HTTP headers to provide for consistency checking. The attached patch makes sure that any upload error is reported. Please include this in your next release. PS: I do not know what kind of error I should raise in case of inconsistency so I just raise a descriptive string. ---------------------------------------------------------------------- >Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-30 09:43 Message: Logged In: YES user_id=24992 Originator: YES Oops, I forgot a couple of lines. Here is the correct patch. File Added: s3cmd-md5-2.patch ---------------------------------------------------------------------- Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-29 20:06 Message: Logged In: YES user_id=24992 Originator: YES Second try... There is now -5, --no-early-md5. Unless it is specified the MD5 is computed. Note that it also affects sync command. Sync computation is used. File Added: s3cmd-md5.patch ---------------------------------------------------------------------- Comment By: Michal Ludvig (ludvigm) Date: 2007-09-26 02:47 Message: Logged In: YES user_id=344740 Originator: NO I like the idea, but will need a better patch: - this behaviour should be optional (i.e. need a new command line switch for it) because it's not that important in case of HTTPS and some users may opt for a faster operation without precomputing MD5. - the response for PUT-Object contains ETag with MD5 of the uploaded file as Amazon stored it. IMO it's better to compute the MD5 in the upload loop so the file is read from the disk only once. After the whole file is uploaded we'll compare our MD5 with ETag from response and see if they match. If not, re-upload. - with 'sync' command we do some MD5 checksums as well. It may be worth to cache them and reuse for upload. Are you keen to rework the patch or should I do it? Michal ---------------------------------------------------------------------- Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-23 12:15 Message: Logged In: YES user_id=24992 Originator: YES I have never had any corruption on Amazon S3 yet. It is sanity check. TCP does not protect against transmission corruption (I have experienced this). ---------------------------------------------------------------------- Comment By: Michal Ludvig (ludvigm) Date: 2007-09-23 03:12 Message: Logged In: YES user_id=344740 Originator: NO Hi, did it ever happen to you that the stored file had a different checksum from the one on your disk? Or is it just a safety check to be sure? Michal ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 |
From: SourceForge.net <no...@so...> - 2008-04-28 14:39:26
|
Bugs item #1800209, was opened at 2007-09-23 03:15 Message generated for change (Comment added) made by ludvigm You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: s3cmd Group: Enhancement request >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: Kim-Minh KAPLAN (kmkaplan) Assigned to: Michal Ludvig (ludvigm) Summary: [PATCH] Check that PUT of files are correct Initial Comment: Amazon S3 uses the Content-MD5 and ETag HTTP headers to provide for consistency checking. The attached patch makes sure that any upload error is reported. Please include this in your next release. PS: I do not know what kind of error I should raise in case of inconsistency so I just raise a descriptive string. ---------------------------------------------------------------------- >Comment By: Michal Ludvig (ludvigm) Date: 2008-04-29 02:39 Message: Logged In: YES user_id=344740 Originator: NO Sorry it took "a bit longer". s3cmd in SVN computes the md5 sum in the upload loop (in S3.send_file()) which makes it more efficient. ---------------------------------------------------------------------- Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-30 19:43 Message: Logged In: YES user_id=24992 Originator: YES Oops, I forgot a couple of lines. Here is the correct patch. File Added: s3cmd-md5-2.patch ---------------------------------------------------------------------- Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-30 06:06 Message: Logged In: YES user_id=24992 Originator: YES Second try... There is now -5, --no-early-md5. Unless it is specified the MD5 is computed. Note that it also affects sync command. Sync computation is used. File Added: s3cmd-md5.patch ---------------------------------------------------------------------- Comment By: Michal Ludvig (ludvigm) Date: 2007-09-26 12:47 Message: Logged In: YES user_id=344740 Originator: NO I like the idea, but will need a better patch: - this behaviour should be optional (i.e. need a new command line switch for it) because it's not that important in case of HTTPS and some users may opt for a faster operation without precomputing MD5. - the response for PUT-Object contains ETag with MD5 of the uploaded file as Amazon stored it. IMO it's better to compute the MD5 in the upload loop so the file is read from the disk only once. After the whole file is uploaded we'll compare our MD5 with ETag from response and see if they match. If not, re-upload. - with 'sync' command we do some MD5 checksums as well. It may be worth to cache them and reuse for upload. Are you keen to rework the patch or should I do it? Michal ---------------------------------------------------------------------- Comment By: Kim-Minh KAPLAN (kmkaplan) Date: 2007-09-23 22:15 Message: Logged In: YES user_id=24992 Originator: YES I have never had any corruption on Amazon S3 yet. It is sanity check. TCP does not protect against transmission corruption (I have experienced this). ---------------------------------------------------------------------- Comment By: Michal Ludvig (ludvigm) Date: 2007-09-23 13:12 Message: Logged In: YES user_id=344740 Originator: NO Hi, did it ever happen to you that the stored file had a different checksum from the one on your disk? Or is it just a safety check to be sure? Michal ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=887015&aid=1800209&group_id=178907 |