From: Jim P. <ji...@gm...> - 2011-01-04 05:00:01
|
> From: Jobe Bittman [mailto:jo...@op...] > Sent: Thursday, November 04, 2010 10:01 PM > To: s3t...@li... > Subject: [S3tools-general] Patches for updating headers without re-uploading > > I modified s3cmd 1.0.0-rc1 to allow replacing headers such as Cache-Control > without having to reupload the file. I'm updating about 400,000 files with > this right now. I used the information here > http://doc.s3.amazonaws.com/proposals/copy.html and proxy sniffing > BucketExplorer to figure it out. Can someone take a look and check it in? > I've seen alot of people looking for this functionality on the net. Jobe, Thanks! The patch works flawlessly on 1.0.0-rc1. Good Job! -Jim P. |
From: Jim P. <ji...@gm...> - 2011-01-09 06:28:11
|
On Mon, Jan 3, 2011 at 23:59, Jim Popovitch <ji...@gm...> wrote: >> From: Jobe Bittman [mailto:jo...@op...] >> Sent: Thursday, November 04, 2010 10:01 PM >> To: s3t...@li... >> Subject: [S3tools-general] Patches for updating headers without re-uploading >> >> I modified s3cmd 1.0.0-rc1 to allow replacing headers such as Cache-Control >> without having to reupload the file. I'm updating about 400,000 files with >> this right now. I used the information here >> http://doc.s3.amazonaws.com/proposals/copy.html and proxy sniffing >> BucketExplorer to figure it out. Can someone take a look and check it in? >> I've seen alot of people looking for this functionality on the net. > > Jobe, > > Thanks! The patch works flawlessly on 1.0.0-rc1. Good Job! *update* Works wee, but resets mime-type to binary/octet-stream -Jim P. |
From: Jobe B. <jo...@op...> - 2011-01-09 06:59:45
|
Were you able to see where to fix that? On Sat, Jan 8, 2011 at 10:28 PM, Jim Popovitch <ji...@gm...> wrote: > On Mon, Jan 3, 2011 at 23:59, Jim Popovitch <ji...@gm...> wrote: > >> From: Jobe Bittman [mailto:jo...@op...] > >> Sent: Thursday, November 04, 2010 10:01 PM > >> To: s3t...@li... > >> Subject: [S3tools-general] Patches for updating headers without > re-uploading > >> > >> I modified s3cmd 1.0.0-rc1 to allow replacing headers such as > Cache-Control > >> without having to reupload the file. I'm updating about 400,000 files > with > >> this right now. I used the information here > >> http://doc.s3.amazonaws.com/proposals/copy.html and proxy sniffing > >> BucketExplorer to figure it out. Can someone take a look and check it > in? > >> I've seen alot of people looking for this functionality on the net. > > > > Jobe, > > > > Thanks! The patch works flawlessly on 1.0.0-rc1. Good Job! > > *update* > > Works wee, but resets mime-type to binary/octet-stream > > -Jim P. > > > ------------------------------------------------------------------------------ > Gaining the trust of online customers is vital for the success of any > company > that requires sensitive data to be transmitted over the Web. Learn how to > best implement a security strategy that keeps consumers' information secure > and instills the confidence they need to proceed with transactions. > http://p.sf.net/sfu/oracle-sfdevnl > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > |
From: Jim P. <ji...@gm...> - 2011-01-09 07:04:26
|
On Sun, Jan 9, 2011 at 01:35, Jobe Bittman <jo...@op...> wrote: > Were you able to see where to fix that? I looked at the patch, but haven't yet tested preserving the existing header data. I'm presuming that the s3cmd headers get stripped too. Will test some tomorrow. -Jim P. > > On Sat, Jan 8, 2011 at 10:28 PM, Jim Popovitch <ji...@gm...> wrote: >> >> On Mon, Jan 3, 2011 at 23:59, Jim Popovitch <ji...@gm...> wrote: >> >> From: Jobe Bittman [mailto:jo...@op...] >> >> Sent: Thursday, November 04, 2010 10:01 PM >> >> To: s3t...@li... >> >> Subject: [S3tools-general] Patches for updating headers without >> >> re-uploading >> >> >> >> I modified s3cmd 1.0.0-rc1 to allow replacing headers such as >> >> Cache-Control >> >> without having to reupload the file. I'm updating about 400,000 files >> >> with >> >> this right now. I used the information here >> >> http://doc.s3.amazonaws.com/proposals/copy.html and proxy sniffing >> >> BucketExplorer to figure it out. Can someone take a look and check it >> >> in? >> >> I've seen alot of people looking for this functionality on the net. >> > >> > Jobe, >> > >> > Thanks! The patch works flawlessly on 1.0.0-rc1. Good Job! >> >> *update* >> >> Works wee, but resets mime-type to binary/octet-stream >> >> -Jim P. >> >> >> ------------------------------------------------------------------------------ >> Gaining the trust of online customers is vital for the success of any >> company >> that requires sensitive data to be transmitted over the Web. Learn how >> to >> best implement a security strategy that keeps consumers' information >> secure >> and instills the confidence they need to proceed with transactions. >> http://p.sf.net/sfu/oracle-sfdevnl >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > ------------------------------------------------------------------------------ > Gaining the trust of online customers is vital for the success of any > company > that requires sensitive data to be transmitted over the Web. Learn how to > best implement a security strategy that keeps consumers' information secure > and instills the confidence they need to proceed with transactions. > http://p.sf.net/sfu/oracle-sfdevnl > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > > |
From: Jim P. <ji...@gm...> - 2011-01-09 22:45:00
|
On Sun, Jan 9, 2011 at 01:35, Jobe Bittman <jo...@op...> wrote: > Were you able to see where to fix that? Hi Jobe, Here's the patch output of what I changed and tested. It's inline here so that can be verified first, rather than having a file that ppl immediate download and apply. ;-) -Jim P. $ diff -r -u S3.py.orig S3.py --- S3.py.orig 2011-01-09 17:02:16.000000000 -0500 +++ S3.py 2011-01-09 17:40:26.000000000 -0500 @@ -305,8 +305,15 @@ headers["x-amz-acl"] = "public-read" if self.config.reduced_redundancy: headers["x-amz-storage-class"] = "REDUCED_REDUNDANCY" - # if extra_headers: - # headers.update(extra_headers) + if extra_headers: + headers.update(extra_headers) + content_type = None + if self.config.guess_mime_type: + content_type = mimetypes.guess_type(src_uri.object())[0] + if not content_type: + content_type = self.config.default_mime_type + debug("Content-Type set to '%s'" % content_type) + headers["content-type"] = content_type request = self.create_request("OBJECT_PUT", uri = dst_uri, headers = headers) response = self.send_request(request) return response @@ -325,6 +332,13 @@ headers["x-amz-storage-class"] = "REDUCED_REDUNDANCY" if extra_headers: headers.update(extra_headers) + content_type = None + if self.config.guess_mime_type: + content_type = mimetypes.guess_type(src_uri.object())[0] + if not content_type: + content_type = self.config.default_mime_type + debug("Content-Type set to '%s'" % content_type) + headers["content-type"] = content_type request = self.create_request("OBJECT_PUT", uri = dst_uri, headers = headers) response = self.send_request(request) return response -Jim P. |
From: Michal L. <ml...@lo...> - 2011-01-10 05:15:32
|
On 01/10/2011 11:44 AM, Jim Popovitch wrote: > On Sun, Jan 9, 2011 at 01:35, Jobe Bittman<jo...@op...> wrote: >> Were you able to see where to fix that? > Hi Jobe, > > Here's the patch output of what I changed and tested. It's inline > here so that can be verified first, rather than having a file that ppl > immediate download and apply. ;-) > Hi Jim, I'm afraid that the only reliable way to copy all the attributes across (i.e. mime type, acl, s3cmd stored attribs, etc) is to run HEAD request on the object to learn all the current attribs and then COPY with them included. Your "guess_type()" approach may work in some cases but you may still lose all the other attribs along the way. Michal |
From: Jim P. <ji...@gm...> - 2011-01-10 05:33:11
|
On Mon, Jan 10, 2011 at 00:17, Michal Ludvig <ml...@lo...> wrote: > Hi Jim, > > I'm afraid that the only reliable way to copy all the attributes across > (i.e. mime type, acl, s3cmd stored attribs, etc) is to run HEAD request > on the object to learn all the current attribs and then COPY with them > included. > > Your "guess_type()" approach may work in some cases but you may still > lose all the other attribs along the way. Hi Michal, I originally started down the HEAD request path. Since my interest was only in updating the TTL I only coded what was necessary to "preserve" mime-type since that was always defaulting to binary/octet-stream. I'll make a second pass at implementing the HEAD request approach, as you stated, it is the best approach. -Jim P. |