From: Jay M. <J.McCanta@F5.com> - 2013-05-12 23:37:05
|
I'm unsure of the proper protocol for proposing patches for the group. My apologies if this isn't the correct way. We keep the md5 checksum of files in an extended attribute. When running s3cmd with --check-md5, this patch allow one to use the --xattr option and specify the name of the extended attribute to use for local md5sums. If the attribute is not there, a regular check is done, so no changes to existing procedures would be necessary. For us, the drastically improves check time since we are not re-reading every local file recalculating every md5sum. Also there was a typo in s3cmd for the 'help' field of --acl-revoke option definition. It's corrected in this patch. diff -ru s3cmd-1.5.0-alpha3/s3cmd s3cmd-1.5.0-alpha3.jtm/s3cmd --- s3cmd-1.5.0-alpha3/s3cmd 2013-03-10 17:06:33.000000000 -0700 +++ s3cmd-1.5.0-alpha3.jtm/s3cmd 2013-05-12 15:20:44.013584123 -0700 @@ -32,6 +32,13 @@ from logging import debug, info, warning, error from distutils.spawn import find_executable +try: + hasXattr = False + import xattr + hasXattr = True +except: + pass + def output(message): sys.stdout.write(message + "\n") sys.stdout.flush() @@ -1141,7 +1148,7 @@ def _invalidate_on_cf(destination_base_uri): cf = CloudFront(cfg) - default_index_file = None + efault_index_file = None if cfg.invalidate_default_index_on_cf or cfg.invalidate_default_index_root_on_cf: info_response = s3.website_info(destination_base_uri, cfg.bucket_location) if info_response: @@ -1507,9 +1514,9 @@ ret_enc = gpg_encrypt(filename) ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], False) hash = [ - Utils.hash_file_md5(filename), - Utils.hash_file_md5(ret_enc[1]), - Utils.hash_file_md5(ret_dec[1]), + Utils.hash_file_md5(filename, options.xattr), + Utils.hash_file_md5(ret_enc[1], options.xattr), + Utils.hash_file_md5(ret_dec[1], options.xattr), ] os.unlink(filename) os.unlink(ret_enc[1]) @@ -1762,11 +1769,13 @@ optparser.add_option( "--skip-existing", dest="skip_existing", action="store_true", help="Skip over files that exist at the destination (only for [get] and [sync] commands).") optparser.add_option("-r", "--recursive", dest="recursive", action="store_true", help="Recursive upload, download or removal.") optparser.add_option( "--check-md5", dest="check_md5", action="store_true", help="Check MD5 sums when comparing files for [sync]. (default)") + if hasXattr: + optparser.add_option ("--xattr", dest="xattr", action="store", help="If possible, use extended file attribute named (default:%default) instead of calculating it [sync].") optparser.add_option( "--no-check-md5", dest="check_md5", action="store_false", help="Do not check MD5 sums when comparing files for [sync]. Only size will be compared. May significantly speed up transfer but may also miss some changed files.") optparser.add_option("-P", "--acl-public", dest="acl_public", action="store_true", help="Store objects with ACL allowing read for anyone.") optparser.add_option( "--acl-private", dest="acl_public", action="store_false", help="Store objects with default ACL allowing access for you only.") optparser.add_option( "--acl-grant", dest="acl_grants", type="s3acl", action="append", metavar="PERMISSION:EMAIL or USER_CANONICAL_ID", help="Grant stated permission to a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all") - optparser.add_option( "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, wr ite_acp, full_control, all") + optparser.add_option( "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all") optparser.add_option( "--delete-removed", dest="delete_removed", action="store_true", help="Delete remote objects with no corresponding local file [sync]") optparser.add_option( "--no-delete-removed", dest="delete_removed", action="store_false", help="Don't delete remote objects.") Only in s3cmd-1.5.0-alpha3.jtm: s3cmd.orig Jay McCanta | Senior Unix Administrator | F5 Networks, Inc. |
From: <j.m...@en...> - 2013-05-14 20:49:55
|
Is there a typo (accidental one-char deletion) in the code block below? J. Merrill -----Original Message----- From: "Jay McCanta" <J.McCanta@F5.com> Sent: Sunday, May 12, 2013 7:36pm To: "s3t...@li..." <s3t...@li...> Subject: [S3tools-general] Patch for using Xattr for checksums [snip] cf = CloudFront(cfg) - default_index_file = None + efault_index_file = None [snip] |
From: Jay M. <J.McCanta@F5.com> - 2013-05-14 20:54:39
|
My cat helped with that first version (rule #1: blame the firewall, rule #2: blame the cat. It was incomplete and included a typo. The reply to that has the correct stuff. Jay McCanta F5 Networks, Inc. From: j.m...@en... [mailto:j.m...@en...] Sent: Tuesday, May 14, 2013 1:50 PM To: s3t...@li... Subject: Re: [S3tools-general] Patch for using Xattr for checksums Is there a typo (accidental one-char deletion) in the code block below? J. Merrill -----Original Message----- From: "Jay McCanta" <J.McCanta@F5.com<mailto:J.McCanta@F5.com>> Sent: Sunday, May 12, 2013 7:36pm To: "s3t...@li...<mailto:s3t...@li...>" <s3t...@li...<mailto:s3t...@li...>> Subject: [S3tools-general] Patch for using Xattr for checksums [snip] cf = CloudFront(cfg) - default_index_file = None + efault_index_file = None [snip] |
From: Jay M. <J.McCanta@F5.com> - 2013-05-14 21:11:35
|
My apologies for cutting/pasting/posting without proofing. This patch allows s3cmd to utilize md5sums stored in the extended attributes of files. If for any reason, that doesn’t work, the usual md5sum mechanisms are used. This option only has impact if checkmd5 is enabled. The last patch block corrects a typo in the s3cmd ‘--acl-revoke’ option help text. diff -ru s3cmd-1.5.0-alpha3.orig/S3/Utils.py s3cmd-1.5.0-alpha3/S3/Utils.py --- s3cmd-1.5.0-alpha3.orig/S3/Utils.py 2013-03-10 15:35:09.000000000 -0700 +++ s3cmd-1.5.0-alpha3/S3/Utils.py 2013-05-13 14:13:22.083721474 -0700 @@ -35,6 +35,12 @@ import elementtree.ElementTree as ET from xml.parsers.expat import ExpatError +try: + import xattr + hasXattr = True +except: + hasXattr = False + __all__ = [] def parseNodes(nodes): ## WARNING: Ignores text nodes from mixed xml/text. @@ -226,7 +232,14 @@ return mktmpsomething(prefix, randchars, createfunc) __all__.append("mktmpfile") -def hash_file_md5(filename): +def hash_file_md5(filename, xattrName=None): + if hasXattr and xattrName is not None: + try: + md5sum = xattr.get(filename, xattrName) + return md5sum + except: + pass + h = md5() f = open(filename, "rb") while True: diff -ru s3cmd-1.5.0-alpha3.orig/s3cmd s3cmd-1.5.0-alpha3/s3cmd --- s3cmd-1.5.0-alpha3.orig/s3cmd 2013-03-10 17:06:33.000000000 -0700 +++ s3cmd-1.5.0-alpha3/s3cmd 2013-05-13 14:10:33.927609005 -0700 @@ -32,6 +32,12 @@ from logging import debug, info, warning, error from distutils.spawn import find_executable +try: + import xattr + hasXattr = True +except: + hasXattr = False + def output(message): sys.stdout.write(message + "\n") sys.stdout.flush() @@ -1507,9 +1513,9 @@ ret_enc = gpg_encrypt(filename) ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], False) hash = [ - Utils.hash_file_md5(filename), - Utils.hash_file_md5(ret_enc[1]), - Utils.hash_file_md5(ret_dec[1]), + Utils.hash_file_md5(filename, options.xattr), + Utils.hash_file_md5(ret_enc[1], options.xattr), + Utils.hash_file_md5(ret_dec[1], options.xattr), ] os.unlink(filename) os.unlink(ret_enc[1]) @@ -1762,11 +1768,13 @@ optparser.add_option( "--skip-existing", dest="skip_existing", action="store_true", help="Skip over files that exist at the destination (only for [get] and [sync] commands).") optparser.add_option("-r", "--recursive", dest="recursive", action="store_true", help="Recursive upload, download or removal.") optparser.add_option( "--check-md5", dest="check_md5", action="store_true", help="Check MD5 sums when comparing files for [sync]. (default)") + if hasXattr: + optparser.add_option ("--xattr", dest="xattr", action="store", help="If possible, use extended file attribute named (default:%default) instead of calculating it [sync].") optparser.add_option( "--no-check-md5", dest="check_md5", action="store_false", help="Do not check MD5 sums when comparing files for [sync]. Only size will be compared. May significantly speed up transfer but may also miss some changed files.") optparser.add_option("-P", "--acl-public", dest="acl_public", action="store_true", help="Store objects with ACL allowing read for anyone.") optparser.add_option( "--acl-private", dest="acl_public", action="store_false", help="Store objects with default ACL allowing access for you only.") optparser.add_option( "--acl-grant", dest="acl_grants", type="s3acl", action="append", metavar="PERMISSION:EMAIL or USER_CANONICAL_ID", help="Grant stated permission to a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all") - optparser.add_option( "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, wr ite_acp, full_control, all") + optparser.add_option( "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all") optparser.add_option( "--delete-removed", dest="delete_removed", action="store_true", help="Delete remote objects with no corresponding local file [sync]") optparser.add_option( "--no-delete-removed", dest="delete_removed", action="store_false", help="Don't delete remote objects.") |
From: Matt D. <ma...@do...> - 2013-05-15 04:32:47
|
Jay - thanks for the patch. I've made some stylistic changes, as well as simplifies testing for xattr module import success, passes the result into the Config object, and ensures the correct namespace is used. Also added a test to the test suite, and the entry on the manpage. I gave you credit in the changelog of course. I've pushed this to my xattr branch, and into my merge branch as well, and have filed a pull request for same. I've tested this with the test suite and with manual tests to be sure it works. https://github.com/mdomsch/s3cmd/tree/merge Thanks much for the idea! On Tue, May 14, 2013 at 4:11 PM, Jay McCanta <J.M...@f5...> wrote: > My apologies for cutting/pasting/posting without proofing. > > This patch allows s3cmd to utilize md5sums stored in the extended > attributes of files. If for any reason, that doesn’t work, the usual > md5sum mechanisms are used. This option only has impact if check md5 is > enabled. > The last patch block corrects a typo in the s3cmd ‘--acl-revoke’ option > help text. > > diff -ru s3cmd-1.5.0-alpha3.orig/S3/Utils.py s3cmd-1.5.0-alpha3/S3/Utils.py > --- s3cmd-1.5.0-alpha3.orig/S3/Utils.py 2013-03-10 15:35:09.000000000 -0700 > +++ s3cmd-1.5.0-alpha3/S3/Utils.py 2013-05-13 14:13:22.083721474 -0700 > @@ -35,6 +35,12 @@ > import elementtree.ElementTree as ET > from xml.parsers.expat import ExpatError > > +try: > + import xattr > + hasXattr = True > +except: > + hasXattr = False > + > __all__ = [] > def parseNodes(nodes): > ## WARNING: Ignores text nodes from mixed xml/text. > @@ -226,7 +232,14 @@ > return mktmpsomething(prefix, randchars, createfunc) > __all__.append("mktmpfile") > > -def hash_file_md5(filename): > +def hash_file_md5(filename, xattrName=None): > + if hasXattr and xattrName is not None: > + try: > + md5sum = xattr.get(filename, xattrName) > + return md5sum > + except: > + pass > + > h = md5() > f = open(filename, "rb") > while True: > diff -ru s3cmd-1.5.0-alpha3.orig/s3cmd s3cmd-1.5.0-alpha3/s3cmd > --- s3cmd-1.5.0-alpha3.orig/s3cmd 2013-03-10 17:06:33.000000000 -0700 > +++ s3cmd-1.5.0-alpha3/s3cmd 2013-05-13 14:10:33.927609005 -0700 > @@ -32,6 +32,12 @@ > from logging import debug, info, warning, error > from distutils.spawn import find_executable > > +try: > + import xattr > + hasXattr = True > +except: > + hasXattr = False > + > def output(message): > sys.stdout.write(message + "\n") > sys.stdout.flush() > @@ -1507,9 +1513,9 @@ > ret_enc = gpg_encrypt(filename) > ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], > False) > hash = [ > - Utils.hash_file_md5(filename), > - Utils.hash_file_md5(ret_enc[1]), > - Utils.hash_file_md5(ret_dec[1]), > + Utils.hash_file_md5(filename, options.xattr), > + Utils.hash_file_md5(ret_enc[1], > options.xattr), > + Utils.hash_file_md5(ret_dec[1], > options.xattr), > ] > os.unlink(filename) > os.unlink(ret_enc[1]) > @@ -1762,11 +1768,13 @@ > optparser.add_option( "--skip-existing", dest="skip_existing", > action="store_true", help="Skip over files that exist at the destination > (only for [get] and [sync] commands).") > optparser.add_option("-r", "--recursive", dest="recursive", > action="store_true", help="Recursive upload, download or removal.") > optparser.add_option( "--check-md5", dest="check_md5", > action="store_true", help="Check MD5 sums when comparing files for [sync]. > (default)") > + if hasXattr: > + optparser.add_option ("--xattr", dest="xattr", action="store", > help="If possible, use extended file attribute named (default:%default) > instead of calculating it [sync].") > optparser.add_option( "--no-check-md5", dest="check_md5", > action="store_false", help="Do not check MD5 sums when comparing files for > [sync]. Only size will be compared. May significantly speed up transfer but > may also miss some changed files.") > optparser.add_option("-P", "--acl-public", dest="acl_public", > action="store_true", help="Store objects with ACL allowing read for > anyone.") > optparser.add_option( "--acl-private", dest="acl_public", > action="store_false", help="Store objects with default ACL allowing access > for you only.") > optparser.add_option( "--acl-grant", dest="acl_grants", > type="s3acl", action="append", metavar="PERMISSION:EMAIL or > USER_CANONICAL_ID", help="Grant stated permission to a given amazon user. > Permission is one of: read, write, read_acp, write_acp, full_control, all") > - optparser.add_option( "--acl-revoke", dest="acl_revokes", > type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", > help="Revoke stated permission for a given amazon user. Permission is one > of: read, write, read_acp, wr ite_acp, full_control, all") > + optparser.add_option( "--acl-revoke", dest="acl_revokes", > type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", > help="Revoke stated permission for a given amazon user. Permission is one > of: read, write, read_acp, write_acp, full_control, all") > > optparser.add_option( "--delete-removed", dest="delete_removed", > action="store_true", help="Delete remote objects with no corresponding > local file [sync]") > optparser.add_option( "--no-delete-removed", > dest="delete_removed", action="store_false", help="Don't delete remote > objects.") > > > > ------------------------------------------------------------------------------ > AlienVault Unified Security Management (USM) platform delivers complete > security visibility with the essential security capabilities. Easily and > efficiently configure, manage, and operate all of your security controls > from a single console and one unified framework. Download a free trial. > http://p.sf.net/sfu/alienvault_d2d > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > > |
From: Matt D. <ma...@do...> - 2013-05-23 13:03:16
|
In talking with Michal last night, he wants a different approach to the xattr work, based on using xattrs as alternate storage for what's currently stored externally in HashCache pickles, but offering the same kind of protection (e.g. detection of file changes which cause the md5 to be recalculated). If someone wants to take a stab at that approach, great. Until then, the xattr patches from this thread have been reverted on the upstream master branch. Use the --cache-file=FILE option for similar functionality. Thanks, Matt On Tue, May 14, 2013 at 11:32 PM, Matt Domsch <ma...@do...> wrote: > Jay - thanks for the patch. I've made some stylistic changes, as well as > simplifies testing for xattr > module import success, passes the result into the Config object, > and ensures the correct namespace is used. Also added a test to the > test > suite, and the entry on the manpage. > > I gave you credit in the changelog of course. > > I've pushed this to my xattr branch, and into my merge branch as well, and > have filed a pull request for same. I've tested this with the test suite > and with manual tests to be sure it works. > https://github.com/mdomsch/s3cmd/tree/merge > > Thanks much for the idea! > > > > On Tue, May 14, 2013 at 4:11 PM, Jay McCanta <J.M...@f5...> wrote: > >> My apologies for cutting/pasting/posting without proofing. >> >> This patch allows s3cmd to utilize md5sums stored in the extended >> attributes of files. If for any reason, that doesn’t work, the usual >> md5sum mechanisms are used. This option only has impact if check md5 is >> enabled. >> The last patch block corrects a typo in the s3cmd ‘--acl-revoke’ option >> help text. >> >> diff -ru s3cmd-1.5.0-alpha3.orig/S3/Utils.py >> s3cmd-1.5.0-alpha3/S3/Utils.py >> --- s3cmd-1.5.0-alpha3.orig/S3/Utils.py 2013-03-10 15:35:09.000000000 >> -0700 >> +++ s3cmd-1.5.0-alpha3/S3/Utils.py 2013-05-13 14:13:22.083721474 >> -0700 >> @@ -35,6 +35,12 @@ >> import elementtree.ElementTree as ET >> from xml.parsers.expat import ExpatError >> >> +try: >> + import xattr >> + hasXattr = True >> +except: >> + hasXattr = False >> + >> __all__ = [] >> def parseNodes(nodes): >> ## WARNING: Ignores text nodes from mixed xml/text. >> @@ -226,7 +232,14 @@ >> return mktmpsomething(prefix, randchars, createfunc) >> __all__.append("mktmpfile") >> >> -def hash_file_md5(filename): >> +def hash_file_md5(filename, xattrName=None): >> + if hasXattr and xattrName is not None: >> + try: >> + md5sum = xattr.get(filename, xattrName) >> + return md5sum >> + except: >> + pass >> + >> h = md5() >> f = open(filename, "rb") >> while True: >> diff -ru s3cmd-1.5.0-alpha3.orig/s3cmd s3cmd-1.5.0-alpha3/s3cmd >> --- s3cmd-1.5.0-alpha3.orig/s3cmd 2013-03-10 17:06:33.000000000 >> -0700 >> +++ s3cmd-1.5.0-alpha3/s3cmd 2013-05-13 14:10:33.927609005 -0700 >> @@ -32,6 +32,12 @@ >> from logging import debug, info, warning, error >> from distutils.spawn import find_executable >> >> +try: >> + import xattr >> + hasXattr = True >> +except: >> + hasXattr = False >> + >> def output(message): >> sys.stdout.write(message + "\n") >> sys.stdout.flush() >> @@ -1507,9 +1513,9 @@ >> ret_enc = gpg_encrypt(filename) >> ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], >> False) >> hash = [ >> - Utils.hash_file_md5(filename), >> - Utils.hash_file_md5(ret_enc[1]), >> - Utils.hash_file_md5(ret_dec[1]), >> + Utils.hash_file_md5(filename, options.xattr), >> + Utils.hash_file_md5(ret_enc[1], >> options.xattr), >> + Utils.hash_file_md5(ret_dec[1], >> options.xattr), >> ] >> os.unlink(filename) >> os.unlink(ret_enc[1]) >> @@ -1762,11 +1768,13 @@ >> optparser.add_option( "--skip-existing", dest="skip_existing", >> action="store_true", help="Skip over files that exist at the destination >> (only for [get] and [sync] commands).") >> optparser.add_option("-r", "--recursive", dest="recursive", >> action="store_true", help="Recursive upload, download or removal.") >> optparser.add_option( "--check-md5", dest="check_md5", >> action="store_true", help="Check MD5 sums when comparing files for [sync]. >> (default)") >> + if hasXattr: >> + optparser.add_option ("--xattr", dest="xattr", action="store", >> help="If possible, use extended file attribute named (default:%default) >> instead of calculating it [sync].") >> optparser.add_option( "--no-check-md5", dest="check_md5", >> action="store_false", help="Do not check MD5 sums when comparing files for >> [sync]. Only size will be compared. May significantly speed up transfer but >> may also miss some changed files.") >> optparser.add_option("-P", "--acl-public", dest="acl_public", >> action="store_true", help="Store objects with ACL allowing read for >> anyone.") >> optparser.add_option( "--acl-private", dest="acl_public", >> action="store_false", help="Store objects with default ACL allowing access >> for you only.") >> optparser.add_option( "--acl-grant", dest="acl_grants", >> type="s3acl", action="append", metavar="PERMISSION:EMAIL or >> USER_CANONICAL_ID", help="Grant stated permission to a given amazon user. >> Permission is one of: read, write, read_acp, write_acp, full_control, all") >> - optparser.add_option( "--acl-revoke", dest="acl_revokes", >> type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", >> help="Revoke stated permission for a given amazon user. Permission is one >> of: read, write, read_acp, wr ite_acp, full_control, all") >> + optparser.add_option( "--acl-revoke", dest="acl_revokes", >> type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", >> help="Revoke stated permission for a given amazon user. Permission is one >> of: read, write, read_acp, write_acp, full_control, all") >> >> optparser.add_option( "--delete-removed", >> dest="delete_removed", action="store_true", help="Delete remote objects >> with no corresponding local file [sync]") >> optparser.add_option( "--no-delete-removed", >> dest="delete_removed", action="store_false", help="Don't delete remote >> objects.") >> >> >> >> ------------------------------------------------------------------------------ >> AlienVault Unified Security Management (USM) platform delivers complete >> security visibility with the essential security capabilities. Easily and >> efficiently configure, manage, and operate all of your security controls >> from a single console and one unified framework. Download a free trial. >> http://p.sf.net/sfu/alienvault_d2d >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> >> > |
From: Matt D. <Matt_Domsch@Dell.com> - 2013-12-06 17:34:19
|
On Sun, May 12, 2013 at 06:36:54PM -0500, Jay McCanta wrote: > I’m unsure of the proper protocol for proposing patches for the > group. My apologies if this isn’t the correct way. Thanks for the patch Jay. Patches by email is fine, though pull requests filed in github to pull from a specific bug or feature branch in your own github-forked repository also works quite well, and makes it harder to forget over time as the pull request remains open until merged or closed manually. :-) > We keep the md5 checksum of files in an extended attribute. When > running s3cmd with --check-md5, this patch allow one to use the > --xattr option and specify the name of the extended attribute to use > for local md5sums. If the attribute is not there, a regular check > is done, so no changes to existing procedures would be necessary. > For us, the drastically improves check time since we are not > re-reading every local file recalculating every md5sum. Cool idea. What are you using to put such md5sums into the extended attribute already? > Also there was a typo in s3cmd for the ‘help’ field of --acl-revoke > option definition. It’s corrected in this patch. Ideally that would be split into a second patch. > diff -ru s3cmd-1.5.0-alpha3/s3cmd s3cmd-1.5.0-alpha3.jtm/s3cmd > --- s3cmd-1.5.0-alpha3/s3cmd 2013-03-10 17:06:33.000000000 -0700 > +++ s3cmd-1.5.0-alpha3.jtm/s3cmd 2013-05-12 15:20:44.013584123 -0700 > @@ -32,6 +32,13 @@ > from logging import debug, info, warning, error > from distutils.spawn import find_executable > > +try: > + hasXattr = False > + import xattr > + hasXattr = True > +except: > + pass > + This can be shortened to: try: import xattr except: pass ... if 'xattr' in sys.modules.keys(): optparser.add_option ("--xattr", dest="xattr", action="store", help="If possible, use extended file attribute named (default:%default) instead of calculating it [sync].") This should be default=False, action="store_true". That way we know it exists later. > def output(message): > sys.stdout.write(message + "\n") > sys.stdout.flush() > @@ -1141,7 +1148,7 @@ > > def _invalidate_on_cf(destination_base_uri): > cf = CloudFront(cfg) > - default_index_file = None > + efault_index_file = None keystroke typo? > if cfg.invalidate_default_index_on_cf or cfg.invalidate_default_index_root_on_cf: > info_response = s3.website_info(destination_base_uri, cfg.bucket_location) > if info_response: > @@ -1507,9 +1514,9 @@ > ret_enc = gpg_encrypt(filename) > ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], False) > hash = [ > - Utils.hash_file_md5(filename), > - Utils.hash_file_md5(ret_enc[1]), > - Utils.hash_file_md5(ret_dec[1]), > + Utils.hash_file_md5(filename, options.xattr), > + Utils.hash_file_md5(ret_enc[1], options.xattr), > + Utils.hash_file_md5(ret_dec[1], options.xattr), What's the corresponding change to S3/Utils.py please? Given that this should also be a Config option, not just a command line option, please stick it into the Config object, to be retrieved in Utils.py. Thanks, Matt -- Matt Domsch Technology Strategist Dell | Office of the CTO |
From: Matt D. <ma...@do...> - 2013-12-06 17:44:33
|
Please ignore. It seems a mail server at work had a historical hiccup and is resending mail sent in May. Thanks, Matt On Sun, May 12, 2013 at 7:19 PM, Matt Domsch <Mat...@de...> wrote: > On Sun, May 12, 2013 at 06:36:54PM -0500, Jay McCanta wrote: > > I’m unsure of the proper protocol for proposing patches for the > > group. My apologies if this isn’t the correct way. > > Thanks for the patch Jay. > > Patches by email is fine, though pull requests filed in github to pull > from a specific bug or feature branch in your own github-forked > repository also works quite well, and makes it harder to forget over > time as the pull request remains open until merged or closed > manually. :-) > > > We keep the md5 checksum of files in an extended attribute. When > > running s3cmd with --check-md5, this patch allow one to use the > > --xattr option and specify the name of the extended attribute to use > > for local md5sums. If the attribute is not there, a regular check > > is done, so no changes to existing procedures would be necessary. > > For us, the drastically improves check time since we are not > > re-reading every local file recalculating every md5sum. > > > Cool idea. What are you using to put such md5sums into the extended > attribute already? > > > Also there was a typo in s3cmd for the ‘help’ field of --acl-revoke > > option definition. It’s corrected in this patch. > > Ideally that would be split into a second patch. > > > diff -ru s3cmd-1.5.0-alpha3/s3cmd s3cmd-1.5.0-alpha3.jtm/s3cmd > > --- s3cmd-1.5.0-alpha3/s3cmd 2013-03-10 17:06:33.000000000 -0700 > > +++ s3cmd-1.5.0-alpha3.jtm/s3cmd 2013-05-12 15:20:44.013584123 > -0700 > > @@ -32,6 +32,13 @@ > > from logging import debug, info, warning, error > > from distutils.spawn import find_executable > > > > +try: > > + hasXattr = False > > + import xattr > > + hasXattr = True > > +except: > > + pass > > + > > This can be shortened to: > > try: import xattr > except: pass > ... > if 'xattr' in sys.modules.keys(): > optparser.add_option ("--xattr", dest="xattr", action="store", > help="If possible, use extended file attribute named (default:%default) > instead of calculating it [sync].") > > This should be default=False, action="store_true". That way we know > it exists later. > > > > def output(message): > > sys.stdout.write(message + "\n") > > sys.stdout.flush() > > @@ -1141,7 +1148,7 @@ > > > > def _invalidate_on_cf(destination_base_uri): > > cf = CloudFront(cfg) > > - default_index_file = None > > + efault_index_file = None > > keystroke typo? > > > if cfg.invalidate_default_index_on_cf or > cfg.invalidate_default_index_root_on_cf: > > info_response = s3.website_info(destination_base_uri, > cfg.bucket_location) > > if info_response: > > @@ -1507,9 +1514,9 @@ > > ret_enc = gpg_encrypt(filename) > > ret_dec = gpg_decrypt(ret_enc[1], ret_enc[2], > False) > > hash = [ > > - Utils.hash_file_md5(filename), > > - Utils.hash_file_md5(ret_enc[1]), > > - Utils.hash_file_md5(ret_dec[1]), > > + Utils.hash_file_md5(filename, > options.xattr), > > + Utils.hash_file_md5(ret_enc[1], > options.xattr), > > + Utils.hash_file_md5(ret_dec[1], > options.xattr), > > What's the corresponding change to S3/Utils.py please? > > Given that this should also be a Config option, not just a command > line option, please stick it into the Config object, to be retrieved > in Utils.py. > > Thanks, > Matt > > > -- > Matt Domsch > Technology Strategist > Dell | Office of the CTO > > > ------------------------------------------------------------------------------ > Sponsored by Intel(R) XDK > Develop, test and display web and hybrid apps with a single code base. > Download it for free now! > > http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > > |