Menu

#14 KeyError: 'etag' with --acl-private flag

Malfunction
closed-fixed
s3cmd (120)
5
2008-06-30
2008-06-11
Burt
No

s3cmd -f -v -H --acl-private put rbImageV1.* s3://rightscale_XXX/reviewboard/images/

INFO: Sent 10485760 bytes (100 % of 10485760)
Traceback (most recent call last):
File "/usr/bin/s3cmd", line 935, in <module>
cmd_func(args)
File "/usr/bin/s3cmd", line 194, in cmd_object_put
response = s3.object_put_uri(real_filename, uri_final, extra_headers)
File "/usr/lib/python2.5/site-packages/S3/S3.py", line 195, in object_put_uri
return self.object_put(filename, uri.bucket(), uri.object(), extra_headers)
File "/usr/lib/python2.5/site-packages/S3/S3.py", line 175, in object_put
response = self.send_file(request, file)
File "/usr/lib/python2.5/site-packages/S3/S3.py", line 384, in send_file
debug("MD5 sums: computed=%s, received=%s" % (md5_computed, response["headers"]["etag"]))
KeyError: 'etag'

cat /home/reviewboard/.s3cfg
[default]
access_key = XXX
acl_public = False
bucket_location = US
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
encrypt = False
force = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = XXX
guess_mime_type = False
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
preserve_attrs = True
proxy_host =
proxy_port = 0
recv_chunk = 4096
secret_key = XXX
send_chunk = 4096
use_https = True
verbosity = WARNING

s3cmd version 0.9.7

The command works fine w/o the --acl-private flag

Discussion

  • Michal Ludvig

    Michal Ludvig - 2008-06-11

    Logged In: YES
    user_id=344740
    Originator: NO

    Is it reproducible? Works fine for me with your command line and s3cmd-0.9.8-rc1 ...

    $ s3cmd -f -v -H --acl-private put kvm-*.tar.gz s3://...-test/
    [...]
    INFO: Sent 3698688 bytes (99 % of 3706582)
    INFO: Sent 3702784 bytes (99 % of 3706582)
    INFO: Sent 3706582 bytes (100 % of 3706582)
    File 'kvm-64.tar.gz' stored as s3://...-test/kvm-64.tar.gz (3706582 bytes in 39.2 seconds, 92.31 kB/s) [3 of 3]

     
  • Burt

    Burt - 2008-06-11

    Logged In: YES
    user_id=2114223
    Originator: YES

    I was using 0.9.7 - maybe you fixed it since (I see that it was in a debug statement). It happened when I tried it again.

     
  • Michal Ludvig

    Michal Ludvig - 2008-06-11

    Logged In: YES
    user_id=344740
    Originator: NO

    The only difference between 0.9.7 and 0.9.8-rc1 is a fixed compatibility with Python 2.4 and --exclude option.

    Anyway I did a test with 0.9.7 and it worked fine as well. Can you observe any pattern on failures? E.g. some specific buckets only? Some specific files only (size, name, ...)? What Python version have you got?

    Can you send me the output with --debug (email if you prefer).

     
  • Hermann

    Hermann - 2008-06-30

    Logged In: YES
    user_id=778872
    Originator: NO

    I've had the same issue when doing a sync, for some reason Amazon randomly omits the etag header and the script throws an exception and bombs out. I've fixed it by adding an if clause right before the debug function call

    if 'etag' not in response['headers']:
    response['headers']['etag'] = ''

    As far as I can see it's random and the script just retries to send the file again and it always works for me.

     
  • Michal Ludvig

    Michal Ludvig - 2008-06-30

    Logged In: YES
    user_id=344740
    Originator: NO

    Thanks guys, it actually happened to me on a large sync from my US server to S3 as well. Interestingly it never kicked in when syncing from New Zealand, perhaps the connection was slower and gave S3 enough time to compute MD5 :-))

    I have applied hermzz's fix to SVN. Will be included in the next release.

     
  • Michal Ludvig

    Michal Ludvig - 2008-06-30
    • assigned_to: nobody --> ludvigm
    • status: open --> closed-fixed
     

Log in to post a comment.