[S3tools-commits] [ s3tools-Bugs-3306113 ] s3cmd timesouts while mv/renaming large file in a bucket

[SourceForge.net <no...@so...>]

Bugs item #3306113, was opened at 2011-05-23 10:36
Message generated for change (Tracker Item Submitted) made by sbhura
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=887015&aid=3306113&group_id=178907

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: s3cmd
Group: Malfunction
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Shrenik (sbhura)
Assigned to: Nobody/Anonymous (nobody)
Summary: s3cmd timesouts while mv/renaming large file in a bucket

Initial Comment:
Was trying to move an approx 200 MB tar.bz2 archive within a bucket. Basically trying to rename it when this error consistently shows up. At times it works, at others, it doesn't.

WARNING: Retrying failed request: /archives/OT.tar.bz2.3 (The read operation timed out)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /archives/OT.tar.bz2.3 (The read operation timed out)
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /archives/OT.tar.bz2.2 (The read operation timed out)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /archives/OT.tar.bz2.2 (The read operation timed out)
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /archives/OT.tar.bz2.2 (The read operation timed out)
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /archives/OT.tar.bz2.2 (The read operation timed out)
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /archives/OT.tar.bz2.2 (The read operation timed out)
WARNING: Waiting 15 sec...

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please report the following lines to:
   s3t...@li...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Problem: S3RequestError: Request failed for: /archives/OT.tar.bz2.2
S3cmd:   1.0.0

Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 2006, in <module>
    main()
  File "/usr/bin/s3cmd", line 1950, in main
    cmd_func(args)
  File "/usr/bin/s3cmd", line 618, in cmd_mv
    subcmd_cp_mv(args, s3.object_move, "move", "File %(src)s moved to %(dst)s")
  File "/usr/bin/s3cmd", line 607, in subcmd_cp_mv
    response = process_fce(src_uri, dst_uri, extra_headers)
  File "/usr/lib/python2.6/site-packages/S3/S3.py", line 315, in object_move
    response_copy = self.object_copy(src_uri, dst_uri, extra_headers)
  File "/usr/lib/python2.6/site-packages/S3/S3.py", line 311, in object_copy
    response = self.send_request(request)
  File "/usr/lib/python2.6/site-packages/S3/S3.py", line 487, in send_request
    return self.send_request(request, body, retries - 1)
  File "/usr/lib/python2.6/site-packages/S3/S3.py", line 487, in send_request
    return self.send_request(request, body, retries - 1)
  File "/usr/lib/python2.6/site-packages/S3/S3.py", line 487, in send_request
    return self.send_request(request, body, retries - 1)
  File "/usr/lib/python2.6/site-packages/S3/S3.py", line 487, in send_request
    return self.send_request(request, body, retries - 1)
  File "/usr/lib/python2.6/site-packages/S3/S3.py", line 487, in send_request
    return self.send_request(request, body, retries - 1)
  File "/usr/lib/python2.6/site-packages/S3/S3.py", line 489, in send_request
    raise S3RequestError("Request failed for: %s" % resource['uri'])
S3RequestError: Request failed for: /archives/OT.tar.bz2.2

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
    Please report the above lines to:
   s3t...@li...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


This is the piece of code that I am executing to get this done. There is no doubt that the code is working fine as I have tried it several times for a small archive -

#!/bin/bash
S3CMD="s3cmd"
DRY=""
S3BUCKET="<bucket URI>"
MAXARCH=3
BASEDIR=/backups_staging/

${S3CMD} del ${DRY} ${S3BUCKET}/archives/OT.tar.bz2.${MAXARCH}

   for (( i=${MAXARCH}; i>1; i-- )); do
                let j=i-1
                ${S3CMD} mv ${DRY} ${S3BUCKET}/archives/OT.tar.bz2.$j ${S3BUCKET}/archives/OT.tar.bz2.$i
   done;

${S3CMD} put ${DRY} ${BASEDIR}/OT.tar.bz2 ${S3BUCKET}/archives/OT.tar.bz2.1


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=887015&aid=3306113&group_id=178907