S3 tools

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4813295
By: dsbcpas

Here is more detail regarding this error. I don't alway receive any traceback,
only 3 of the following out of 11 include any traceback.
Any idea how to better trace this down?

Example.

ERROR: S3 error: 500 (Internal Server Error): InternalError
Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 767, in <module>
    cmd_func(args)
  File "/usr/bin/s3cmd", line 429, in cmd_sync
    response = s3.object_put_uri(src, uri, attr_header)
  File "/usr/bin/S3/S3.py", line 225, in object_put_uri
    return self.object_put(filename, uri.bucket(), uri.object(), extra_headers)
  File "/usr/bin/S3/S3.py", line 201, in object_put
    response = self.send_file(request, file)
  File "/usr/bin/S3/S3.py", line 377, in send_file
    conn.send(data)
  File "/usr/local/lib/python2.5/httplib.py", line 707, in send
    self.sock.sendall(str)
  File "/usr/local/lib/python2.5/httplib.py", line 1104, in send
    return self._ssl.write(stuff)
socket.error: (32, 'Broken pipe')


______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4813330
By: dsbcpas

PS of the 11 errors, two were 400s with no traceback:
ERROR: S3 error: 400 (Bad Request): RequestTimeout

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4814471
By: dsbcpas

Michal. 
S3cmd has been syncing various parts of our directory structure for over a day
now. It appears that she routinely fails on directories with a large number
of files, over 1,400.  One directory had 7k files for which s3cmd sync managed
1400 or so before dumping. Sometimes she exits after processing only, for example,
42 of 51 files and in another instance 14 of 136. It appears that the large
the number of files in the directory, the more like she is to exit without
completion.

Debug and Verbose parameters are not very helpful on account I can't log their
output for analysis. I am not logging any activity until it completes the sync
command which seems strange and then just the last successful upload, omitting
the file it failed working on. Is there a way of capturing the file s3cmd is
processing when it exits to see if I can duplicate the result with the
put function?



Scott 

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4814476
By: dsbcpas

Michal

Another thought, could the failing to sync a directory, no error given, be a
result of URL size limitation???

Scott

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4815462
By: ludvigm

Hi, can you test s3cmd 0.9.7-pre1 from http://tmp.logix.cz/s3cmd-0.9.7-pre1.tar.gz
please?

It's got two improvements:
1) will retry failed upload on a lower speed which is more likely to succeed.
2) won't exit on failed upload, rather the failing file is skipped over.

Let me know if it helps.

(and BTW s3cmd is "he" ;-)

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4815668
By: dsbcpas

s3cmd 0.9.7-pre1 Report: Still failing with no indication of file in process
that caused problem. Upload "appears" faster and much more stable, but still
dropping out from time to time with error 500.
(Dated line is my script trap for exit 1) Hope this info helps.
Scott

INFO: Sending file '/home/shared/ATB/WORK.TXT.wpd', please wait...
INFO: Sent 583 bytes (100 % of 583)
ERROR: S3 error: 500 (Internal Server Error): InternalError
Tue Mar 4 09:05:01 EST 2008  S3SYNCUPLOAD FAILED SYNCING /home/shared/ATB INTO
S3 AT synchomeshared:ATB
INFO: ConfigParser: Reading file '/root/.s3cfg'
INFO: Processing request, please wait...
INFO: ConfigParser: Reading file '/root/.s3cfg'
INFO: Processing request, please wait...
INFO: ConfigParser: Reading file '/root/.s3cfg'
INFO: Processing request, please wait...
ERROR: S3 error: 500 (Internal Server Error): InternalError
Tue Mar 4 09:05:09 EST 2008  S3SYNCUPLOAD FAILED SYNCING /home/shared/COMPUTAX
INTO S3 AT synchomeshared:COMPUTAX
INFO: ConfigParser: Reading file '/root/.s3cfg'
INFO: Processing request, please wait...
INFO: ConfigParser: Reading file '/root/.s3cfg'
INFO: Processing request, please wait...
INFO: ConfigParser: Reading file '/root/.s3cfg'
INFO: Processing request, please wait...
INFO: Sending file '/home/shared/LIB/RPG/TEST.EXE', please wait...



______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4816971
By: ludvigm

Can you create a new bucket in Europe and rerun your test? That would make sure
that you're talking to a completely different host. I still believe the Internal
Error is a problem on Amazon side, not s3cmd. Thanks Michal

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4817921
By: dsbcpas

Sure! How do I do that? Set up a separate EU account? Scott

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4817939
By: dsbcpas

Question - Could it relate to my configuration? I compiled Python2.5 due to
2.4 rpm conflict with 2.3 rpm dependencies. May be I should compile 2.4 instead?
Also, I changed the first line of s3cmd to point to the compiled version:
#!/usr/local/bin/python
Thank you for working with me on this.
Scott

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4817956
By: dsbcpas

Just caught the last trace back when she bombed with no other error messages.
Appears to be a socket error.

Scott

INFO: Sent 7785 bytes (100 % of 7785)
Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 785, in <module>
    cmd_func(args)
  File "/usr/bin/s3cmd", line 437, in cmd_sync
    response = s3.object_put_uri(src, uri, attr_header)
  File "/usr/bin/S3/S3.py", line 227, in object_put_uri
    return self.object_put(filename, uri.bucket(), uri.object(), extra_headers)
  File "/usr/bin/S3/S3.py", line 204, in object_put
    response = self.send_file(request, file)
  File "/usr/bin/S3/S3.py", line 403, in send_file
    http_response = conn.getresponse()
  File "/usr/local/lib/python2.5/httplib.py", line 924, in getresponse
    response.begin()
  File "/usr/local/lib/python2.5/httplib.py", line 385, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python2.5/httplib.py", line 343, in _read_status
    line = self.fp.readline()
  File "/usr/local/lib/python2.5/httplib.py", line 1043, in readline
    s = self._read()
  File "/usr/local/lib/python2.5/httplib.py", line 999, in _read
    buf = self._ssl.read(self._bufsize)
socket.error: (104, 'Connection reset by peer')
Wed Mar 5 08:59:16 EST 2008  S3SYNCUPLOAD FAILED SYNCING /home/shared/emailarchive
INTO S3 AT synchomeshared:emailarchive


______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4818808
By: ludvigm

You don't need a separate EU account, simply create a bucket with:
s3cmd mb --bucket-location=EU s3://your-new-EU-bucket-name

The EU storage is slightly more expensive ($0.18 instead of $0.15 for US-based
buckets), I hope it won't ruin you :) See http://aws.amazon.com/s3 for pricing
details.

Michal

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4818831
By: ludvigm

It's unlikely to be related to your python version as the problems are intermittent.
If there was something wrong with your python you'd probably observe some problems
on every run. FYI I regularly run s3cmd with both Python 2.4 and Python 2.5
and both work fine for me.

Michal

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4824668
By: nobody

Michal, this may be a clue, --bucket-location=EU, does not work, message - ERROR:
S3 error: 403 (Forbidden): SignatureDoesNotMatch


______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4825886
By: dsbcpas

My error, one problem solved, bucket name must be in lower case!
I will test EU site and report back.
Scott

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4825930
By: dsbcpas

Michal, you are correct, uploaded to EU NO PROBLEM! Appears to be something
with US site. On to other matters!
Thank you for the EU suggestion. I'll continue to experiment with both. On to
new problems.
Scott

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4830844
By: ludvigm

I'm glad to hear it's sorted. It might actually be some anomaly in your US bucket
as most other users don't get Internal Errors in US either. To get a cheaper
service from S3 you may want to create a new bucket in US and try again with
that. Or if you're happy with EU go ahead and use that ;-)

______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865

Read and respond to this message at: 
https://sourceforge.net/forum/message.php?msg_id=4841067
By: dsbcpas

Hi Michal,
I haven't yet deleted my US bucket and tried to start over yet. It's on my list
to do list, but I have 20 gig to move. I think I will wait until after you convert
to curl to see what happens. I am syncing daily so any thing missed is generally
picked up the next day. It just does not like high volumes.

Lastest 500 error completely different then prior errors came up in del mode
is as follows showing httplib.BadStatusLine.

Thank you for your help.
Scott

ERROR: S3 error: 500 (Internal Server Error): InternalError
Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 785, in <module>
    cmd_func(args)
  File "/usr/bin/s3cmd", line 267, in cmd_object_del
    response = s3.object_delete_uri(uri)
  File "/usr/bin/S3/S3.py", line 240, in object_delete_uri
    return self.object_delete(uri.bucket(), uri.object())
  File "/usr/bin/S3/S3.py", line 221, in object_delete
    response = self.send_request(request)
  File "/usr/bin/S3/S3.py", line 344, in send_request
    http_response = conn.getresponse()
  File "/usr/local/lib/python2.5/httplib.py", line 924, in getresponse
    response.begin()
  File "/usr/local/lib/python2.5/httplib.py", line 385, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python2.5/httplib.py", line 349, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine


______________________________________________________________________
You are receiving this email because you elected to monitor this forum.
To stop monitoring this forum, login to SourceForge.net and visit: 
https://sourceforge.net/forum/unmonitor.php?forum_id=618865