From: Rob S. <rs...@bu...> - 2010-01-28 22:19:23
|
I've been using s3cmd for a while, and I haven't needed to use this mailing list until now. It's been a great tool. Unfortunately I've been having a lot of "Connection reset by peer" problems recently, and I don't know where the problem lies. I upload a 4.5 GB file every day, and it's gotten so bad that only one out of five attempts works, and each attempt takes hours. The "Connection reset by peer" failure seems to occur randomly (beginning, middle, or end of transfer). Could it be Amazon's problem, or a network problem? I'm using s3cmd version 0.9.9.91 Thanks Rob -- Rob Scala Buyers Edge, LLC 351 North Frontage Road, Suite A210 New London, CT 06320 Phone: 860-701-0538 x27 Fax: 860-701-0576 |
From: Jeremy H. <jh...@gm...> - 2010-04-01 13:28:08
|
I'm using s3cmd 0.9.9 and trying to do a put to S3 of several 1GB files, but I continually get connection reset by peer errors and I'm not sure what to do. Here are my logs. Any ideas? INFO: Compiling list of local files... INFO: Applying --exclude/--include INFO: Summary: 18 local files to upload INFO: Sending file '/var/backups/myfile-s3-M-W-F_000', please wait... INFO: Sending file '/var/backups/myfile-s3-M-W-F_001', please wait...IINFO: Sending file '/var/backups/myfile-s3-M-W-F_002', please wait... INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] Connection reset by peer) WARNING: Retrying on lower speed (throttle=0.01) WARNING: Waiting 3 sec... INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] Connection reset by peer) WARNING: Retrying on lower speed (throttle=0.05) WARNING: Waiting 6 sec... INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] Connection reset by peer) WARNING: Retrying on lower speed (throttle=0.25) WARNING: Waiting 9 sec...WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_002 ([Errno 104] Connection reset by WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([INFO: Sending file '/var/backups/myfile-s3-M-W-F_002', please wait... WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_002 ([Errno 104] Connection reset by WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] Connection reset by peer) WARNING: Retrying on lower speed (throttle=6.25) WARNING: Waiting 15 sec... INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... |
From: Rob S. <rs...@bu...> - 2010-04-01 14:46:47
|
I learned that this just happens from time to time. The suggested solution was to break the file into pieces so that failures don't set you back hours. Now I break my 4GB file into 0.5 GB chunks. I use a script to split the file and send the chunks, retrying on failure. It's been flawless for weeks now. On Thu, 2010-04-01 at 09:27 -0400, Jeremy Haile wrote: > I'm using s3cmd 0.9.9 and trying to do a put to S3 of several 1GB files, > but I continually get connection reset by peer errors and I'm not sure > what to do. Here are my logs. Any ideas? > > INFO: Compiling list of local files... > INFO: Applying --exclude/--include > INFO: Summary: 18 local files to upload > INFO: Sending file '/var/backups/myfile-s3-M-W-F_000', please wait... > INFO: Sending file '/var/backups/myfile-s3-M-W-F_001', please > wait...IINFO: Sending file '/var/backups/myfile-s3-M-W-F_002', please > wait... > INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... > WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] > Connection reset by peer) > WARNING: Retrying on lower speed (throttle=0.01) > WARNING: Waiting 3 sec... > INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... > WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] > Connection reset by peer) > WARNING: Retrying on lower speed (throttle=0.05) > WARNING: Waiting 6 sec... > INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... > WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] > Connection reset by peer) > WARNING: Retrying on lower speed (throttle=0.25) > WARNING: Waiting 9 sec...WARNING: Upload failed: > /backups/M-W-F/myfile-s3-M-W-F_002 ([Errno 104] Connection reset by > WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([INFO: > Sending file '/var/backups/myfile-s3-M-W-F_002', please wait... > WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_002 ([Errno 104] > Connection reset by WARNING: Upload failed: > /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] Connection reset by peer) > WARNING: Retrying on lower speed (throttle=6.25) > WARNING: Waiting 15 sec... > INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general -- Rob Scala Buyers Edge, LLC 351 North Frontage Road, Suite A210 New London, CT 06320 Phone: 860-701-0538 x27 Fax: 860-701-0576 |
From: Jeremy H. <jh...@gm...> - 2010-04-01 15:25:39
|
Thanks Rob - I'm already splitting my 8GB file into 1GB chunks, but I'll try making the chunks smaller! I've found S3 to be more unreliable than I'd hoped...(mostly due to connection errors on Amazon's side) Any code that uses S3 has to have really good retry logic! (our web application stores a lot of data on S3, so I know from experience!) Rob Scala wrote: > I learned that this just happens from time to time. The suggested > solution was to break the file into pieces so that failures don't set > you back hours. Now I break my 4GB file into 0.5 GB chunks. I use a > script to split the file and send the chunks, retrying on failure. It's > been flawless for weeks now. > > > On Thu, 2010-04-01 at 09:27 -0400, Jeremy Haile wrote: > >> I'm using s3cmd 0.9.9 and trying to do a put to S3 of several 1GB files, >> but I continually get connection reset by peer errors and I'm not sure >> what to do. Here are my logs. Any ideas? >> >> INFO: Compiling list of local files... >> INFO: Applying --exclude/--include >> INFO: Summary: 18 local files to upload >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_000', please wait... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_001', please >> wait...IINFO: Sending file '/var/backups/myfile-s3-M-W-F_002', please >> wait... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.01) >> WARNING: Waiting 3 sec... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.05) >> WARNING: Waiting 6 sec... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.25) >> WARNING: Waiting 9 sec...WARNING: Upload failed: >> /backups/M-W-F/myfile-s3-M-W-F_002 ([Errno 104] Connection reset by >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([INFO: >> Sending file '/var/backups/myfile-s3-M-W-F_002', please wait... >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_002 ([Errno 104] >> Connection reset by WARNING: Upload failed: >> /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=6.25) >> WARNING: Waiting 15 sec... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... >> >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> |
From: Michal L. <ml...@lo...> - 2010-01-28 22:54:38
|
Hi Rob, > Unfortunately I've been having a lot of "Connection reset by peer" > problems recently, and I don't know where the problem lies. > [...] > Could it be Amazon's problem, or a network problem? On some days this happens to me as well, even on files as small as 100MB. After a couple of hours the problem usually disappears and s3cmd runs all right for days. I don't know where the problem is, I blame Amazon though ;-) I suggest you split the file into smaller chunks before uploading, say 200MB each. With that s3cmd will only have to reupload the chunks that failed instead of the whole 4.5GB file. Michal |
From: Rob S. <rs...@bu...> - 2010-01-29 23:34:39
|
Thank you both - Jeff and Michal. I guess I'll have to write a more robust archiving script. But I'll be ready when my database grows to over 5 gig :) On Fri, 2010-01-29 at 11:55 +1300, Michal Ludvig wrote: > Hi Rob, > > > Unfortunately I've been having a lot of "Connection reset by peer" > > problems recently, and I don't know where the problem lies. > > [...] > > Could it be Amazon's problem, or a network problem? > > On some days this happens to me as well, even on files as small as > 100MB. After a couple of hours the problem usually disappears and s3cmd > runs all right for days. I don't know where the problem is, I blame > Amazon though ;-) > > I suggest you split the file into smaller chunks before uploading, say > 200MB each. With that s3cmd will only have to reupload the chunks that > failed instead of the whole 4.5GB file. > > Michal > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general |
From: Jeff R. <jr...@op...> - 2010-01-28 22:59:35
|
Rob Scala wrote: > I've been using s3cmd for a while, and I haven't needed to use this > mailing list until now. It's been a great tool. > > Unfortunately I've been having a lot of "Connection reset by peer" > problems recently, and I don't know where the problem lies. > > I upload a 4.5 GB file every day, and it's gotten so bad that only one > out of five attempts works, and each attempt takes hours. The > "Connection reset by peer" failure seems to occur randomly (beginning, > middle, or end of transfer). > > Could it be Amazon's problem, or a network problem? > > I'm using s3cmd version 0.9.9.91 > > Thanks > > Rob > http://www.mail-archive.com/s3t...@li.../msg00197.html Jeff |