From: Jeremy H. <jh...@gm...> - 2010-04-01 15:25:39
|
Thanks Rob - I'm already splitting my 8GB file into 1GB chunks, but I'll try making the chunks smaller! I've found S3 to be more unreliable than I'd hoped...(mostly due to connection errors on Amazon's side) Any code that uses S3 has to have really good retry logic! (our web application stores a lot of data on S3, so I know from experience!) Rob Scala wrote: > I learned that this just happens from time to time. The suggested > solution was to break the file into pieces so that failures don't set > you back hours. Now I break my 4GB file into 0.5 GB chunks. I use a > script to split the file and send the chunks, retrying on failure. It's > been flawless for weeks now. > > > On Thu, 2010-04-01 at 09:27 -0400, Jeremy Haile wrote: > >> I'm using s3cmd 0.9.9 and trying to do a put to S3 of several 1GB files, >> but I continually get connection reset by peer errors and I'm not sure >> what to do. Here are my logs. Any ideas? >> >> INFO: Compiling list of local files... >> INFO: Applying --exclude/--include >> INFO: Summary: 18 local files to upload >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_000', please wait... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_001', please >> wait...IINFO: Sending file '/var/backups/myfile-s3-M-W-F_002', please >> wait... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.01) >> WARNING: Waiting 3 sec... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.05) >> WARNING: Waiting 6 sec... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] >> Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=0.25) >> WARNING: Waiting 9 sec...WARNING: Upload failed: >> /backups/M-W-F/myfile-s3-M-W-F_002 ([Errno 104] Connection reset by >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_003 ([INFO: >> Sending file '/var/backups/myfile-s3-M-W-F_002', please wait... >> WARNING: Upload failed: /backups/M-W-F/myfile-s3-M-W-F_002 ([Errno 104] >> Connection reset by WARNING: Upload failed: >> /backups/M-W-F/myfile-s3-M-W-F_003 ([Errno 104] Connection reset by peer) >> WARNING: Retrying on lower speed (throttle=6.25) >> WARNING: Waiting 15 sec... >> INFO: Sending file '/var/backups/myfile-s3-M-W-F_003', please wait... >> >> >> ------------------------------------------------------------------------------ >> Download Intel® Parallel Studio Eval >> Try the new software tools for yourself. Speed compiling, find bugs >> proactively, and fine-tune applications for parallel performance. >> See why Intel Parallel Studio got high marks during beta. >> http://p.sf.net/sfu/intel-sw-dev >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> |