From: Alex E. <pow...@po...> - 2009-03-05 19:58:39
|
Hi! I've large file (~5.5 GB - /home dir backup), and tried several times to upload it to S3 using s3cmd 0.9.8.4. Each time it took several hours and finally failed. I've just tried to upgrade to 0.9.9, it also failed but with another error message (see at bottom of this message). All other ~1600 files uploaded without troubles using 'sync' command, while this file failed both using 'sync' and 'put' commands. Also I should note this file is largest from all ~1600 files - next large file size is ~5.2GB and it uploaded normally. # time md5sum home.full.20090302_031507.tar.gpg d3722332b583f2d2e5f53413ee14fe57 home.full.20090302_031507.tar.gpg real 2m10.812s user 0m25.490s sys 0m8.570s # ls -l home.full.20090302_031507.tar.gpg -rw-r--r-- 1 backup backup 5586801747 2009-03-02 03:57 home.full.20090302_031507.tar.gpg # time s3cmd put home.full.20090302_031507.tar.gpg s3://backup.ourcompany WARNING: MD5 Sums don't match! WARNING: MD5 Sums don't match! WARNING: MD5 Sums don't match! WARNING: MD5 Sums don't match! ERROR: Upload of 'home.full.20090302_031507.tar.gpg' failed too many times. Skipping that file. real 303m38.849s user 7m56.840s sys 1m8.640s # time /etc/cron.daily/backup_to_s3 Compiling list of local files... Retrieving list of remote files... Found 1663 local files, 1632 remote files Verifying checksums... Summary: 8 local files to upload, 0 remote files to delete WARNING: MD5 Sums don't match! WARNING: MD5 Sums don't match! WARNING: MD5 Sums don't match! WARNING: MD5 Sums don't match! ERROR: home.full.20090302_031507.tar.gpg: upload failed too many times. Skipping that file. File 'mysql.incr.20090304_030008.tar.gpg' stored as s3://backup.ourcompany/mysql.incr.20090304_030008.tar.gpg (80084541 bytes in 171.0 seconds, 457.26 kB/s) [2 of 8] File 'home.incr.20090304_030010.tar.gpg' stored as s3://backup.ourcompany/home.incr.20090304_030010.tar.gpg (14791469 bytes in 1.9 seconds, 7.47 MB/s) [3 of 8] File 'mysql.incr.20090304_030010.tar.gpg' stored as s3://backup.ourcompany/mysql.incr.20090304_030010.tar.gpg (1554 bytes in 0.0 seconds, 9.43 MB/s) [4 of 8] File 'svn.incr.20090304_030010.tar.gpg' stored as s3://backup.ourcompany/svn.incr.20090304_030010.tar.gpg (3553 bytes in 0.0 seconds, 22.17 MB/s) [5 of 8] File 'config.incr.20090304_071720.tar.gpg' stored as s3://backup.ourcompany/config.incr.20090304_071720.tar.gpg (32459 bytes in 0.3 seconds, 125.75 kB/s) [6 of 8] File 'home.incr.20090304_071720.tar.gpg' stored as s3://backup.ourcompany/home.incr.20090304_071720.tar.gpg (2705201 bytes in 1.0 seconds, 2.60 MB/s) [7 of 8] File 'mysql.incr.20090304_071720.tar.gpg' stored as s3://backup.ourcompany/mysql.incr.20090304_071720.tar.gpg (979 bytes in 0.0 seconds, 6.06 MB/s) [8 of 8] Done. Uploaded 97619756 bytes in 34733.5 seconds, 2.74 kB/s real 599m43.248s user 10m35.850s sys 2m7.890s 0.9.9 fail in different way: # time s3cmd put home.full.20090302_031507.tar.gpg s3://backup.ourcompany home.full.20090302_031507.tar.gpg -> s3://backup.ourcompany/ [1 of 1] 5586801747 of 5586801747 100% in 11194s 487.39 kB/s done ERROR: S3 error: 400 (MalformedXML): The XML you provided was not well-formed or did not validate against our published schema real 186m34.334s user 2m27.220s sys 0m28.430s -- WBR, Alex. |
From: Fr. R. B. <frr...@ba...> - 2009-03-05 20:18:51
|
According to Amazon http://docs.amazonwebservices.com/AmazonS3/2006-03-01/gsg/ "Each object can hold up to 5 GB of data." You may have got lucky with the 5. 2 file and it finally failed on the 5.5 Thanks, Fr. Robert Bower Batushka's Web Hosting & Design frr...@ba... 574.850.7982 http://www.batushkas.com http://www.batushkas.net Alex Efros wrote: > Hi! > > I've large file (~5.5 GB - /home dir backup), and tried several times to > upload it to S3 using s3cmd 0.9.8.4. Each time it took several hours and > finally failed. I've just tried to upgrade to 0.9.9, it also failed but > with another error message (see at bottom of this message). > > All other ~1600 files uploaded without troubles using 'sync' command, > while this file failed both using 'sync' and 'put' commands. > Also I should note this file is largest from all ~1600 files - next > large file size is ~5.2GB and it uploaded normally. > > # time md5sum home.full.20090302_031507.tar.gpg > d3722332b583f2d2e5f53413ee14fe57 home.full.20090302_031507.tar.gpg > > real 2m10.812s > user 0m25.490s > sys 0m8.570s > > # ls -l home.full.20090302_031507.tar.gpg > -rw-r--r-- 1 backup backup 5586801747 2009-03-02 03:57 home.full.20090302_031507.tar.gpg > > # time s3cmd put home.full.20090302_031507.tar.gpg s3://backup.ourcompany > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > ERROR: Upload of 'home.full.20090302_031507.tar.gpg' failed too many times. Skipping that file. > > real 303m38.849s > user 7m56.840s > sys 1m8.640s > > # time /etc/cron.daily/backup_to_s3 > Compiling list of local files... > Retrieving list of remote files... > Found 1663 local files, 1632 remote files > Verifying checksums... > Summary: 8 local files to upload, 0 remote files to delete > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > ERROR: home.full.20090302_031507.tar.gpg: upload failed too many times. Skipping that file. > File 'mysql.incr.20090304_030008.tar.gpg' stored as s3://backup.ourcompany/mysql.incr.20090304_030008.tar.gpg (80084541 bytes in 171.0 seconds, 457.26 kB/s) [2 of 8] > File 'home.incr.20090304_030010.tar.gpg' stored as s3://backup.ourcompany/home.incr.20090304_030010.tar.gpg (14791469 bytes in 1.9 seconds, 7.47 MB/s) [3 of 8] > File 'mysql.incr.20090304_030010.tar.gpg' stored as s3://backup.ourcompany/mysql.incr.20090304_030010.tar.gpg (1554 bytes in 0.0 seconds, 9.43 MB/s) [4 of 8] > File 'svn.incr.20090304_030010.tar.gpg' stored as s3://backup.ourcompany/svn.incr.20090304_030010.tar.gpg (3553 bytes in 0.0 seconds, 22.17 MB/s) [5 of 8] > File 'config.incr.20090304_071720.tar.gpg' stored as s3://backup.ourcompany/config.incr.20090304_071720.tar.gpg (32459 bytes in 0.3 seconds, 125.75 kB/s) [6 of 8] > File 'home.incr.20090304_071720.tar.gpg' stored as s3://backup.ourcompany/home.incr.20090304_071720.tar.gpg (2705201 bytes in 1.0 seconds, 2.60 MB/s) [7 of 8] > File 'mysql.incr.20090304_071720.tar.gpg' stored as s3://backup.ourcompany/mysql.incr.20090304_071720.tar.gpg (979 bytes in 0.0 seconds, 6.06 MB/s) [8 of 8] > Done. Uploaded 97619756 bytes in 34733.5 seconds, 2.74 kB/s > > real 599m43.248s > user 10m35.850s > sys 2m7.890s > > > 0.9.9 fail in different way: > > # time s3cmd put home.full.20090302_031507.tar.gpg s3://backup.ourcompany > home.full.20090302_031507.tar.gpg -> s3://backup.ourcompany/ [1 of 1] > 5586801747 of 5586801747 100% in 11194s 487.39 kB/s done > ERROR: S3 error: 400 (MalformedXML): The XML you provided was not well-formed or did not validate against our published schema > > real 186m34.334s > user 2m27.220s > sys 0m28.430s > > > |
From: Alex E. <pow...@po...> - 2009-03-05 22:59:02
|
Hi! On Thu, Mar 05, 2009 at 03:18:39PM -0500, Fr. Robert Bower wrote: > According to Amazon > http://docs.amazonwebservices.com/AmazonS3/2006-03-01/gsg/ > > "Each object can hold up to 5 GB of data." Yeah, you right. I wasn't expected such random limitation on S3. Thanks! > You may have got lucky with the 5. 2 file and it finally failed on the 5.5 5.2 was estimate at a glance. That file size is actually 5359396204 bytes = 4.99 GB, i.e. <5 GB. Probably it's nice to have more informative warning from s3cmd in this case. -- WBR, Alex. |
From: Michal L. <mi...@lo...> - 2009-03-05 23:30:28
|
That's right, the maximum upload size is 5GB. As a short term solution I'm going to implement a check for the file size prior to upload to prevent these useless uploads. And yes, the "WARNING: MD5 Sums don't match!" message is misleading. That's already fixed in s3cmd 0.9.9 though. Michal Fr. Robert Bower wrote: > According to Amazon > http://docs.amazonwebservices.com/AmazonS3/2006-03-01/gsg/ > > "Each object can hold up to 5 GB of data." > > You may have got lucky with the 5. 2 file and it finally failed on the 5.5 > > Thanks, > > Fr. Robert Bower > Batushka's Web Hosting & Design > frr...@ba... > 574.850.7982 > http://www.batushkas.com > http://www.batushkas.net > > > > Alex Efros wrote: >> Hi! >> >> I've large file (~5.5 GB - /home dir backup), and tried several times to >> upload it to S3 using s3cmd 0.9.8.4. Each time it took several hours and >> finally failed. I've just tried to upgrade to 0.9.9, it also failed but >> with another error message (see at bottom of this message). >> >> All other ~1600 files uploaded without troubles using 'sync' command, >> while this file failed both using 'sync' and 'put' commands. >> Also I should note this file is largest from all ~1600 files - next >> large file size is ~5.2GB and it uploaded normally. >> >> # time md5sum home.full.20090302_031507.tar.gpg >> d3722332b583f2d2e5f53413ee14fe57 home.full.20090302_031507.tar.gpg >> >> real 2m10.812s >> user 0m25.490s >> sys 0m8.570s >> >> # ls -l home.full.20090302_031507.tar.gpg >> -rw-r--r-- 1 backup backup 5586801747 2009-03-02 03:57 home.full.20090302_031507.tar.gpg >> >> # time s3cmd put home.full.20090302_031507.tar.gpg s3://backup.ourcompany >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> ERROR: Upload of 'home.full.20090302_031507.tar.gpg' failed too many times. Skipping that file. >> >> real 303m38.849s >> user 7m56.840s >> sys 1m8.640s >> >> # time /etc/cron.daily/backup_to_s3 >> Compiling list of local files... >> Retrieving list of remote files... >> Found 1663 local files, 1632 remote files >> Verifying checksums... >> Summary: 8 local files to upload, 0 remote files to delete >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> ERROR: home.full.20090302_031507.tar.gpg: upload failed too many times. Skipping that file. >> File 'mysql.incr.20090304_030008.tar.gpg' stored as s3://backup.ourcompany/mysql.incr.20090304_030008.tar.gpg (80084541 bytes in 171.0 seconds, 457.26 kB/s) [2 of 8] >> File 'home.incr.20090304_030010.tar.gpg' stored as s3://backup.ourcompany/home.incr.20090304_030010.tar.gpg (14791469 bytes in 1.9 seconds, 7.47 MB/s) [3 of 8] >> File 'mysql.incr.20090304_030010.tar.gpg' stored as s3://backup.ourcompany/mysql.incr.20090304_030010.tar.gpg (1554 bytes in 0.0 seconds, 9.43 MB/s) [4 of 8] >> File 'svn.incr.20090304_030010.tar.gpg' stored as s3://backup.ourcompany/svn.incr.20090304_030010.tar.gpg (3553 bytes in 0.0 seconds, 22.17 MB/s) [5 of 8] >> File 'config.incr.20090304_071720.tar.gpg' stored as s3://backup.ourcompany/config.incr.20090304_071720.tar.gpg (32459 bytes in 0.3 seconds, 125.75 kB/s) [6 of 8] >> File 'home.incr.20090304_071720.tar.gpg' stored as s3://backup.ourcompany/home.incr.20090304_071720.tar.gpg (2705201 bytes in 1.0 seconds, 2.60 MB/s) [7 of 8] >> File 'mysql.incr.20090304_071720.tar.gpg' stored as s3://backup.ourcompany/mysql.incr.20090304_071720.tar.gpg (979 bytes in 0.0 seconds, 6.06 MB/s) [8 of 8] >> Done. Uploaded 97619756 bytes in 34733.5 seconds, 2.74 kB/s >> >> real 599m43.248s >> user 10m35.850s >> sys 2m7.890s >> >> >> 0.9.9 fail in different way: >> >> # time s3cmd put home.full.20090302_031507.tar.gpg s3://backup.ourcompany >> home.full.20090302_031507.tar.gpg -> s3://backup.ourcompany/ [1 of 1] >> 5586801747 of 5586801747 100% in 11194s 487.39 kB/s done >> ERROR: S3 error: 400 (MalformedXML): The XML you provided was not well-formed or did not validate against our published schema >> >> real 186m34.334s >> user 2m27.220s >> sys 0m28.430s >> >> >> > > ------------------------------------------------------------------------------ > Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA > -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise > -Strategies to boost innovation and cut costs with open source participation > -Receive a $600 discount off the registration fee with the source code: SFAD > http://p.sf.net/sfu/XcvMzF8H > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general > |
From: Philip J. <pj...@ne...> - 2009-03-05 23:12:37
|
Hi Alex, IIRC the max file size is 5GB for S3, isn't it? Cheers, Philip Am 05.03.2009 um 20:58 schrieb Alex Efros: > Hi! > > I've large file (~5.5 GB - /home dir backup), and tried several > times to > upload it to S3 using s3cmd 0.9.8.4. Each time it took several hours > and > finally failed. I've just tried to upgrade to 0.9.9, it also failed > but > with another error message (see at bottom of this message). > > All other ~1600 files uploaded without troubles using 'sync' > command, > while this file failed both using 'sync' and 'put' commands. > Also I should note this file is largest from all ~1600 files - next > large file size is ~5.2GB and it uploaded normally. > > # time md5sum home.full.20090302_031507.tar.gpg > d3722332b583f2d2e5f53413ee14fe57 home.full.20090302_031507.tar.gpg > > real 2m10.812s > user 0m25.490s > sys 0m8.570s > > # ls -l home.full.20090302_031507.tar.gpg > -rw-r--r-- 1 backup backup 5586801747 2009-03-02 03:57 home.full. > 20090302_031507.tar.gpg > > # time s3cmd put home.full.20090302_031507.tar.gpg s3:// > backup.ourcompany > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > ERROR: Upload of 'home.full.20090302_031507.tar.gpg' failed too many > times. Skipping that file. > > real 303m38.849s > user 7m56.840s > sys 1m8.640s > > # time /etc/cron.daily/backup_to_s3 > Compiling list of local files... > Retrieving list of remote files... > Found 1663 local files, 1632 remote files > Verifying checksums... > Summary: 8 local files to upload, 0 remote files to delete > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > WARNING: MD5 Sums don't match! > ERROR: home.full.20090302_031507.tar.gpg: upload failed too many > times. Skipping that file. > File 'mysql.incr.20090304_030008.tar.gpg' stored as s3:// > backup.ourcompany/mysql.incr.20090304_030008.tar.gpg (80084541 bytes > in 171.0 seconds, 457.26 kB/s) [2 of 8] > File 'home.incr.20090304_030010.tar.gpg' stored as s3:// > backup.ourcompany/home.incr.20090304_030010.tar.gpg (14791469 bytes > in 1.9 seconds, 7.47 MB/s) [3 of 8] > File 'mysql.incr.20090304_030010.tar.gpg' stored as s3:// > backup.ourcompany/mysql.incr.20090304_030010.tar.gpg (1554 bytes in > 0.0 seconds, 9.43 MB/s) [4 of 8] > File 'svn.incr.20090304_030010.tar.gpg' stored as s3:// > backup.ourcompany/svn.incr.20090304_030010.tar.gpg (3553 bytes in > 0.0 seconds, 22.17 MB/s) [5 of 8] > File 'config.incr.20090304_071720.tar.gpg' stored as s3:// > backup.ourcompany/config.incr.20090304_071720.tar.gpg (32459 bytes > in 0.3 seconds, 125.75 kB/s) [6 of 8] > File 'home.incr.20090304_071720.tar.gpg' stored as s3:// > backup.ourcompany/home.incr.20090304_071720.tar.gpg (2705201 bytes > in 1.0 seconds, 2.60 MB/s) [7 of 8] > File 'mysql.incr.20090304_071720.tar.gpg' stored as s3:// > backup.ourcompany/mysql.incr.20090304_071720.tar.gpg (979 bytes in > 0.0 seconds, 6.06 MB/s) [8 of 8] > Done. Uploaded 97619756 bytes in 34733.5 seconds, 2.74 kB/s > > real 599m43.248s > user 10m35.850s > sys 2m7.890s > > > 0.9.9 fail in different way: > > # time s3cmd put home.full.20090302_031507.tar.gpg s3:// > backup.ourcompany > home.full.20090302_031507.tar.gpg -> s3://backup.ourcompany/ [1 of 1] > 5586801747 of 5586801747 100% in 11194s 487.39 kB/s done > ERROR: S3 error: 400 (MalformedXML): The XML you provided was not > well-formed or did not validate against our published schema > > real 186m34.334s > user 2m27.220s > sys 0m28.430s > > > -- > WBR, Alex. > > ------------------------------------------------------------------------------ > Open Source Business Conference (OSBC), March 24-25, 2009, San > Francisco, CA > -OSBC tackles the biggest issue in open source: Open Sourcing the > Enterprise > -Strategies to boost innovation and cut costs with open source > participation > -Receive a $600 discount off the registration fee with the source > code: SFAD > http://p.sf.net/sfu/XcvMzF8H > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general |