[Errno 32] Broken pipe and [Errno 104] when putting large file on S3

2013-05-10
2013-08-06
  • Dusan Dordevic

    Dusan Dordevic - 2013-05-10

    Hi,

    When I try to put large file (23gb) to S3, I receive one of two mentioned errors:

    root:~# /usr/bin/s3cmd sync /var/s3backups/ s3://clavis-backups/clients/gs1das-srvr1-rack-g2/backups/ --delete-removed
    /var/s3backups/cin_messages.tar -> s3://clavis-backups/clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar [1 of 1]
    3395584 of 23655065600 0% in 1s 2.75 MB/s failed
    WARNING: Upload failed: /clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar ([Errno 32] Broken pipe)
    WARNING: Retrying on lower speed (throttle=0.00)
    WARNING: Waiting 3 sec...
    /var/s3backups/cin_messages.tar -> s3://clavis-backups/clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar [1 of 1]
    3256320 of 23655065600 0% in 1s 2.55 MB/s failed
    WARNING: Upload failed: /clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar ([Errno 104] Connection reset by peer)
    WARNING: Retrying on lower speed (throttle=0.01)
    WARNING: Waiting 6 sec...
    /var/s3backups/cin_messages.tar -> s3://clavis-backups/clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar [1 of 1]
    413696 of 23655065600 0% in 1s 341.96 kB/s failed
    WARNING: Upload failed: /clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar ([Errno 32] Broken pipe)
    WARNING: Retrying on lower speed (throttle=0.05)
    WARNING: Waiting 9 sec...
    /var/s3backups/cin_messages.tar -> s3://clavis-backups/clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar [1 of 1]
    90112 of 23655065600 0% in 1s 70.83 kB/s failed
    WARNING: Upload failed: /clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar ([Errno 32] Broken pipe)
    WARNING: Retrying on lower speed (throttle=0.25)
    WARNING: Waiting 12 sec...
    /var/s3backups/cin_messages.tar -> s3://clavis-backups/clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar [1 of 1]
    28672 of 23655065600 0% in 1s 17.16 kB/s failed
    WARNING: Upload failed: /clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar ([Errno 32] Broken pipe)
    WARNING: Retrying on lower speed (throttle=1.25)
    WARNING: Waiting 15 sec...
    /var/s3backups/cin_messages.tar -> s3://clavis-backups/clients/gs1das-srvr1-rack-g2/backups/cin_messages.tar [1 of 1]
    12288 of 23655065600 0% in 2s 4.48 kB/s failed
    ERROR: /var/s3backups/cin_messages.tar: upload failed too many times. Skipping that file.

    Any idea what might be the problem ?

     
  • Barrett Johnson

    Barrett Johnson - 2013-08-06

    I started experiencing this same issue about a week ago (last worked on 7/30). I get similar output when using a simple PUT, and with a quite small file.

    vcloud@glproddb2:~$ s3cmd put glproddb2-13-08-06-01-30.sql.gz s3://egd/db_backup
    glproddb2-13-08-06-01-30.sql.gz -> s3://egd/db_backup [1 of 1]
    4521984 of 7101334 63% in 1s 3.46 MB/s failed
    WARNING: Upload failed: /db_backup ([Errno 104] Connection reset by peer)
    WARNING: Retrying on lower speed (throttle=0.00)
    WARNING: Waiting 3 sec...
    glproddb2-13-08-06-01-30.sql.gz -> s3://egd/db_backup [1 of 1]
    4800512 of 7101334 67% in 1s 3.92 MB/s failed
    WARNING: Upload failed: /db_backup ([Errno 104] Connection reset by peer)
    WARNING: Retrying on lower speed (throttle=0.01)
    WARNING: Waiting 6 sec...
    glproddb2-13-08-06-01-30.sql.gz -> s3://egd/db_backup [1 of 1]
    397312 of 7101334 5% in 1s 333.43 kB/s failed
    WARNING: Upload failed: /db_backup ([Errno 32] Broken pipe)
    WARNING: Retrying on lower speed (throttle=0.05)
    WARNING: Waiting 9 sec...
    glproddb2-13-08-06-01-30.sql.gz -> s3://egd/db_backup [1 of 1]
    90112 of 7101334 1% in 1s 73.58 kB/s failed
    WARNING: Upload failed: /db_backup ([Errno 32] Broken pipe)
    WARNING: Retrying on lower speed (throttle=0.25)
    WARNING: Waiting 12 sec...
    glproddb2-13-08-06-01-30.sql.gz -> s3://egd/db_backup [1 of 1]
    24576 of 7101334 0% in 1s 17.35 kB/s failed
    WARNING: Upload failed: /db_backup ([Errno 32] Broken pipe)
    WARNING: Retrying on lower speed (throttle=1.25)
    WARNING: Waiting 15 sec...
    glproddb2-13-08-06-01-30.sql.gz -> s3://egd/db_backup [1 of 1]
    12288 of 7101334 0% in 2s 4.56 kB/s failed
    ERROR: Upload of 'glproddb2-13-08-06-01-30.sql.gz' failed too many times. Skipping that file.

    I set up a stop gap solution, but was going to try to troubleshoot a bit more this coming week. I am just curious if you or others may have discovered anything in your investigations.

     

Log in to post a comment.