I'm running CentOS 5.x with Webmin 1.770. For the past 3 years, I have been relying upon scheduled backup jobs that I've configured in the Filesystem Backup provision of Webmin. Unfortunately, it has come to my attention that the transferred gz backups are finishing their sessions before a complete file has been transferred (i.e., the files are ever-so-slightly truncated at random points (they are usually short 25 to 40 kb). This fact results in an unusuable backup zip each time. ProFTPd is used on both the source and destination servers (v1.3.4 and v1.3.3 respectively) and each ProFTPd instance is running in binary mode.
Below is some pasted text of a few consecutive test backup sessions I executed of a particular directory this morning. I used the /etc directory on the source server since it is of a convenient size for testing (7.6 MB). As you will see, the final size of the tranfered file is always different. None of these files will open/extract with WinRAR on a Windows box. The very last session listed is one that somehow made a full transfer and will in fact open. The point at which each session finishes is sporadically intermittent.
Displayed completed file size: 7.62 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (7998576 bytes)
7998576 bytes received successfully. (16.41 KBps) (00:07:56).
226 Transfer complete
Displayed completed file size:7.64 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (8013460 bytes)
8013460 bytes received successfully. (16.48 KBps) (00:07:55).
226 Transfer complete
Displayed completed file size:7.63 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (8010180 bytes)
8010180 bytes received successfully. (16.40 KBps) (00:07:57).
226 Transfer complete
This is the only session that completed the full transfer and will thus open: Displayed completed file size:7.66 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (7998576 bytes)
8038400 bytes received successfully. (16.44 KBps) (00:07:57).
226 Transfer complete
When I try to open the truncated zips, WinRAR gives me the following error messages:
! D:\Documents and Settings\User1\Documents\File Store (temp)\transfer\host2_backup_etc.gz: Unexpected end of archive
! D:\Documents and Settings\User1\Documents\File Store (temp)\transfer\host2_backup_etc.gz: CRC failed in host2_backup_etc. The file is corrupt
Via Google, I've noticed a couple of other Webmin users reporting this exact problem. But I've found no resolution for it. Any thoughts or suggestions will be appreciated.
Thank you
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Slight paste error:
I copied the wrong text line on the last example and then only backspaced part of it then pasted in the last part correctly, leaving the erroneous first part from the incorrect paste (so it shows the wrong file size at the beginning). It's the only session that completed and it's the one I pasted at the end of my list. The actual text is as follows:
Displayed completed file size:7.66 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (8038400 bytes)
8038400 bytes received successfully. (16.44 KBps) (00:07:57).
226 Transfer complete
Sorry about that.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Update:
I've found that a workaround is to configure the scheduled tranfers to use SSH mode. But this slows the transfer time considerably. With FTP, the tranfer of a 2.35 GB TAR backup file would complete in approximately 30 minutes. Whereas when SSH is used, the same 2.35 GB TAR backup file takes 6.33 hours to complete. That's over 12 times longer! So using SSH as the backup transfer protocol on these systems seems to impose a tranfer speed limit of 372 MB/hour. I assume it has to do with the encryption/decryption process, but I'm guessing there's something not quite right with my configuration as well. I don't think it should transfer data this slowly when using SSH. But at least now the backup files are usable and the CRCs verify.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm running CentOS 5.x with Webmin 1.770. For the past 3 years, I have been relying upon scheduled backup jobs that I've configured in the Filesystem Backup provision of Webmin. Unfortunately, it has come to my attention that the transferred gz backups are finishing their sessions before a complete file has been transferred (i.e., the files are ever-so-slightly truncated at random points (they are usually short 25 to 40 kb). This fact results in an unusuable backup zip each time. ProFTPd is used on both the source and destination servers (v1.3.4 and v1.3.3 respectively) and each ProFTPd instance is running in binary mode.
Below is some pasted text of a few consecutive test backup sessions I executed of a particular directory this morning. I used the /etc directory on the source server since it is of a convenient size for testing (7.6 MB). As you will see, the final size of the tranfered file is always different. None of these files will open/extract with WinRAR on a Windows box. The very last session listed is one that somehow made a full transfer and will in fact open. The point at which each session finishes is sporadically intermittent.
Displayed completed file size: 7.62 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (7998576 bytes)
7998576 bytes received successfully. (16.41 KBps) (00:07:56).
226 Transfer complete
Displayed completed file size:7.64 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (8013460 bytes)
8013460 bytes received successfully. (16.48 KBps) (00:07:55).
226 Transfer complete
Displayed completed file size:7.63 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (8010180 bytes)
8010180 bytes received successfully. (16.40 KBps) (00:07:57).
226 Transfer complete
This is the only session that completed the full transfer and will thus open:
Displayed completed file size:7.66 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (7998576 bytes)
8038400 bytes received successfully. (16.44 KBps) (00:07:57).
226 Transfer complete
When I try to open the truncated zips, WinRAR gives me the following error messages:
! D:\Documents and Settings\User1\Documents\File Store (temp)\transfer\host2_backup_etc.gz: Unexpected end of archive
! D:\Documents and Settings\User1\Documents\File Store (temp)\transfer\host2_backup_etc.gz: CRC failed in host2_backup_etc. The file is corrupt
Via Google, I've noticed a couple of other Webmin users reporting this exact problem. But I've found no resolution for it. Any thoughts or suggestions will be appreciated.
Thank you
Slight paste error:
I copied the wrong text line on the last example and then only backspaced part of it then pasted in the last part correctly, leaving the erroneous first part from the incorrect paste (so it shows the wrong file size at the beginning). It's the only session that completed and it's the one I pasted at the end of my list. The actual text is as follows:
Displayed completed file size:7.66 MB
150 Opening BINARY mode data connection for host2_backup_etc.gz (8038400 bytes)
8038400 bytes received successfully. (16.44 KBps) (00:07:57).
226 Transfer complete
Sorry about that.
Update:
I've found that a workaround is to configure the scheduled tranfers to use SSH mode. But this slows the transfer time considerably. With FTP, the tranfer of a 2.35 GB TAR backup file would complete in approximately 30 minutes. Whereas when SSH is used, the same 2.35 GB TAR backup file takes 6.33 hours to complete. That's over 12 times longer! So using SSH as the backup transfer protocol on these systems seems to impose a tranfer speed limit of 372 MB/hour. I assume it has to do with the encryption/decryption process, but I'm guessing there's something not quite right with my configuration as well. I don't think it should transfer data this slowly when using SSH. But at least now the backup files are usable and the CRCs verify.