Rescuezilla v1.0.6.1 does not have any file size limitations for both the source drive and the destination drive. In fact, the backup image gets split into 4GB chunks so that even a very large drive can safely be written into a FAT32 filesystem destination, as long as there is sufficient free space available in the destination.
The destination directory contains a log file (eg. "rescuezilla.log.202007XYZ.txt"). Please upload it and I will examine what went wrong.
By the way, it sounds like your 4.5TB RAID disk is your destination drive. This should be fine if this is a hardware RAID system that's transparently used by the operating system through external configuration (eg, BIOS or external RAID enclosure).
Given that you're a user of RAID disks, I should re-iterate that Rescuezilla v1.0.6.1 has extremely limited support for backup of RAID drives, as the Rescuezilla Limitations page documents:
Does not yet support drives without a partition table (filesystem directly on the disk). This likely applies to most software/hardware RAID setups (now captured as its own task: #43) (this will be fixed in Rescuezilla v1.0.7)
Only supports /dev/sdX and /dev/nvmeXnY device nodes. Does not yet support device nodes like /dev/mmcblkX etc, due to pre-existing architectural decisions (now captured as its own task #95) (this will be fixed in Rescuezilla v1.0.7)
Software RAID not supported by graphical wizard (to be fixed by #45)
Hardware RAID where multiple drives transparently appear as a single device should work without issues ONLY if the drive has a partition table (#43) (this will be investigated for Rescuezilla v1.0.7)
(Side note: Support for arbitrary RAID configurations will be vastly improved in Rescuezilla v1.0.7, which will be released in mid-October 2020.)
Again, when you get a chance, please upload the log file. I am very interested in understanding the situation you experienced, and I will update the "bug advisories" section of the Rescuezilla v1.0.6.1 release notes if required.
Last edit: Rescuezilla 2020-07-21
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you for forwarding your partclone log files to my 'users.sourceforge.net' Rescuezilla email address. Also thank you for including the quoted correspondence with your colleague as it provided some further context that I missed from your original post: that your RAID disk is being used as your source drive.
From your partclone log files your source "hardware RAID" drive does indeed contain a partition table. The partition table is what allows your disk to contain multiple partitions. A drive without a partition table can only have one and only one filesystem written onto it.
By the way, you didn't attach the Rescuezilla log file (eg. "rescuezilla.log.202007XYZ.txt"), so I don't know the complete partition table and filesystem information of your disk, or have direct knowledge of how Rescuezilla is operating internally. Regardless, there are (at least) 3 partitions which appear to be correctly presenting themselves to Rescuezilla as /dev/sda1, /dev/sda2 and /dev/sda3. This kind of configuration commonly known as "hardware RAID" is transparent to the operating system, so Rescuezilla handles it like any other disk. This is good since as mentioned above Rescuezilla v1.0.6.1 has limited support for other kinds of RAID configuration.
From your logs, partclone was able to backup the first two partitions: dev/sda1 (raw byte-for-byte backup) and /dev/sda2 (FAT32 filesytem) without issue. However, the third partclone command output the following:
For completeness I note that the filesytem on this third partition is being backed up in raw mode. This means that partclone is making a complete byte-for-byte backup of the partition. I am presuming this is expected because the filesystem is either encrypted or is an unencrypted filesystem that happens to not be supported by partclone highly space efficient "filesystem-aware" mode (such filesystems include Apple's recent APFS filesystem used on macOS and Microsoft's ReFS used on Windows Server), and so Rescuezilla has correctly decided to use partclone's raw byte-for-byte backup fallback mode to ensure it copies everything though that uses up a lot of disk space.
In the absence of the full rescuezilla log file, as far as I can tell, Rescuezilla itself is operating completely correctly. When it reaches the third partition it's the partclone utility used internally that starts processing bad sectors and cannot continue. As you may be aware, a bad sector indicates physical damage to a hard drive. As your colleague mentioned in the email thread you forwarded me:
My suspicion is that it’s not a problem with any of the RAID drives causing the problem we’re having, because of the info I can see reported by the SmartMonTools program, especially now that I’ve replaced drive #4 which contained “uncorrected errors” during earlier backup attempts.
In the absence of the full rescuezilla log file, as best I can tell the error is in your drive array. I don't know what RAID level your drive is operating at, or other details of your environment so I can't yet provide specific advice about how best to fix bad sectors. However, Rescuezilla also provides the SmartMonTools, so I recommend you start with a full non-destructive read-only scan of your drive for bad sectors, before potentially having smartctl reallocate them. This looks like a good tutorial: https://www.smartmontools.org/wiki/BadBlockHowto
(EDIT/Side note: subsequent email says says the drive array is RAID5. I am not an expert in system administration of RAID configurations or rebuilding RAID at this very moment, so right now can't provide much pointers beyond "try using smartctl to do a bad sector scan" as described above. I should note a recent user experienced a similar issue with what appears to be filesystem issue with their destination disk. After this was corrected the backup worked smoothly.)
For completion, I should mention that to assist debugging it's possible to run the partclone utility from the command-line removing Rescuezilla from the equation. It will still take ~2.75 hours for partclone reach this point as your first two partitions together are less than 1GB, and your third partition is 4.0TB. Until the bad sector issue is repaired, partclone likely will not work.
As you have seen, the partclone utility is not particularly resilient to bad sectors. While your RAID disk sounds healthy aside from some bad sectors, I should mention that Rescuezilla provides the extremely powerful ddrescue command-line tool to provide extremely resilient byte-for-byte imaging (and cloning) of failing media with many bad sectors (whether hard drives or optical media). It's difficult to use and is probably not something you need to be using here, but it's worth being aware of. (A future version of Rescuezilla will provide a graphical frontend to this tool).
By the way, for my own investigation, how did the Rescuezilla v1.0.6.1 frontend behave when it reaches the partclone bad sector error? Does it show some kind of error box that provides an indication of the error, or does it simply get "stuck" (as can be the case until I complete task #29 as part of v1.0.7)? I have just created task #114 to display the best possible error message for Rescuezilla v1.0.7 by first creating a test environment containing bad sectors.
Last edit: Rescuezilla 2020-07-23
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Is there any backup file restriction?
I am trying to do a full backup on a 4.5 TB raid
and it always stops at 2 3/4 hours into it.
Hi Mike,
Thanks for attempting to use Rescuezilla!
Rescuezilla v1.0.6.1 does not have any file size limitations for both the source drive and the destination drive. In fact, the backup image gets split into 4GB chunks so that even a very large drive can safely be written into a FAT32 filesystem destination, as long as there is sufficient free space available in the destination.
The destination directory contains a log file (eg. "rescuezilla.log.202007XYZ.txt"). Please upload it and I will examine what went wrong.
By the way, it sounds like your 4.5TB RAID disk is your destination drive. This should be fine if this is a hardware RAID system that's transparently used by the operating system through external configuration (eg, BIOS or external RAID enclosure).
Given that you're a user of RAID disks, I should re-iterate that Rescuezilla v1.0.6.1 has extremely limited support for backup of RAID drives, as the Rescuezilla Limitations page documents:
(Side note: Support for arbitrary RAID configurations will be vastly improved in Rescuezilla v1.0.7, which will be released in mid-October 2020.)
Again, when you get a chance, please upload the log file. I am very interested in understanding the situation you experienced, and I will update the "bug advisories" section of the Rescuezilla v1.0.6.1 release notes if required.
Last edit: Rescuezilla 2020-07-21
Hi Mike,
Thank you for forwarding your partclone log files to my 'users.sourceforge.net' Rescuezilla email address. Also thank you for including the quoted correspondence with your colleague as it provided some further context that I missed from your original post: that your RAID disk is being used as your source drive.
From your partclone log files your source "hardware RAID" drive does indeed contain a partition table. The partition table is what allows your disk to contain multiple partitions. A drive without a partition table can only have one and only one filesystem written onto it.
By the way, you didn't attach the Rescuezilla log file (eg. "rescuezilla.log.202007XYZ.txt"), so I don't know the complete partition table and filesystem information of your disk, or have direct knowledge of how Rescuezilla is operating internally. Regardless, there are (at least) 3 partitions which appear to be correctly presenting themselves to Rescuezilla as /dev/sda1, /dev/sda2 and /dev/sda3. This kind of configuration commonly known as "hardware RAID" is transparent to the operating system, so Rescuezilla handles it like any other disk. This is good since as mentioned above Rescuezilla v1.0.6.1 has limited support for other kinds of RAID configuration.
From your logs, partclone was able to backup the first two partitions: dev/sda1 (raw byte-for-byte backup) and /dev/sda2 (FAT32 filesytem) without issue. However, the third partclone command output the following:
That warning message was repeated 127 times.
For completeness I note that the filesytem on this third partition is being backed up in raw mode. This means that partclone is making a complete byte-for-byte backup of the partition. I am presuming this is expected because the filesystem is either encrypted or is an unencrypted filesystem that happens to not be supported by partclone highly space efficient "filesystem-aware" mode (such filesystems include Apple's recent APFS filesystem used on macOS and Microsoft's ReFS used on Windows Server), and so Rescuezilla has correctly decided to use partclone's raw byte-for-byte backup fallback mode to ensure it copies everything though that uses up a lot of disk space.
In the absence of the full rescuezilla log file, as far as I can tell, Rescuezilla itself is operating completely correctly. When it reaches the third partition it's the partclone utility used internally that starts processing bad sectors and cannot continue. As you may be aware, a bad sector indicates physical damage to a hard drive. As your colleague mentioned in the email thread you forwarded me:
In the absence of the full rescuezilla log file, as best I can tell the error is in your drive array. I don't know what RAID level your drive is operating at, or other details of your environment so I can't yet provide specific advice about how best to fix bad sectors. However, Rescuezilla also provides the SmartMonTools, so I recommend you start with a full non-destructive read-only scan of your drive for bad sectors, before potentially having smartctl reallocate them. This looks like a good tutorial: https://www.smartmontools.org/wiki/BadBlockHowto
(EDIT/Side note: subsequent email says says the drive array is RAID5. I am not an expert in system administration of RAID configurations or rebuilding RAID at this very moment, so right now can't provide much pointers beyond "try using smartctl to do a bad sector scan" as described above. I should note a recent user experienced a similar issue with what appears to be filesystem issue with their destination disk. After this was corrected the backup worked smoothly.)
For completion, I should mention that to assist debugging it's possible to run the partclone utility from the command-line removing Rescuezilla from the equation. It will still take ~2.75 hours for partclone reach this point as your first two partitions together are less than 1GB, and your third partition is 4.0TB. Until the bad sector issue is repaired, partclone likely will not work.
As you have seen, the partclone utility is not particularly resilient to bad sectors. While your RAID disk sounds healthy aside from some bad sectors, I should mention that Rescuezilla provides the extremely powerful ddrescue command-line tool to provide extremely resilient byte-for-byte imaging (and cloning) of failing media with many bad sectors (whether hard drives or optical media). It's difficult to use and is probably not something you need to be using here, but it's worth being aware of. (A future version of Rescuezilla will provide a graphical frontend to this tool).
By the way, for my own investigation, how did the Rescuezilla v1.0.6.1 frontend behave when it reaches the partclone bad sector error? Does it show some kind of error box that provides an indication of the error, or does it simply get "stuck" (as can be the case until I complete task #29 as part of v1.0.7)? I have just created task #114 to display the best possible error message for Rescuezilla v1.0.7 by first creating a test environment containing bad sectors.
Last edit: Rescuezilla 2020-07-23