VeraCrypt / Forums / Technical Topics: Identical backup of a container has differences in its contents

Mark - 2021-02-01

Hi,

I am using VeraCrypt 1.24 on Windows 10 to store personal files inside a container file. The container is approx. 1.3TB in size.

The container file is backupped periodically on external USB harddisks.

After copying it to the external drive, the container backup file is then bit-by-bit compared to the original to make sure there is no corruption.

However, when after such a backup I mounted both the backup and the original container and did a bit-by-bit comparison of their contents and several hundred files where found that differ in the backup container from the original one (i.e. are corrupted).

How can that be, when both container files themselves appear to be identical when compared bit-by-bit?

Thanks!

Best,
Mark

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Enigma2Illusion - 2021-02-01

Are you backing-up the container file when it is unmounted?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-01

Yes.

What I did was:

unmount the container

copy the container file to an external harddisk using total commander with verification on

copying and verification was successfull

then mount both containers (the original and the copy) each with an own drive letter

use total commander's comparison tool to bit-by-bit read and compare all files to check for any differences between the two "drives"

quite a few differing files were found

I opened several of them to find the files in the copied container were corrupted

I tried again using a differen harddisk and copying it via windows explorer instead of total commander. Same result.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Enigma2Illusion - 2021-02-01

Thanks for the detail process and for trying to a different HDD using Windows Explorer instead of Total Commander. I have never use Total Commander.

I wonder if this has to do with the 1.3 TB size of the file that is causing the issue?

Another test is to copy the unmounted file container using Microsoft robocopy utility to see if this results in the backup container file still experiencing missing/corrupted files when mounting the backup container file. You run the robocopy from the command line.

Example of robocopy command copying one file called Test.hc from source drive C directory MyFileContainer to destination drive N directory MyFileContainer.

https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy

On my system, I have to run the command as Administrator to avoid the errors of:

Backup and Restore Files user rights from the /ZB switch

Copying NTFS Security to Destination File from the /COPY:DATS switch

.

robocopy "C:\MyFileContainer" "N:\MyFileContainer" "Test.hc" /ZB /R:10 /W:3 /COPY:DATS /DCOPY:T /NP /NDL /NFL /Log:"C:\MyFileContainer\robocopy.log"

After the robocopy completes, can you first configure Total Commander to check the mounted
source and backup file containers to check the existing directories and files match? Not checking the contents of the files.

I do not know if Total Commander is causing the problem when checking the contents of files.

If you still are experiencing missing/corrupt files, try using a new cable to the external HDD and a different USB port directly attached to the PC verses using a USB hub (you did not mention how you were connecting the USB from the PC to the external HDD).

Last edit: Enigma2Illusion 2021-02-02
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-01

Thanks! I'll try the robocopy command.

I tried different ports and all are directly at the PC.

Also, maybe this is helpful:

I already checked the contents of source and backup container just for completeness. All directories and files are there, none are missing.

It is only when checking the contents, too, bit-by-bit, that the corrupted files are found.

And: the corrupted files in the copied container still have the exact same file size as the non-corrupted in the original container. Only thing is there are some bits changed within them.

So if I just do a file-by-file comparison without contents (only checking if all files are there, names are the same, sizes are there same), no errors would be found.

Maybe it really is the size (and the large number of files within the container). I also have several smaller containers (<100GB) that do not exhibit these problems.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Enigma2Illusion - 2021-02-01

Another possibility is Total Commander is causing the damage when it performs the bit-by-bit comparison when larger number of files are involved in the operation.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Enigma2Illusion - 2021-02-01

Also, mount the backed-up file container as read-only using the options for mounting the VeraCrypt volume. This will prevent Total Commander or some other process from changing the files during your compare.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-01

Thank you - I will do that and will write back when the processes have finished. It will take a while though, as the copying and comparing takes quite a time through USB...

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-01

By the way, is there another tool for comparing the contents of two containers you could recommend (bit by bit), so I could see if they get to different results?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Enigma2Illusion - 2021-02-01

You could try FreeFileSync and configure Compare for File Content and the configure Synchronize for Update.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

infodan36 - 2021-02-01

Vice Versa Pro via Windows (or under Wine on Linux)
Otherwise I think you could get a try to BTRFS which should check copies, (authenticity + integrity)
(BTRFS has been implementend in VC update 7)

Last edit: infodan36 2021-02-01

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-02

Sorry, I just can't get robocopy to work.

It always tells me the syntax of the directory or filename is wrong, even though it is exactly right...

I tried renaming the file and the folder, but still get this:

Gestartet: Dienstag, 2. Februar 2021 13:44:26
Quelle : C:\VC\1.hc\
Ziel : D:\

Dateien : *.*

Optionen: . /NDL /NFL /S /E /DCOPY:T /COPY:DATS /ZB /NP /IM /R:10 /W:3

2021/02/02 13:44:26 FEHLER 123 (0x0000007B) Zugriff auf Quellverzeichnis C:\VC\1.hc\
Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-02

If I try a regular copy command, it works:

copy "C:\VC\1.hc" "D:\1.hc"

But if I try to copy the same file with robocopy using your line arguments, I always get this syntax error.

robocopy "C:\VC\1.hc" "D:\1.hc" /E /ZB /R:10 /W:3 /COPY:DATS /DCOPY:T /NP /NDL /NFL /Log:D:\robocopy.log

then the log reads:

Gestartet: Dienstag, 2. Februar 2021 14:09:04
Quelle : C:\VC\1.hc\
Ziel : D:\1.hc\

Dateien : *.*

Optionen: . /NDL /NFL /S /E /DCOPY:T /COPY:DATS /ZB /NP /IM /R:10 /W:3

2021/02/02 14:09:04 FEHLER 123 (0x0000007B) Zugriff auf Quellverzeichnis C:\VC\1.hc\
Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Adrian Kentleton - 2021-02-02

Robocopy is essentially a utility for copying directories (and their contents). It is interpreting your command as robocopy C:\VC\1.hc\*.* to D:\1.hc\*.*(as indicated in your log).

Last edit: Adrian Kentleton 2021-02-02

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-02

Thanks! I'll use

robocopy "C:!VC" "D:" /E /ZB /R:10 /W:3 /COPY:DATS /DCOPY:T /NP /NDL /NFL /Log:D:\robocopy.log

then and see if the file is copied correctly...

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Adrian Kentleton - 2021-02-02

If you had more than one .hc file in C:\VC, the syntax to robocopy just 1.hc would be robocopy C:\VC D: 1.hc (or, more explicit, probably safer!) robocopy C:\VC\ D:\ 1.hc

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Enigma2Illusion - 2021-02-02

@adriankit
Thank you for the correct syntax for copying one file verse directories and their files. I have updated my post with the correct syntax.

@mr123
Sorry Mark. I should have performed a quick test. I am normally using robocopy to copy directories and I provided the wrong syntax.

I just performed a quick test using the modified command in my post above and it works correctly.

My apologies.

EDIT: Okay, I tested again and made some command changes based on errors in the log file. On my system, I have to run the command as Administrator to avoid the errors of:

Backup and Restore Files user rights due to the restart switch /ZB

Copying NTFS Security to Destination File from the switch /COPY:DATS

.
Changed the robocopy command switches:

Added back switch /ZB

Removed switch /E which copies subdirectories including empty subdirectories.

Added double quotes for log file path and filename in case spaces are in the name.

.
I learned something new today on how to copy a single file using robocopy. :)

👍
1

Last edit: Enigma2Illusion 2021-02-02
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-02

Thanks again! It is now copying. That will take quite a while for the 1.5TB to go through USB (even with USB-C, the write transfer rate is usually about 100-110MB/sec).

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-03

Hi, the copying and comparing processes have succeeded. All files in the copied container appear to be bit-by-bit identical to the source container.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mark - 2021-02-03

I still wonder what it might have been. I'll be continuing to do the backups this way and if it happens again, I'll write... Maybe we can see what has happened. Especially since the container file itself seemed to be identical. That is what bugs me most...

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Adrian Kentleton - 2021-02-03

Good to hear. I use Robocopy for all my backup operations (usually 'mirroring'), tho' I only ever robocopy small unmounted VC volumes, preferring to mount and mirror the contents of larger volumes (into other mounted VC volumes). But I only ever backup over a secure network eg I'm not backing up into 'the cloud'.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Identical backup of a container has differences in its contents

Open source disk encryption with strong security for the Paranoid

Forums

Help

Identical backup of a container has differences in its contents

Identical backup of a container has differences in its contents

Open source disk encryption with strong security for the Paranoid

Forums

Help

Identical backup of a container has differences in its contents document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

Identical backup of a container has differences in its contents