Re: [Dar-support] Need help determining sliced archive corruption
For full, incremental, compressed and encrypted backups or archives
Brought to you by:
edrusb
Menu
▾
▴
Re: [Dar-support] Need help determining sliced archive corruption
|
From: mannino <ma...@of...> - 2026-06-02 21:06:50
|
As usual, thank you for responding, Denis!
I have conducted some tests but, as it often happens, an attempt to
explicitly repro an issue goes without a hassle. Guess I will have to
keep an eye on new backups to see if this resurfaces. Meanwhile, I still
have some questions/suggestions for retrospective analysis.
>> I have two different systems that back up data using incremental
>> archives. Those incremental archives (there's about 200 of them) are
>> fine according to dar -t. However, base (full) archives produced on
>> both systems at different times are reported as corrupted.
>>
>> Adding -va to -t provides little insight into what's going on:
>>
>> # dar version 2.7.17
>>
>> Auto detecting min-digits to be 2
>> Opening archive aa ...
>> Opening the archive using the multi-slice abstraction layer...
>> Reading the archive trailer...
>> Final memory cleanup...
>> FATAL error, aborting operation: Unexpected value while reading
>> archive version
>
> this means that the format version field is not properly formatted and
> dar cannot know which is the internal structure of the archive.
Data corruption is unlikely to happen on two different systems to just
the header block out of all other 100s of GiBs of dar data. Also, since
the corruption is in the last slice (?), last slice is never larger than
other slices, each slice is transferred to another system and deleted
(-E) and non-last slices were not truncated, free space shortage
shouldn't be the cause either.
Other than a bug in dar itself (which I prefer to not consider thanks to
your exceptional job), I still think it has something to do with wrong
order/number of slices. But I cannot be certain - validating this (or
any other) theory is hard without enough metadata:
1. Archive UUID, as it's currently implemented, is perfect: (1) it
practically guarantees that one cannot mix up slices from different
archives; (2) dar itself checks that when testing; (3) can be displayed
with `file`:
arc.10.dar: dar archive, label "82af71c2 00000000 fffff752"
I could only suggest storing UUID of the reference archive in the -@
catalogue too (since -@ gets its own different UUID) and for dar itself
to provide some way to view this UUID. In any case, UUID helps one work
out the consistency among archives - but there's nothing similar for
working out slices for the archive itself, and it's a problem for me.
2. Byte sizes of non/last slices would allow basic integrity check but
(1) `file` doesn't show them; (2) dar doesn't show them in -l -0 mode
(so unable to extract from invalid file); (3) dar -l -q works on
isolated catalogue but only reports non-last slice size - given our
problem here is likely last slice being truncated, it doesn't really help.
3. Slice number is what I truly miss. According to my tests, messing
with non-last slices (removing, duplicating, reordering) is usually
detected by dar as different types of data inconsistency so even if you
have all slices perfectly good and well, just in the wrong order, you
would think the backup is botched, and those errors would only be
misleading.
For example, on one of my backups (40+ slices, 200 GiB) I accidentally
missed the first slice. dar -t reported:
Reached End of File while reading archive version
dar -t -0:
Unexpected value while reading archive version
Also, out of curiosity, I tried copying one slice twice and got a ton of
CRC errors.
With the help of 4 extra bytes (= slice number) in each slice's header,
we would be able to identify the problem precisely. Not only will it
tell us slice order, it will also tell us the total number of slices
(since identifying last slice is trivial). Storing total number of
slices in the catalogue would take it even further, but even just
counters would be an incredible step forward for me.
>> I tried checking with strace to see what files dar -t opens. For both
>> archives, it starts with the last one in the directory, then opens
>> some seemingly random one in the 2/3rd of the set (I have ~30 and ~100
>> slices so it opens the ~30th/~70th respectively), then after a bunch
>> of seeks it fails.
>
> Weird, the archive version information is fetched very early in the
> archive reading process, if this field is corrupted, dar should open the
> last slice only and fail with the message you reported.
Yet this is exactly what happens:
# strace -efile dar -t arc -E "echo reading slice %N"
...
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
reading slice 29
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19001,
si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
openat(AT_FDCWD, "/tmp/arc.29.dar", O_RDONLY) = 4
reading slice 21
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19002,
si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
openat(AT_FDCWD, "/tmp/arc.21.dar", O_RDONLY) = 4
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/dar.mo",
O_RDONLY) = -1 ENOENT (No such file or directory)
<locale stuff redacted>
Final memory cleanup...
FATAL error, aborting operation: Unexpected value while reading
archive version
+++ exited with 2 +++
>> I tried -t -0. It resulted in an entry like so for 4 files:
>>
>> can't read data CRC: No escape mark found for that file
>>
>> ...followed by this output:
>>
>> --------------------------------------------
>> 1619434 item(s) treated
>> 4 item(s) with error
>> 0 item(s) ignored (excluded by filters)
>> --------------------------------------------
>> Total number of items considered: 1619438
>> --------------------------------------------
>> Final memory cleanup...
>> Some files are corrupted in the archive and it will not be possible
>> to restore them
>> +++ exited with 5 +++
>
> the final error is probably because the archive has been interrupted or
> because the disk was full when it was created and dar was not in
> interactive mode (it was run from as a cron job, for example).
>
> This would also explain why without -0 (--sequential-read) option dar is
> not able to find the archive version within a proper format, because
> this information is lacking at the end of the last slice.
>
> But the data at the end of the archive seems having a normal terminator
> and slice trailer is present (else the error would have been different)...
>
> The good thing however is that you should be able to restore 1619434
> files and directories from your backup using the --sequential-read mode.
Yes, the recovery went fine. Comparing it with the isolated catalogue
that was created on the fly when the backup was made, it seems
everything apart from those 4 files was restored correctly. My main
issue here isn't recovery of these particular archives per se, it's
trying to find how exactly they are corrupted so I can go on finding the
cause and avoid this in the future.
>> No other errors were logged. None were logged for the first and last
>> slices.
>
> When testing the backup in sequential-read mode (-0/--sequential-read
> option) could you add the following option:
> -E "echo reading slice %N"
> and report the last slice being reported as being read by this added -E
> option, before the failure takes place?
As per the above output, it says "reading slice 21".
> Also could you tell me the number of the last slice available?
There are arc.1.dar through arc.29.dar, the total of 29 slices.
> > > Does it indicate that the archive is fine sans for 4 files' data? Then
>> what's with that "Unexpected value" error in direct mode?
>
> it indicates that before the data corruption (= Unexpected value) you
> can restore 1619434 files and directories and 4 entries (probably 4
> nested directories opened but not closed (?)) are reported as having an
> error.
No, those entries correspond to regular files (some tiny, some huge)
spread among different slices.
> Doesn't dar report the entry names of that 4 failed entries???
It does, I just redacted it - sorry for the confusion. They're all in
different slices and directories so it shouldn't really matter.
>> -l -q -0: fails with (???):
>>
>> Sorry, file size is unknown at this step of the program.
>> The last file of the set is not present in file:///, please provide
>> it.
>
> interesting, this gives weight to the scenario of the truncated backup...
>
>>
>> Finally, I tried -al to no avail:
>>
>> LAX MODE: Failed to read the archive format version.
>> LAX MODE: Please provide the format MAJOR number: 11
>> LAX MODE: Please provide the format MINOR number: 3
>> LAX MODE: Using archive format "11.3"? [return = YES | Esc = NO]
>> Continuing...
>
> OK so we dar continues as if the format field was correct
>
>> LAX MODE: Unknown compression algorithm used, assuming data
>> corruption occurred. Please help me, answering with one of the
>> following words "none", "gzip", [...] at the next prompt:gzip
>
> This means that not only the archive format field is corrupted but also
> the compression algo field...
>
>> Unknown crypto algorithm used in archive, ignoring that field and
>> simply assuming the archive has been encrypted, if not done you will
>> need to specify the crypto algorithm to use in order to read this archive
>> Error met while reading archive of reference slicing layout,
>> ignoring this field and continuing
>> Aborting due to exception: Badly formed "infinint" or not supported
>> format
>
> Here you could get one step further adding '-K none:' on command-line to
> tell dar that the archive was not encrypted (if it was not, of course).
It wasn't encrypted. Adding -K none: didn't change anything except the
"Unknown crypto algorithm" line disappeared:
...
LAX MODE: Unknown compression algorithm used, assuming data
corruption occurred. Please help me, answering with one of the following
words "none", "gzip", "bzip2", "lzo", "xz", "zstd" or "lz4" at the next
prompt:gzip
Error met while reading archive of reference slicing layout, ignoring
this field and continuing
Final memory cleanup...
FATAL error, aborting operation: Badly formed "infinint" or not
supported format
> Do you still have the isolated catalog of the full backups?
>
> If so you could try the following command:
> dar -t <backup> -A <isolated cat> -affs
>
> the -affs is necessary because the last slice is obviously corrupted,
> this lead dar to read the archive header format and other important
> stuff from the copy located in the first slice. This option/feature was
> added in 2.8.x.
>
> Doing so will give you an exact picture of the amount of data you are
> missing from your backup.
dar -t -A -affs:
- 790 x CRC error: data corruption
- 3046 x compressed data CRC error
--------------------------------------------
715129 item(s) treated
3836 item(s) with error
0 item(s) ignored (excluded by filters)
--------------------------------------------
Total number of items considered: 718965
--------------------------------------------
dar -t -0:
- 4 x can't read data CRC: No escape mark found for that file
--------------------------------------------
1619434 item(s) treated
4 item(s) with error
0 item(s) ignored (excluded by filters)
--------------------------------------------
Total number of items considered: 1649928
--------------------------------------------
dar -t <catalogue>:
WARNING! This is an isolated catalogue, no data or EA is present in
this archive, only the catalogue structure can be checked
--------------------------------------------
0 item(s) treated
0 item(s) with error
0 item(s) ignored (excluded by filters)
--------------------------------------------
Total number of items considered: 0
--------------------------------------------
dar -l <catalogue> -q:
Archive version format : 11.3
Compression algorithm used : bzip2
Compression block size used : 0
Symmetric key encryption used : none
Asymmetric key encryption used : none
Archive is signed : no
Sequential reading marks : present
User comment :
Catalogue size in archive : 35328204 bytes
Archive is composed of 1 file(s)
File size: 35328416 bytes
The global data compression ratio is: 37%
WARNING! This archive only contains the catalogue of another archive,
it can only be used as reference for differential backup or as rescue in
case of corruption of the original archive's content. You cannot restore
any data from this archive alone
Archive of reference slicing:
Other slices: 10737418240 byte(s)
in-place path: redacted
CATALOGUE CONTENTS :
total number of inode : 1589967
fully saved : 1589967
binay delta patch : 0
inode metadata only : 0
distribution of inode(s)
- directories : 236542
- plain files : 1322581
- symbolic links : 30513
- named pipes : 16
- unix sockets : 33
- character devices : 280
- block devices : 2
- Door entries : 0
hard links information
- number of inode with hard link : 49600
- number of reference to hard linked inodes: 109555
destroyed entries information
0 file(s) have been record as destroyed since backup of reference
file arc.cat.1.dar arc.1.dar:
arc.cat.1.dar: dar archive, label "7929f969 00000000 8a33" end slice
arc.1.dar: dar archive, label "0503f969 00000000 8a33"
Why -affs, -0, -l -q <cat> all report different file counts?
|
