Re: [Dar-support] Need help determining sliced archive corruption

[Denis C. <dar...@fr...>]

Le 02/06/2026 à 23:05, mannino a écrit :
> As usual, thank you for responding, Denis!

you are welcome

[...]

> Data corruption is unlikely to happen on two different systems to just 
> the header block out of all other 100s of GiBs of dar data.

I fully agree, this is weird...

> Also, since 
> the corruption is in the last slice (?), last slice is never larger than 
> other slices, each slice is transferred to another system and deleted (- 
> E) and non-last slices were not truncated, free space shortage shouldn't 
> be the cause either.

correct

> 
> Other than a bug in dar itself (which I prefer to not consider thanks to 
> your exceptional job)

exceptional or not, all software can have bugs and I never excluded it, 
the problem is the lack of reproducibility in your context...

But as you mentioned you are not using a complex mix of features from 
dar and those you are using are broadly used and tested: there are 
non-regression tests running several hundred thousands times dar, mixing 
a lot of feature on and off and taking weeks to complete, that is run 
fully at least for each major release... So there may still be a bug 
that express from time to time, but I suspect it is somehow also 
strongly related to your context...

>, I still think it has something to do with wrong 
> order/number of slices.

to be honest, I don't see how mixing or even renumbering slices could 
cause the behavior you hit...

> But I cannot be certain - validating this (or 
> any other) theory is hard without enough metadata:

yep.

> 
> 1. Archive UUID, as it's currently implemented, is perfect: (1) it 
> practically guarantees that one cannot mix up slices from different 
> archives; (2) dar itself checks that when testing; (3) can be displayed 
> with `file`:
> 
> arc.10.dar: dar archive, label "82af71c2 00000000 fffff752"
> 
> I could only suggest storing UUID of the reference archive in the -@ 
> catalogue too (since -@ gets its own different UUID) and for dar itself 
> to provide some way to view this UUID. In any case, UUID helps one work 
> out the consistency among archives - but there's nothing similar for 
> working out slices for the archive itself, and it's a problem for me.

if you have a look at dar's documentation about the archive format, you 
will see that this is done with more flexibility:
- internal name : links slices of an archive together
- data name : links a catalog (or isolated catalog) to the data set to 
use. This prevents one trying to restore an archive with the help of an 
unrelated isolated catalog.

http://dar.linux.free.fr/doc/Notes.html#archive_structure

But yes, it could be shown when listing archive details...

> 
> 2. Byte sizes of non/last slices would allow basic integrity check but 
> (1) `file` doesn't show them; (2) dar doesn't show them in -l -0 mode 
> (so unable to extract from invalid file); (3) dar -l -q works on 
> isolated catalogue but only reports non-last slice size - given our 
> problem here is likely last slice being truncated, it doesn't really help.

dar cannot know in advance how much slices will be necessary nor what 
will be the size of the last slice. The sar layer (which manages slices) 
processes a stream of bytes from upper layers (ciphering, compression, 
filtering, and so forth) which is generated on the fly from the 
filesystem to backup, the sar takes this flow and creates slices from it 
up to the time the flow dries up.

What is stored at the beginning of *each slice* is the so called 
slice-layout:
- first slice max size,
- other slices max size,
- slice header size
- and slice format (2 formats so far).

Only this could be shown in sequential-read mode, not what is displayed 
today in direct mode which adds to this the overall size of the archive 
and the size of the last slice (easy to get because in that mode this is 
the last slice that get read first).

> 
> 3. Slice number is what I truly miss. According to my tests, messing 
> with non-last slices (removing, duplicating, reordering) is usually 
> detected by dar as different types of data inconsistency so even if you 
> have all slices perfectly good and well, just in the wrong order, you 
> would think the backup is botched, and those errors would only be 
> misleading.

the slice number is expected to be in the filename and it is not 
expected that users would change/renumber it...

> 
> For example, on one of my backups (40+ slices, 200 GiB) I accidentally 
> missed the first slice. dar -t reported:
> 
>    Reached End of File while reading archive version
> 
> dar -t -0:
> 
>    Unexpected value while reading archive version

yes, both way open the archive in a different way, so the inconsistency 
is reported differently. What I don't understand is how you could 
accidentally miss the first slice, I mean how do you let dar accessing 
the slices as dar, even in sequential-read mode open a slice after a 
given filename and if it is not present it complains and waits.

> 
> Also, out of curiosity, I tried copying one slice twice and got a ton of 
> CRC errors.
 > > With the help of 4 extra bytes (= slice number) in each slice's 
header,
> we would be able to identify the problem precisely. 

why only 4 bytes? This would limit the number of slices... even if today 
you think it is large enough, this is the same way of thinking that led 
to the year 2000 bug, the problem of year 2038, the upper and high 
memory in DOS some decades ago, the 2 GB files boundary (and need of 
large file support)... and so forth.

A better approach could be to add a random fixed size field at end of 
slice and have this same random value at the beginning of the next. But 
I don't see the use case as mentioned earlier about number of slices in 
filenames.

> Not only will it 
> tell us slice order, it will also tell us the total number of slices 
> (since identifying last slice is trivial).

not to my standpoint, as all slices may not be visible at a given time 
in particular when using removal media or remote storage.

>  Storing total number of 
> slices in the catalogue would take it even further, 

this would break the ability to re-slice (dar_xform) an archive, once 
the isolated catalogue would have been created.

> but even just 
> counters would be an incredible step forward for me.

This is something unclear to me: you have this slice number information 
in the slice filename... isn't it the easiest way to know what slice 
number is a slice? And how much slice do one have? And also what is the 
latest slice? By the way, you even have the --min-digit option to have 
slices sorted in lexical order... my 2 cents...

> 

[...]

>>
>> The good thing however is that you should be able to restore 1619434
>> files and directories from your backup using the --sequential-read mode.
> 
> Yes, the recovery went fine. 

OK, good to know!

> Comparing it with the isolated catalogue 
> that was created on the fly when the backup was made, it seems 
> everything apart from those 4 files was restored correctly. My main 
> issue here isn't recovery of these particular archives per se, it's 
> trying to find how exactly they are corrupted so I can go on finding the 
> cause and avoid this in the future.

sure!

[...]

>>
>> When testing the backup in sequential-read mode (-0/--sequential-read
>> option) could you add the following option:
>>      -E "echo reading slice %N"
>> and report the last slice being reported as being read by this added -E
>> option, before the failure takes place?
> 
> As per the above output, it says "reading slice 21".
> 
>> Also could you tell me the number of the last slice available?
> 
> There are arc.1.dar through arc.29.dar, the total of 29 slices.

OK so you don't miss slices, but have those corrupted at different places.

> 
>>  > > Does it indicate that the archive is fine sans for 4 files' data? 
>> Then
>>> what's with that "Unexpected value" error in direct mode?
>>
>> it indicates that before the data corruption (= Unexpected value) you
>> can restore 1619434 files and directories and 4 entries (probably 4
>> nested directories opened but not closed (?)) are reported as having an
>> error.
> 
> No, those entries correspond to regular files (some tiny, some huge) 
> spread among different slices.

OK, that's interesting!

> 
>> Doesn't dar report the entry names of that 4 failed entries???
> 
> It does, I just redacted it - sorry for the confusion. They're all in 
> different slices and directories so it shouldn't really matter.

In sequential read mode, obviously, the archive is read in sequence, 
after having read the data of a files, dar reads right after the 
expected CRC and compares it with the one calculated on the retreived 
data and eventually reports a difference. Then it skips forward looking 
for the next tape mark and once again and the loop repeats...

Here, corruption occurred in four different files (and slices), not only 
the very last files (in the last slice) as I had understood, which is 
not the pattern of a truncated slice.

[...]

> 
>> Do you still have the isolated catalog of the full backups?
>>
>> If so you could try the following command:
>> dar -t <backup> -A <isolated cat> -affs
>>
>> the -affs is necessary because the last slice is obviously corrupted,
>> this lead dar to read the archive header format and other important
>> stuff from the copy located in the first slice. This option/feature was
>> added in 2.8.x.
>>
>> Doing so will give you an exact picture of the amount of data you are
>> missing from your backup.
> 
> dar -t -A -affs:
> 
> - 790 x CRC error: data corruption
> - 3046 x compressed data CRC error
> 
>    --------------------------------------------
>    715129 item(s) treated
>    3836 item(s) with error
>    0 item(s) ignored (excluded by filters)
>    --------------------------------------------
>    Total number of items considered: 718965
>    --------------------------------------------
> 

718965 inodes tested, 790 with CRC error (not compressed ones) and 3046 
with error in data compressed structure (compressed ones)... I would 
really double check the RAM... but read further:

> 
> dar -t -0:
> 
> - 4 x can't read data CRC: No escape mark found for that file
> 
>    --------------------------------------------
>    1619434 item(s) treated
>    4 item(s) with error
>    0 item(s) ignored (excluded by filters)
>    --------------------------------------------
>    Total number of items considered: 1649928
>    --------------------------------------------
> 

It is possible to explain the different number of faults on CRC between 
the two reading modes, due to the fact this is not the same CRC copy 
that is used to compare with the data (interleaved with data in 
sequential-read mode, stored in memory during the backup process and 
wrote to disk at the end in the form of a catalogue, in direct mode).

But I cannot understand why you get 3046 compressed data error on one 
side only, unless the compression algorithm used has been tampered in 
memory (not on storage because then it would have lead to a CRC mismatch 
when loading the catalogue to memory).

Now about the total number of items...

Accordingly to the details report you provided from the isolated 
catalogue, I can explain almost the 1649928 entries:
it corresponds to:
	total number of inode : 1589967
plus
	number of reference to hard linked inodes: 109555
but less:
	number of inode with hard link           : 49600

because each first time an hard linked inode is met is also the first 
time a hard link to this item is met and both counts as 1, then 
subsequent hard links count also 1. So you must avoid counting twice the 
first hard link, but need to add to the inodes count those of hard-links 
found in the catalogue content summary.

This makes 1649922, though I can't explain the difference of 6 files, 
but I double checked the code, did some tests and get this expected 
counts and got the same in both sequential and direct modes and even 
when manually corrupting the archive file.

Now I cannot explain why the same isolated catalogue when applied to the 
corrupted archive as backup only report 718965 entries, this is really 
weird. They should all be tried in turn and lead to increase either the 
pass or fail counter, for a total count to match the one found in 
sequential read mode.

Thus, I would really double check the RAM, not the storage, because if 
there was a corruption at storage level, the catalogue would not have 
been loaded properly and an error would have shown.

Can you double check with for example memtest86+ ?

> 
> dar -t <catalogue>:
> 
>    WARNING! This is an isolated catalogue, no data or EA is present in 
> this archive, only the catalogue structure can be checked
> 
>    --------------------------------------------
>    0 item(s) treated
>    0 item(s) with error
>    0 item(s) ignored (excluded by filters)
>    --------------------------------------------
>    Total number of items considered: 0
>    --------------------------------------------
> 
> 
> dar -l <catalogue> -q:
> 
>    Archive version format               : 11.3
>    Compression algorithm used           : bzip2
>    Compression block size used          : 0
>    Symmetric key encryption used        : none
>    Asymmetric key encryption used       : none
>    Archive is signed                    : no
>    Sequential reading marks             : present
>    User comment                         :
>    Catalogue size in archive            : 35328204 bytes
> 
>    Archive is composed of 1 file(s)
>    File size: 35328416 bytes
>    The global data compression ratio is:   37%
> 
>    WARNING! This archive only contains the catalogue of another archive, 
> it can only be used as reference for differential backup or as rescue in 
> case of corruption of the original archive's content. You cannot restore 
> any data from this archive alone
> 
>    Archive of reference slicing:
>            Other slices: 10737418240 byte(s)
> 
>    in-place path: redacted
> 
>    CATALOGUE CONTENTS :
> 
>    total number of inode : 1589967
>    fully saved           : 1589967
>    binay delta patch     : 0
>    inode metadata only   : 0
>    distribution of inode(s)
>     - directories        : 236542
>     - plain files        : 1322581
>     - symbolic links     : 30513
>     - named pipes        : 16
>     - unix sockets       : 33
>     - character devices  : 280
>     - block devices      : 2
>     - Door entries       : 0
>    hard links information
>     - number of inode with hard link           : 49600
>     - number of reference to hard linked inodes: 109555
>    destroyed entries information
>       0 file(s) have been record as destroyed since backup of reference
> 
> 
> file arc.cat.1.dar arc.1.dar:
> 
>    arc.cat.1.dar: dar archive, label "7929f969 00000000 8a33" end slice
>    arc.1.dar:     dar archive, label "0503f969 00000000 8a33"
> 
> 
> Why -affs, -0, -l -q <cat> all report different file counts?

good question, as detailed above.

>