Re: [Dar-support] Need help determining sliced archive corruption
For full, incremental, compressed and encrypted backups or archives
Brought to you by:
edrusb
Menu
▾
▴
Re: [Dar-support] Need help determining sliced archive corruption
|
From: mannino <ma...@of...> - 2026-06-03 19:50:30
|
>> Other than a bug in dar itself (which I prefer to not consider thanks
>> to your exceptional job)
>
> exceptional or not, all software can have bugs and I never excluded it,
> the problem is the lack of reproducibility in your context...
That's true, until a new backup comes up with the same issue.
Full-system backups take a while to finish and they don't run every week
so let's stay tuned.
Meanwhile, you suggested we rule out disk errors, transfer errors and
wrong order of slices. But, I'm afraid, faulty RAM won't get us anywhere
either because, as I have mentioned in the first message, I have two
backups like that, created 1 month apart, from two different systems
(different hardware, locations, age, usage patterns, etc. - only backup
mechanism and Ubuntu/dar versions are the same). Both backups report
Sorry, file size is unknown at this step of the program.
to -l -0 -q; strace shows reading of the last slice, then some preceding
slice; file counts in -affs and -0 don't (fully) match:
dar -t -A -affs:
- 5689 x "compressed data CRC error"
- 1448 x "CRC error: data corruption"
- I haven't read through the entire list but it seems there are no
directories and corruptions affect files of various sizes from tiny to
huge and in different slices
--------------------------------------------
274471 item(s) treated
7137 item(s) with error
0 item(s) ignored (excluded by filters)
--------------------------------------------
Total number of items considered: 281608
--------------------------------------------
dar -t -0:
- 8 x "No escape mark found for that file"
--------------------------------------------
724494 item(s) treated
8 item(s) with error
0 item(s) ignored (excluded by filters)
--------------------------------------------
Total number of items considered: 724502
--------------------------------------------
dar -l <catalogue> -q:
Archive version format : 11.3
Compression algorithm used : bzip2
Compression block size used : 0
Symmetric key encryption used : none
Asymmetric key encryption used : none
Archive is signed : no
Sequential reading marks : present
User comment : redacted
Catalogue size in archive : 24538535 bytes
Archive is composed of 1 file(s)
File size: 24538747 bytes
The global data compression ratio is: 25%
WARNING! This archive only contains the catalogue of another archive,
it can only be used as reference for differential backup or as rescue in
case of corruption of the original archive's content. You cannot restore
any data from this archive alone
Archive of reference slicing:
Other slices: 2147483648 byte(s)
in-place path: redacted
CATALOGUE CONTENTS :
total number of inode : 724464
fully saved : 724464
binay delta patch : 0
inode metadata only : 0
distribution of inode(s)
- directories : 28009
- plain files : 687898
- symbolic links : 8403
- named pipes : 12
- unix sockets : 138
- character devices : 3
- block devices : 1
- Door entries : 0
hard links information
- number of inode with hard link : 21
- number of reference to hard linked inodes: 51
destroyed entries information
0 file(s) have been record as destroyed since backup of reference
This backup consists of 95 slices totaling 190 GiB.
Certainly it'd be too much of a coincidence if two servers exhibited
abnormal (RAM) behavior only during the creation of dar backups?
>> I still think it has something to do with wrong order/number of slices.
>
> to be honest, I don't see how mixing or even renumbering slices could
> cause the behavior you hit...
I only suggested that because dar doesn't detect out-of-sequence slices,
emitting symptomatic errors instead. If there were such a combination of
factors that could theoretically result in what I'm observing, I'd opt
for the mixed slices cause. But I surely trust you on this one... then
what else remains? What could potentially result in such archives?
>>> When testing the backup in sequential-read mode (-0/--sequential-read
>>> option) could you add the following option:
>>> -E "echo reading slice %N"
>>> and report the last slice being reported as being read by this added -E
>>> option, before the failure takes place?
>>
>> As per the above output, it says "reading slice 21".
>>
>>> Also could you tell me the number of the last slice available?
>>
>> There are arc.1.dar through arc.29.dar, the total of 29 slices.
>
> OK so you don't miss slices, but have those corrupted at different
> places.
But why does it report "Unexpected value while reading archive version"
after reading slice 21 after 29? It also happens with the other backup
except it reads slice 95 then 76.
> In sequential read mode, obviously, the archive is read in sequence,
> after having read the data of a files, dar reads right after the
> expected CRC and compares it with the one calculated on the retreived
> data and eventually reports a difference. Then it skips forward looking
> for the next tape mark and once again and the loop repeats...
>
> Here, corruption occurred in four different files (and slices), not only
> the very last files (in the last slice) as I had understood, which is
> not the pattern of a truncated slice.
It makes sense. It happens to the other backup too.
> Can you double check with for example memtest86+ ?
The system needs to be powered down for that, it's not something we can
do at this time...
>> 1. Archive UUID, as it's currently implemented, is perfect: (1) it
>> practically guarantees that one cannot mix up slices from different
>> archives; (2) dar itself checks that when testing; (3) can be
>> displayed with `file`:
>>
>> arc.10.dar: dar archive, label "82af71c2 00000000 fffff752"
>>
>> I could only suggest storing UUID of the reference archive in the -@
>> catalogue too (since -@ gets its own different UUID) and for dar
>> itself to provide some way to view this UUID. In any case, UUID helps
>> one work out the consistency among archives - but there's nothing
>> similar for working out slices for the archive itself, and it's a
>> problem for me.
>
> if you have a look at dar's documentation about the archive format, you
> will see that this is done with more flexibility:
> - internal name : links slices of an archive together
> - data name : links a catalog (or isolated catalog) to the data set to
> use. This prevents one trying to restore an archive with the help of an
> unrelated isolated catalog.
>
> http://dar.linux.free.fr/doc/Notes.html#archive_structure
I have glanced over that page but I have missed this. It's great and
eliminates any confusion in mismatched slices/catalogues. Now, if only
dar could provide some way to view this info, preferably in -0 mode for
potentially corrupted slices. For now, my only diagnostic tool is
`file`, and it's only reporting internal name it seems.
>> 2. Byte sizes of non/last slices would allow basic integrity check but
>> (1) `file` doesn't show them; (2) dar doesn't show them in -l -0 mode
>> (so unable to extract from invalid file); (3) dar -l -q works on
>> isolated catalogue but only reports non-last slice size - given our
>> problem here is likely last slice being truncated, it doesn't really
>> help.
>
> dar cannot know in advance how much slices will be necessary nor what
> will be the size of the last slice.
Of course it's streaming so I'm not suggesting storing total number of
slices in each slice. My suggestion here is to store each slice's size -
which is of course known beforehand. (I've written "non/last" but I
shall recede from the "last" part indeed.)
> The sar layer (which manages slices)
> processes a stream of bytes from upper layers (ciphering, compression,
> filtering, and so forth) which is generated on the fly from the
> filesystem to backup, the sar takes this flow and creates slices from it
> up to the time the flow dries up.
>
> What is stored at the beginning of *each slice* is the so called
> slice-layout:
> - first slice max size,
> - other slices max size,
> - slice header size
> - and slice format (2 formats so far).
>
> Only this could be shown in sequential-read mode, not what is displayed
> today in direct mode which adds to this the overall size of the archive
> and the size of the last slice (easy to get because in that mode this is
> the last slice that get read first).
Okay, the ability to view the details of slice-layout in -0 mode would
be welcome. I'd say any header info available to dar should be made
visible on equal terms.
>> 3. Slice number is what I truly miss. According to my tests, messing
>> with non-last slices (removing, duplicating, reordering) is usually
>> detected by dar as different types of data inconsistency so even if
>> you have all slices perfectly good and well, just in the wrong order,
>> you would think the backup is botched, and those errors would only be
>> misleading.
>
> the slice number is expected to be in the filename and it is not
> expected that users would change/renumber it...
Let me explain our storage strategy briefly. Backup files are held off
site for obvious reasons; the storage space is a set of buckets (could
be one or more per server but they're static) storing opaque blobs whose
only characteristics are acceptance date and data itself. Date is stored
for FIFO-like purging. Blobs can be moved or duplicated elsewhere behind
the scenes.
I understand it may not be the most elegant or efficient solution but
that's what we have now. Allowing free form metadata like file names
would complicate blob management. For backup purposes, this is not a
problem because each backup location is tied to a particular bucket
where archives and slices go in chronological order so by inspecting
date and data size one can clearly divide entries into individual
archives even if there are no file names. We store additional data in
dar's user comment field, but it's obviously archive-wise.
So essentially our backup directory looks like this:
402 | Sun Feb 22 11:43:58 AM UTC 2026 | 167511621 bytes
403 | Tue Mar 3 06:56:46 PM UTC 2026 | 10613869728 bytes
404 | Tue Mar 3 07:13:29 PM UTC 2026 | 10613869728 bytes
405 | Tue Mar 3 07:36:56 PM UTC 2026 | 635571924 bytes
> This is something unclear to me: you have this slice number information
> in the slice filename... isn't it the easiest way to know what slice
> number is a slice? And how much slice do one have? And also what is the
> latest slice?
My point is that in some scenarios file names are unavailable. Even if
not for our storage system, file names may be missing as a result of
data recovery from a faulty drive, for example. Sure, that should not
normally happen, and if you use par2 then it would fix the names for you
as well, but we're talking about last-chance recovery here, and that's
why dar has the -0 mode, so why not take it a step further with one or
two simple new fields in the header?
> What I don't understand is how you could
> accidentally miss the first slice, I mean how do you let dar accessing
> the slices as dar, even in sequential-read mode open a slice after a
> given filename and if it is not present it complains and waits.
To deploy or even inspect a backup, its blobs should be pulled from
storage somewhere dar can run, then renamed or symlinked to form
arc.N.dar. Human factor makes such mistakes possible. If you forgot one
or more leading slices yet they're named arc.1.dar, arc.2.dar, etc. -
dar won't "complain and wait", it'd proceed straight to throwing errors
at you.
>> Also, out of curiosity, I tried copying one slice twice and got a ton
>> of CRC errors.
> > > With the help of 4 extra bytes (= slice number) in each slice's
> header,
>> we would be able to identify the problem precisely.
>
> why only 4 bytes? This would limit the number of slices... even if today
> you think it is large enough, this is the same way of thinking that led
> to the year 2000 bug, the problem of year 2038, the upper and high
> memory in DOS some decades ago, the 2 GB files boundary (and need of
> large file support)... and so forth.
I understand your point. But even if we take 1 KiB per slice, 2^32
slices would add up to 4.4 TiB of data... and more "realistic" 1 MiB per
slice would be 4.5 PiB... but if that is a concern, you could use 64
bits, that would be (let me look that up on Wikipedia) around 2 exabytes
of data if each slice only held 1 byte of payload... I'm afraid we'd
need more than quantum computers to cross that boundary...
> A better approach could be to add a random fixed size field at end of
> slice and have this same random value at the beginning of the next. But
> I don't see the use case as mentioned earlier about number of slices in
> filenames.
If we're in the territory where 2^32 slices may not be enough, our
random numbers may not be random enough too, especially within millions
of slices... If we were choosing, I'd just use a fixed length numeric
field for the sake of simplicity. VLQ if you must, although I'd grade
that as an overkill...
>> Not only will it tell us slice order, it will also tell us the total
>> number of slices (since identifying last slice is trivial).
>
> not to my standpoint, as all slices may not be visible at a given time
> in particular when using removal media or remote storage.
Again, I'm not advocating storing *total* number of slices anywhere. My
point is that, with a simple "current slice index" field embedded in
each slice-layout, it's possible to eliminate all slice order-related
errors and usually even derive the total number of slices by taking last
slice's counter (given that last slice is smaller than others, detecting
it empirically is trivial). Sparing several bytes in each slice is a
small price to pay for this benefit, IMHO.
>> Storing total number of slices in the catalogue would take it even
>> further,
>
> this would break the ability to re-slice (dar_xform) an archive, once
> the isolated catalogue would have been created.
Okay, no worries, I take this suggestion back.
Thank you for still being on board with me on this!
|
