Re: [Dar-support] Need help determining sliced archive corruption

[Denis C. <dar...@fr...>]

Le 03/06/2026 à 21:49, mannino a écrit :

Hi,

I have finally found some time to answer the feature related part or 
your email:

>>> 1. Archive UUID, as it's currently implemented, is perfect: (1) it
>>> practically guarantees that one cannot mix up slices from different
>>> archives; (2) dar itself checks that when testing; (3) can be
>>> displayed with `file`:
>>>
>>> arc.10.dar: dar archive, label "82af71c2 00000000 fffff752"
>>>
>>> I could only suggest storing UUID of the reference archive in the -@
>>> catalogue too (since -@ gets its own different UUID) and for dar
>>> itself to provide some way to view this UUID. In any case, UUID helps
>>> one work out the consistency among archives - but there's nothing
>>> similar for working out slices for the archive itself, and it's a
>>> problem for me.

it is partially the case in 2.8.x and is now fully the case in 2.9.x 
(this is the feature I was working on these last days). So now an 
isolated catalogue has all the needed information to operate the backup 
of reference and does not more need the first or last slice to know the 
slice-layout, archive format, compression algo, ciphering salt and other 
parameters of that sort.

>>
>> if you have a look at dar's documentation about the archive format, you
>> will see that this is done with more flexibility:
>> - internal name : links slices of an archive together
>> - data name : links a catalog (or isolated catalog) to the data set to
>> use. This prevents one trying to restore an archive with the help of an
>> unrelated isolated catalog.
>>
>> http://dar.linux.free.fr/doc/Notes.html#archive_structure
> 
[...]
> Now, if only 
> dar could provide some way to view this info, preferably in -0 mode for 
> potentially corrupted slices. 
this is in the pipe of features: the data_name and internal_name will be 
shown when listing the detail of a backup.

However this information will not tell you whether a slice is corrupted 
or not. The slice header takes a few tens of bytes at the beginning of 
each slice (and 1 byte is used at the end of each slice). Corruption 
taking place in the middle will be reported the same as today (CRC of 
the data or random other error depending on the place the corruption 
occurs).

Last, dar manipulates a backup (= a set of slices as a whole) it will 
report this information as fetched from the first slice seen for the 
listing operation (the first in sequential more, the last in default 
mode) any other slices involved in the operation will either match both 
slices attributes or an error will be reported as of today in that same 
situation.

If you want the file command to display this information for any slice 
individually, you could ask such feature to the maintainer of this command.

[...]
> 
>>> 2. Byte sizes of non/last slices would allow basic integrity check but
>>> (1) `file` doesn't show them; (2) dar doesn't show them in -l -0 mode
>>> (so unable to extract from invalid file); (3) dar -l -q works on
>>> isolated catalogue but only reports non-last slice size - given our
>>> problem here is likely last slice being truncated, it doesn't really
>>> help.

dar has no need to show this information to work normally, either this 
is the expected slice or a warning shows to the users for they take the 
corrective action.

>>
>> dar cannot know in advance how much slices will be necessary nor what
>> will be the size of the last slice.
> 
> Of course it's streaming so I'm not suggesting storing total number of 
> slices in each slice. My suggestion here is to store each slice's size - 
> which is of course known beforehand. (I've written "non/last" but I 
> shall recede from the "last" part indeed.)

All slices except the last one of a backup (= set of slices) have a 
known size which is described in the slice layout already stored at the 
beginning of each slice.

> 
>> The sar layer (which manages slices)
>> processes a stream of bytes from upper layers (ciphering, compression,
>> filtering, and so forth) which is generated on the fly from the
>> filesystem to backup, the sar takes this flow and creates slices from it
>> up to the time the flow dries up.
>>
>> What is stored at the beginning of *each slice* is the so called
>> slice-layout:
>> - first slice max size,
>> - other slices max size,
>> - slice header size
>> - and slice format (2 formats so far).
>>
>> Only this could be shown in sequential-read mode, not what is displayed
>> today in direct mode which adds to this the overall size of the archive
>> and the size of the last slice (easy to get because in that mode this is
>> the last slice that get read first).
> 
> Okay, the ability to view the details of slice-layout in -0 mode would 
> be welcome. I'd say any header info available to dar should be made 
> visible on equal terms.

I guess when you drive a car, you don't seen all the internal sensor 
values (there is a lot of them in today's car), but over the classical 
few indicators (oil temperature, speed, motor RPM,...) this is only when 
something wrong happens that these internal sensors trigger an alarm on 
your car dashboard in human understandable form, no?

For dar this is the same, first it would need a lot of work to expose 
everything and second it would lead to an unreadable output, dar is 
already too much talkative to my standpoint and I guess to the common 
opinion.

Another mistake you apparently do is to consider dar would manipulate a 
slice individually. Dar manipulate at once a set of slice, *by design*.
Using the sequential-read mode to feed a arbitrary slice in the hope to 
get some information of it is just misuse of dar.

> 
>>> 3. Slice number is what I truly miss. According to my tests, messing
>>> with non-last slices (removing, duplicating, reordering) is usually
>>> detected by dar as different types of data inconsistency so even if
>>> you have all slices perfectly good and well, just in the wrong order,
>>> you would think the backup is botched, and those errors would only be
>>> misleading.
>>
>> the slice number is expected to be in the filename and it is not
>> expected that users would change/renumber it...
> 
> Let me explain our storage strategy briefly. Backup files are held off 
> site for obvious reasons; the storage space is a set of buckets (could 
> be one or more per server but they're static) storing opaque blobs whose 
> only characteristics are acceptance date and data itself. Date is stored 
> for FIFO-like purging. Blobs can be moved or duplicated elsewhere behind 
> the scenes.
> 
> I understand it may not be the most elegant or efficient solution but 
> that's what we have now. Allowing free form metadata like file names 
> would complicate blob management. For backup purposes, this is not a 
> problem because each backup location is tied to a particular bucket 
> where archives and slices go in chronological order so by inspecting 
> date and data size one can clearly divide entries into individual 
> archives even if there are no file names. We store additional data in 
> dar's user comment field, but it's obviously archive-wise.

If by "bucket" you mean those of an S3 storage or Azure Blob Storage, 
why not using the metadata you can assign to these objects to record the 
slice information???

This would pretty make sense, no?

> 
> So essentially our backup directory looks like this:
> 
>    402 | Sun Feb 22 11:43:58 AM UTC 2026 | 167511621 bytes
>    403 | Tue Mar  3 06:56:46 PM UTC 2026 | 10613869728 bytes
>    404 | Tue Mar  3 07:13:29 PM UTC 2026 | 10613869728 bytes
>    405 | Tue Mar  3 07:36:56 PM UTC 2026 | 635571924 bytes
> 
>> This is something unclear to me: you have this slice number information
>> in the slice filename... isn't it the easiest way to know what slice
>> number is a slice? And how much slice do one have? And also what is the
>> latest slice?
> 
> My point is that in some scenarios file names are unavailable. Even if 
> not for our storage system, file names may be missing as a result of 
> data recovery from a faulty drive, for example. Sure, that should not 
> normally happen, and if you use par2 then it would fix the names for you 
> as well, but we're talking about last-chance recovery here, and that's 
> why dar has the -0 mode, so why not take it a step further with one or 
> two simple new fields in the header?

So to face some data corruption, you ask all applications to add the 
necessary redundant information that could be lost by the storage solution?

I think you have to take dar for what it is: a tool that expects slices 
names of that format <basename>.<slice number>.dar which is however 
flexible enough for those slices be fed through pipes or other 
mechanisms, like file regeneration based on blobs or other opaque data 
structure as you described.  But this is on the storage side to handle 
failures and provide the necessary environment to adapt to the tool 
which are expected to be used... not the inverse.

Moreover reading blobs metadata is very light, while instead fetching 
the many megabytes of a object/blob/slice to just pass it to a tool 
(file command or dar) that would tells you this is not the good slice is 
un-optimal at all and quite costly ($), no?

> 
>> What I don't understand is how you could
>> accidentally miss the first slice, I mean how do you let dar accessing
>> the slices as dar, even in sequential-read mode open a slice after a
>> given filename and if it is not present it complains and waits.
> 
> To deploy or even inspect a backup, its blobs should be pulled from 
> storage somewhere dar can run, then renamed or symlinked to form 
> arc.N.dar. Human factor makes such mistakes possible. If you forgot one 
> or more leading slices yet they're named arc.1.dar, arc.2.dar, etc. - 
> dar won't "complain and wait", it'd proceed straight to throwing errors 
> at you.

same remark as above: the bucket/object/blob metadata is the logical 
place of the slice information you remove from filesystem metadata (= 
filename).

You have to find a way to translate from filesystem metadata to 
S3/bucket/objet/blob metadata and vice versa, this is not the purpose of 
to do that very context specific task.

> 
>>> Also, out of curiosity, I tried copying one slice twice and got a ton
>>> of CRC errors.
>>  > > With the help of 4 extra bytes (= slice number) in each slice's
>> header,
>>> we would be able to identify the problem precisely.
>>
>> why only 4 bytes? This would limit the number of slices... even if today
>> you think it is large enough, this is the same way of thinking that led
>> to the year 2000 bug, the problem of year 2038, the upper and high
>> memory in DOS some decades ago, the 2 GB files boundary (and need of
>> large file support)... and so forth.
> 
> I understand your point. But even if we take 1 KiB per slice, 2^32 
> slices would add up to 4.4 TiB of data... and more "realistic" 1 MiB per 
> slice would be 4.5 PiB... but if that is a concern, you could use 64 
> bits, that would be (let me look that up on Wikipedia) around 2 exabytes 
> of data if each slice only held 1 byte of payload... I'm afraid we'd 
> need more than quantum computers to cross that boundary...

an so now, the smallest slice which is a few tens of bytes would becomes 
a few kilobytes... what a waste...

> 
>> A better approach could be to add a random fixed size field at end of
>> slice and have this same random value at the beginning of the next. But
>> I don't see the use case as mentioned earlier about number of slices in
>> filenames.
> 
> If we're in the territory where 2^32 slices may not be enough, our 
> random numbers may not be random enough too, especially within millions 
> of slices... If we were choosing, I'd just use a fixed length numeric 
> field for the sake of simplicity. VLQ if you must, although I'd grade 
> that as an overkill...

why a "fixed length numeric field" would be enough in the "2^32 
territory" while a "random number" (which size is still undefined) would 
not?

I don't argue for this feature or not just don't understand your 
reasoning...

> 
>>> Not only will it tell us slice order, it will also tell us the total
>>> number of slices (since identifying last slice is trivial).
>>
>> not to my standpoint, as all slices may not be visible at a given time
>> in particular when using removal media or remote storage.
> 
> Again, I'm not advocating storing *total* number of slices anywhere. My 
> point is that, with a simple "current slice index" field embedded in 
> each slice-layout, it's possible to eliminate all slice order-related 
> errors and usually even derive the total number of slices by taking last 
> slice's counter (given that last slice is smaller than others, detecting 
> it empirically is trivial). Sparing several bytes in each slice is a 
> small price to pay for this benefit, IMHO.

what length (in byte, for example) would you give to this "simple 
current slice index"?

> 
>>>  Storing total number of slices in the catalogue would take it even
>>> further,
>>
>> this would break the ability to re-slice (dar_xform) an archive, once
>> the isolated catalogue would have been created.
> 
> Okay, no worries, I take this suggestion back.
> 
> Thank you for still being on board with me on this!
> 

Cheers,
Denis