Re: [Dar-support] Need help determining sliced archive corruption

[Denis C. <dar...@fr...>]

Le 18/06/2026 à 22:45, mannino a écrit :
> Hi Denis,

Hi Mannino,

> 
> We have powered down both systems and ran memtest86+ for one full pass. 
> No RAM errors were detected.

Good thing. Thanks for having taken the time to test it. This means also 
that the hash (--hash) generated by dar should be significant, as they 
are computed in memory, thus you can further know whether storage has 
brought some corruption or not.

> 
> Just before the shutdown, one of those systems has created a full dar 
> archive with the same kind of corruption (direct mode doesn't work, -t 
> -0 reports 5 CRC errors). After powering it on and manually running full 
> dar backup, a valid archive was produced.

this is quite weird... we get close to the invocation of the cosmic 
particle that have changed something somewhere in some memory resident 
code... (I'm almost kidding!)

> 
> Since we seem to be out of ideas, I will have to keep an eye on the 
> backups coming from these two systems and let you know if it happens 
> again. I have enabled --hash and we now know that the RAM is fine so 
> we'd be a step forward if it does happen.

exactly

> 
> NB: typo on dar(1): "These hash files can be processes by md5sum" - s/ 
> processes/processed/

Oh! I'll will fix that shortly, thanks you for the feedback.

[...]

> 
>>>>> 2. Byte sizes of non/last slices would allow basic integrity check but
>>>>> (1) `file` doesn't show them; (2) dar doesn't show them in -l -0 mode
>>>>> (so unable to extract from invalid file); (3) dar -l -q works on
>>>>> isolated catalogue but only reports non-last slice size - given our
>>>>> problem here is likely last slice being truncated, it doesn't really
>>>>> help.
>>
>> dar has no need to show this information to work normally, either this
>> is the expected slice or a warning shows to the users for they take the
>> corrective action.
> 
> It already shows a lot of info (including sizes) in -l -q mode. Why not 
> make it available in sequential read mode?

because this information is not stored inside the archive data, but 
found in the metadata (here the filenames) with in addition the side 
effect of code modularity:

Follows the modularity explained in a long answer, skip over it if you 
don't want to known the libdar internals:

at the bottom floor takes place either:
- a sar object (segment and reassembly) which manages slices and exposes 
to the upper level a virtual large file
- or a trivial_sar object, which mimics the sar behavior when the data 
is read from a pipe
- a zapette which is a remote control of a sar or trivial_sar hosted in 
dar_slave program.

At the next floors, you will find some or all possible options:
- an encryption layer (of a cache layer if no encryption is used)
- escape layer which adds/read tape mark needed to interlace file's data 
and their metadata
- compression layer on top.

This stack of objects, reading from each others, is provided to an 
archive object (which realizes the floor 3 to 6 depending on the 
existence of layers below it) and this object just have access to:
- the unciphered, desinterlaced, decompressed data in the form of a 
virtual single big file containing all data and EA of all saved files.
- and as an option (if tape marks are present in the archive) an access 
to the escape layer to seek forward (floor 2) for the next mark of a 
given type.

In direct mode the listing process only involves reading the catalogue 
stored in memory, which has been loaded by opening the last slice at 
which time libdar can get the last slice number and its size, which none 
are stored in the archive, because storing them might change them if for 
example this addition would create a new slice.

In sequential read mode, at the top of the stack the archive avoids 
reading any filesystem metadata, it does not seek() forward or backward 
in this virtual big file, but just read(). Eventually it uses the handle 
provided by the escape layer to read forward up to the next tape mark of 
a given type (to fetch the next file metadata: filename, ownership, 
permission, dates...) stored in-band. But as the stack may be sitting on 
a named or even anonymous pipe for which there is not such filesystem 
metadata as the filename, the code avoids seeking() at a given position 
and works the same whatever the underneath stack content is (sliced, 
unsliced, pipe, remote control of dar_slave).

Moreover, both approaches share the same filtering code (its one level 
above the archive) and here the below layers are completely abstracted.

It only has access to the catalogue (or the escape_catalogue object in 
sequential-read mode) read the next entry, consider the filter used 
(based on filename, path, EA...) and eventually do something with it 
(listing or fetching the data for testing, restoration,...) before 
looping with the next archive entry.

What is abstracted at this layer is the way the saved file metatada is 
fetched:
- in direct mode this is already loaded in memory in the catalogue 
object, this is just a memory read in this data-structure.
- in sequential-read mode, the escape_catalogue when asked for the next 
entry, drives the escape_layer to read() up to the next mark, fetches 
reads the metadata and builds a catalogue in memory during this process.
(escape_catalogue inherits from class escape).

In sequential read mode when listing the details of an archive (-l -q), 
the offset and size of the last slice are not available because the 
below layer may be just a pipe nor a file and this code must works the 
same on pipes, single sliced, multi-sliced archive or remote control for 
dar_slave.

So yes, this is a long explanation to say I will not break this 
abstraction making a hole in all the floors to expose something that has 
not to be know at this a higher level.  Having abstraction between parts 
of code ease maintenance, evolution and reduce the risk of bugs. So I'm 
sorry I'll not break that.

Here I will rather hide behind the "design" principle and the fact a 
design is a choice, a policy.


[...]

>>>
>>> Okay, the ability to view the details of slice-layout in -0 mode would
>>> be welcome. I'd say any header info available to dar should be made
>>> visible on equal terms.
>>
>> I guess when you drive a car, you don't seen all the internal sensor
>> values (there is a lot of them in today's car), but over the classical
>> few indicators (oil temperature, speed, motor RPM,...) this is only when
>> something wrong happens that these internal sensors trigger an alarm on
>> your car dashboard in human understandable form, no?
> 
> This is why I'm always adding "in -0 mode". This mode is exactly for 
> "when something wrong happens", right?

No. The sequential-read mode is mainly to cope with devices that do not 
provide direct access mode to a given offset of a file. It was 
absolutely not part of the initial design but could be added --- thanks 
to libdar internal abstractions and modularity --- to address mainly 
tape support and as a side use case the reading through any type of 
piped data.

Today it is discouraged to remove tape marks while this would speed up 
the backup and reading processes, because tape marks (with the metadata 
that follows) can also be used to rebuild the internal catalogue of a 
truncated archive (-y option), it provides saved files metadata 
redundancy within a given archive.

However you should consider the sequential-read mode as restricted mode 
of operation (due to the tape/pipe context that is the target here). The 
direct (and original) mode of operation is much more efficient as it 
directly reads and only reads the needed data for restoration, testing, 
listing, diffing,... especially when using filtering mechanisms.

> 
>> For dar this is the same, first it would need a lot of work to expose
>> everything and second it would lead to an unreadable output, dar is
>> already too much talkative to my standpoint and I guess to the common
>> opinion.
> 
> Verbose header info output is meant for specific modes (-l -q), not 
> regular modus operandi.

I've added the following in the TODO list (at sourceforge) : expose the 
internal_name and data_name a given archive in that -l -q mode (or -l 
-v). This will be a per archive information and not per slice, as dar 
does not handle slices individually.

> 
>> Another mistake you apparently do is to consider dar would manipulate a
>> slice individually. Dar manipulate at once a set of slice, *by design*.
>> Using the sequential-read mode to feed a arbitrary slice in the hope to
>> get some information of it is just misuse of dar.
> 
> I never said I wanted to feed random slices to dar. I only said that 
> "the ability to view the details of slice-layout in -0 mode would
> be welcome". Currently, -l -q works but -l -q -0 doesn't, even if I feed 
> it the first slice.

OK, I misunderstood what you expected to do, sorry.

[...]

>>
>> If by "bucket" you mean those of an S3 storage or Azure Blob Storage,
>> why not using the metadata you can assign to these objects to record the
>> slice information???
>>
>> This would pretty make sense, no?
> 
> We don't use S3, we use a homebrew system. I only used the term "bucket" 
> as our system somewhat resembles an object storage.

I think your homebrew system is missing metadata support. Without it, 
you are stuck to the them same approach as tapes and all the 
restrictions it has. In fact, your homebrew system is probably even more 
restrictive than a tape system, as users usually add a sticker on their 
tapes when manually handled or the robot is hopefully able to locate 
which tape to load without reading each tape one by one to find the good 
one! -> metadata.

Now, I'm sorry to say so but, if you need to keep this metadata 
associated to the slice data (metadata dar relies on but which you 
suppressed when you removed the slice filenames) that's your duty to 
have your howebrew storage system carrying this need, either by:
- wrapping the slices within a file format that contains the slice 
number followed by the slice content (thought this would be inefficient 
as it would require to load and read a slice to know whether this is the 
good one),
- or add support for metadata beside the file/slice data to store this 
information (slice number).

Doing this inside dar format would lead to add a finite size field to 
store slice number, which would limit the number of slices a 
backup/archive can have, which I have managed to avoid since day 1 --- 
see infinint family classes to handle arbitrarily large integers, a post 
Y2K symptom ;).

[...]

>> So to face some data corruption, you ask all applications to add the
>> necessary redundant information that could be lost by the storage 
>> solution?
> 
> I merely propose to further recovery mode (-0) already exiting in dar. 
> Why would dar add tape marks if it trusted the storage? 

Not a lack of trust toward the storage, but a lack of feature: Tape 
marks are used to identify the next file just by using the read() system 
call, this is needed when the underlying media has not support to seek() 
to a given offset (pipes) or has so poor performance doing so that it is 
better using read (tape drives).

> Why would it add 
> UUIDs into every slice header if you insist on storage integrity? 

to avoid users (as I am) to make a mistake of mixing slices of different 
archives without being notified.

> Why 
> implement -0 in the first place with this mindset?

as written above -0 mode was added later to address tape and pipe support.

> 
>> I think you have to take dar for what it is: a tool that expects slices
>> names of that format <basename>.<slice number>.dar which is however
>> flexible enough for those slices be fed through pipes or other
>> mechanisms, like file regeneration based on blobs or other opaque data
>> structure as you described.  But this is on the storage side to handle
>> failures and provide the necessary environment to adapt to the tool
>> which are expected to be used... not the inverse.
> 
> Why store UUID then? <basename> is expected to be unique, right? Why not 
> use that, trust that the user will make sure to name all slices 
> correctly (if you require that user maintains <slice number> himself, 
> extending this requirement to <basename>.<slice number> is all the more 
> logical) and forego any checks for slice correspondence?

This is not because you have a seat belt in your car that your are free 
and safe to bump into any tree.

This feature is my seat belt to reduce my time spent in support request 
and to ease fault isolation: If users are notified when they mix slices 
of different backups/archives, the do not need to and usually don't come 
to me. Though, this may not catch all type of human error as a seat belt 
cannot prevent all injuries, better having it than nothing.

> 
>> Moreover reading blobs metadata is very light, while instead fetching
>> the many megabytes of a object/blob/slice to just pass it to a tool
>> (file command or dar) that would tells you this is not the good slice is
>> un-optimal at all and quite costly ($), no?
> 
> It is, but we avoid that by examining date of admission since there's a 
> predictable time gap between incoming slices and archives. I use `file` 
> after fetching the data only when investigating issues.

Sorry to say so again, but that's your choice and design (and policy), I 
respect it and I will no more discuss about the metadata support I think 
your homebrew storage is lacking.

On my side too I have my design, choice and policy with dar/libdar: I 
don't want to support user specific/homebrew storage systems.


[...]
>>
>> same remark as above: the bucket/object/blob metadata is the logical
>> place of the slice information you remove from filesystem metadata (=
>> filename).
>>
>> You have to find a way to translate from filesystem metadata to
>> S3/bucket/objet/blob metadata and vice versa, this is not the purpose of
>> to do that very context specific task.
> 
> You seem to be constantly misinterpreting my words in spite of me 
> explicitly stating that all my suggestions are concerning recovery mode, 
> not normal operation. 

it was not intentional, sorry

> We have been using our (arguably weird) storage 
> system along with dar for over a decade and it works just fine - without 
> dar's built-in slice counters, extra UUIDs, etc. But our current problem 
> highlighted what I believe is a shortcoming on dar's diagnostic/recovery 
> side and I simply propose some improvements (very modest really).

As mentioned above, I have added a feature request in the todo list to 
expose the internal_name and data_names of an archive when using -l -q 
(or -l -v),

> I even 
> mentioned that it might come in handy for other cases (broken FAT with 
> unavailable file names). Again, it seems very in-line with the existing 
> tape marks and sequential reading mode.

If you need a robust filesystem or face to filesystem corruptions, this 
is a filesystem feature that is needed (RAID, journals, ...) but for 
those as I am that do not want so robust filesystem, we do backups ;).

Adding a slice number inside each slice would not help much: if a file 
(slice) is retrieved in different blocks/files, due to filesystem 
corruption, it will not tell you how to stick them back together, 
assuming no part is missing, which is improbable in that context.

> 
>>>> why only 4 bytes? This would limit the number of slices... even if 
>>>> today
>>>> you think it is large enough, this is the same way of thinking that led
>>>> to the year 2000 bug, the problem of year 2038, the upper and high
>>>> memory in DOS some decades ago, the 2 GB files boundary (and need of
>>>> large file support)... and so forth.
>>>
>>> I understand your point. But even if we take 1 KiB per slice, 2^32
>>> slices would add up to 4.4 TiB of data... and more "realistic" 1 MiB
>>> per slice would be 4.5 PiB... but if that is a concern, you could use
>>> 64 bits, that would be (let me look that up on Wikipedia) around 2
>>> exabytes of data if each slice only held 1 byte of payload... I'm
>>> afraid we'd need more than quantum computers to cross that boundary...
>>
>> an so now, the smallest slice which is a few tens of bytes would becomes
>> a few kilobytes... what a waste...
> 
> What's the practical reason for having over 4 billions of slices, each a 
> few tens of bytes? Besides, the counter may be rotating, it's only a 
> hint anyway.

this is what they say when storing years on 2 digits last century, or 
sotring dates as a number of seconds since 1969 ended in a 32 bit fields 
(year 2038 problem, we are getting to it...), and so forth.

I would reverse the question: why limiting the number of slices and thus 
the overall backup size (because slice size depends on the underlying 
filesystem ability to support them) when you can avoid it?

> 
>>>> A better approach could be to add a random fixed size field at end of
>>>> slice and have this same random value at the beginning of the next. But
>>>> I don't see the use case as mentioned earlier about number of slices in
>>>> filenames.
>>>
>>> If we're in the territory where 2^32 slices may not be enough, our
>>> random numbers may not be random enough too, especially within
>>> millions of slices... If we were choosing, I'd just use a fixed length
>>> numeric field for the sake of simplicity. VLQ if you must, although
>>> I'd grade that as an overkill...
>>
>> why a "fixed length numeric field" would be enough in the "2^32
>> territory" while a "random number" (which size is still undefined) would
>> not?
> 
> I assumed that both fields would be of the same size because a random 
> number field that's wider than a fixed number field looks pointless to 
> me - it takes more space, it doesn't guarantee uniqueness even within 
> its range, it doesn't allow comparing positions, etc. What's the 
> advantage of such a random field?

the advantage in that particular case is the fixes width size, which 
here is interesting to know at which time, while writing down a slice, 
you stop adding data to start adding this random field.

It is also necessary at reading time to translate the location of a 
saved file data, stored as an offset in the virtual big file I explained 
above, into a slice number and offset in that slice. Having variable 
width field to store slices number would make it impossible without 
reading all slices header and trailers... and storing slice sizes in a 
fixed width field would look like one shooting at his foot.

> 
>> what length (in byte, for example) would you give to this "simple
>> current slice index"?
> 
> Practically, 2 bytes and allow it to overflow. I don't have statistics 

that's precisely the point: even the following references do not reflect 
the variety of use cases dar currently addresses:

http://dar.linux.free.fr/doc/References.html

I will this not limit this to address a user specific context.

> but I don't think 65k+ slices per archive are a norm, and even if this 
> practice exists somewhere, a rotating 16-bit counter would make it 2^16 
> times less likely that an incorrect ordering of slices would go 
> undetected (and 2^16 times easier to determine the number of any given 
> slice) compared to when there's no such counter at all. On dar's side, 
> the check would simply go from 0 to 2^16, then overflow and repeat.
> 

Regards,
Denis