Re: [Dar-support] Need help determining sliced archive corruption

[mannino <ma...@of...>]

>> Just before the shutdown, one of those systems has created a full dar
>> archive with the same kind of corruption (direct mode doesn't work, -t
>> -0 reports 5 CRC errors). After powering it on and manually running
>> full dar backup, a valid archive was produced.
>
> this is quite weird... we get close to the invocation of the cosmic
> particle that have changed something somewhere in some memory resident
> code... (I'm almost kidding!)

I actually thought about that but for that to happen at least 3 times in 
2 different systems/locations and time...

>> 2. Byte sizes of non/last slices would allow basic integrity check
>> but
>> (1) `file` doesn't show them; (2) dar doesn't show them in -l -0 mode
>> (so unable to extract from invalid file); (3) dar -l -q works on
>> isolated catalogue but only reports non-last slice size - given our
>> problem here is likely last slice being truncated, it doesn't really
>> help.
> because this information is not stored inside the archive data, but
> found in the metadata (here the filenames) with in addition the side
> effect of code modularity:

Thanks for the explanation. You have previously said that slice-layout 
holds slice sizes and apparently it's considered part of "archive data"...


> The sar layer (which manages slices)
> processes a stream of bytes from upper layers (ciphering, compression,
> filtering, and so forth) which is generated on the fly from the
> filesystem to backup, the sar takes this flow and creates slices from it
> up to the time the flow dries up.
>
> What is stored at the beginning of *each slice* is the so called
> slice-layout:
> - first slice max size,
> - other slices max size,
> - slice header size
> - and slice format (2 formats so far).
>
> Only this could be shown in sequential-read mode, not what is displayed
> today in direct mode which adds to this the overall size of the archive
> and the size of the last slice (easy to get because in that mode this is
> the last slice that get read first).

Do I understand it correctly that the abstraction doesn't allow 
reading/provide access to slice-layout data, making it impossible to 
view slice header in -l -q -0 (or any other) mode? Isn't at least the 
first slice's header exposed? Or the main concern here is output 
verbosity (see next)?

>>>> Okay, the ability to view the details of slice-layout in -0 mode would
>>>> be welcome. I'd say any header info available to dar should be made
>>>> visible on equal terms.
>>>
>>> I guess when you drive a car, you don't seen all the internal sensor
>>> values (there is a lot of them in today's car), but over the classical
>>> few indicators (oil temperature, speed, motor RPM,...) this is only when
>>> something wrong happens that these internal sensors trigger an alarm on
>>> your car dashboard in human understandable form, no?
>>
>> This is why I'm always adding "in -0 mode". This mode is exactly for
>> "when something wrong happens", right?
>
> No. The sequential-read mode is mainly to cope with devices that do not
> provide direct access mode to a given offset of a file.

[...]

> Today it is discouraged to remove tape marks while this would speed up
> the backup and reading processes, because tape marks (with the metadata
> that follows) can also be used to rebuild the internal catalogue of a
> truncated archive (-y option), it provides saved files metadata
> redundancy within a given archive.

Okay, -0 is "mainly" for unseekable devices but you state that tape 
marks are recommended even in normal backups for fault 
tolerance/redundancy meaning that -0's secondary purpose is still "when 
something wrong happens". I think I've also seen in the docs that for 
recovery, you're expected to try -0 first, then -al, then -al -0 as last 
resort. In this case, provided that the abstraction permits, why not 
make extra info (slice sizes here) available to the user in -0 mode? 
(It's useless for -al since user has to supply headers' data himself.) 
dar already provides a dozen of -v(erbose) options, if output is a 
concern then it may be brought under -vm or something.

> I've added the following in the TODO list (at sourceforge) : expose the
> internal_name and data_name a given archive in that -l -q mode (or -l
> -v). This will be a per archive information and not per slice, as dar
> does not handle slices individually.

Thanks, I'm sure this will be really handy sometimes.

> I think your homebrew system is missing metadata support. Without it,
> you are stuck to the them same approach as tapes and all the
> restrictions it has. In fact, your homebrew system is probably even more
> restrictive than a tape system, as users usually add a sticker on their
> tapes when manually handled or the robot is hopefully able to locate
> which tape to load without reading each tape one by one to find the good
> one! -> metadata.

I wouldn't argue about our system, in part because you're correct, in 
part because it's not something we can/want to change, as long as it 
works. Like I said, current nuisance in locating slices doesn't warrant 
extending it. I still stand by the fact that my argument for slice 
counters is valid outside of our particular context, otherwise I 
wouldn't have brought it up.

> Doing this inside dar format would lead to add a finite size field to
> store slice number, which would limit the number of slices a
> backup/archive can have, which I have managed to avoid since day 1 ---
> see infinint family classes to handle arbitrarily large integers, a post
> Y2K symptom ;).

I understand your point that VLQ, i.e. arbitrarily large integer to 
store slice number is inefficient. I understand that you don't want to 
put a hard limit on the slice count either. But I don't understand why 
you argue against a rotating slice counter.

You have said earlier that even with the UUID, there is some chance of 
getting the same value:

> There is little chance (not to say no chance at all), that two
> different backups have the same internal name.

So UUIDs do not guarantee 100% uniqueness (and take way more space than 
a counter I suggest). Why do you oppose a fixed-width rotating counter 
so much then?

>>> what length (in byte, for example) would you give to this
>>> "simple current slice index"?
>>
>> Practically, 2 bytes and allow it to overflow. I don't have
>> statistics
>
> that's precisely the point: even the following references do not
> reflect the variety of use cases dar currently addresses:
>
> http://dar.linux.free.fr/doc/References.html

I agree that we lack exhaustive statistics and there may be exotic 
cases. But dar has settled on some practical defaults. Slice counter is 
the same kind of non-definite measure as is the archive label (UUID). 
UUIDs can duplicate; there may be more slices than 2^16 (2^24, 2^32, 
...). Does this fact change their utility?

>> Why would it add UUIDs into every slice header if you insist on
>> storage integrity?
>
> to avoid users (as I am) to make a mistake of mixing slices of different
> archives without being notified.

You oppose me in saying that one cannot reorder slices (even by 
accident) and that file name is paramount, but you still state that one 
can mix slices from different archives? Doesn't it require a comparable 
level of mistake? To accidentally add a slice from another archive, one 
would need to rename it specifically, right? And not just change 
<basename> but also <slice number>, otherwise there'd be a noticeable 
gap or often an overwrite prompt? Even if two directories have slices 
with the same <basename> and you move a slice from one to another, 
<slice number>s would still clash.

arc.1.dar
arc.2.dar
arc.3.dar <- real last slice
arc.23.dar <- added-in slice from another archive

Also, it's usually obvious when something like that happens because last 
slice is smaller than others so you'd see 2 last slices of different 
sizes than the rest - unlike when you reorder slices.

 From this standpoint, to my argument: why one cannot accidentally 
remove one of the slices (first or last is easiest) or mess up with 
renaming them (if we're talking about renaming) or duplicating them so 
that their <slice number> change but <basename> doesn't? Or, if it's 
directories with slices, fail to move first/last slices to new location?

>> Why store UUID then? <basename> is expected to be unique, right? Why
>> not use that, trust that the user will make sure to name all slices
>> correctly (if you require that user maintains <slice number> himself,
>> extending this requirement to <basename>.<slice number> is all the
>> more logical) and forego any checks for slice correspondence?
>
> This is not because you have a seat belt in your car that your are free
> and safe to bump into any tree.
>
> This feature is my seat belt to reduce my time spent in support request
> and to ease fault isolation: If users are notified when they mix slices
> of different backups/archives, the do not need to and usually don't come
> to me. Though, this may not catch all type of human error as a seat belt
> cannot prevent all injuries, better having it than nothing.

Excellent point. Why doesn't it apply to slice numbers?

> I would reverse the question: why limiting the number of slices and
> thus the overall backup size (because slice size depends on the
> underlying filesystem ability to support them) when you can avoid
> it?

Like I have repeatedly stated, the counter may be rotating. We are not 
forced to make it bulletproof just like archive label isn't bulletproof.

>>> why a "fixed length numeric field" would be enough in the "2^32
>>> territory" while a "random number" (which size is still
>>> undefined) would not?
>>
>> I assumed that both fields would be of the same size because a
>> random number field that's wider than a fixed number field looks
>> pointless to me - it takes more space, it doesn't guarantee
>> uniqueness even within its range, it doesn't allow comparing
>> positions, etc. What's the advantage of such a random field?
>
> the advantage in that particular case is the fixes width size, which
> here is interesting to know at which time, while writing down a
> slice, you stop adding data to start adding this random field.

I understand your point about variable-length fields. But my question 
was how a fixed-width random field is better than a fixed-width rotating 
integer counter. Both do not guarantee uniqueness but the latter has 
advantages over the former.

>> I even mentioned that it might come in handy for other cases (broken
>> FAT with unavailable file names). Again, it seems very in-line with
>> the existing tape marks and sequential reading mode.
>
> If you need a robust filesystem or face to filesystem corruptions, this
> is a filesystem feature that is needed (RAID, journals, ...) but for
> those as I am that do not want so robust filesystem, we do backups ;).

You made me understand that tape marks' and -0's potential for recovery 
is an (undesired) side effect. But why does dar have the lax mode then? 
Its only purpose is to fight FS corruptions and lack of backups.

> Adding a slice number inside each slice would not help much: if a file
> (slice) is retrieved in different blocks/files, due to filesystem
> corruption, it will not tell you how to stick them back together,
> assuming no part is missing, which is improbable in that context.

 From my experience, and I have tended to a recovery of a dozen or so 
faulty drives including a SSDs using Windows and Linux file systems, the 
recovery process of course cannot guess block allocation but usually FAT 
isn't obliterated entirely - recovery often still produces consistent 
result for some files. Yet, even if you managed to get a consistent 
file, you may not manage to get its file name.

Moreover, one of the popular reasons for recovery is when a directory 
was accidentally deleted or a FS was formatted. In this case file 
entries don't (immediately) disappear and it's possible to fully recover 
deleted files, but again, their file names may be mangled, fully or 
partially.

I'm sure you're aware of the renowned FOUND.000 directory and how it's 
populated. https://en.wikipedia.org/wiki/Found.000#cite_ref-14
Many times, I have personally seen entirely consistent files recovered 
in that directory, but without file names. It still happens with USB 
sticks, and I can easily imagine dar slices residing on one (it may be 
irresponsible on the user's behalf but dar already has mechanisms to 
address irresponsibility, as we've hopefully established).