Re: [Dar-support] Need help determining sliced archive corruption

[mannino <ma...@of...>]

Hi Denis,

We have powered down both systems and ran memtest86+ for one full pass. 
No RAM errors were detected.

Just before the shutdown, one of those systems has created a full dar 
archive with the same kind of corruption (direct mode doesn't work, -t 
-0 reports 5 CRC errors). After powering it on and manually running full 
dar backup, a valid archive was produced.

Since we seem to be out of ideas, I will have to keep an eye on the 
backups coming from these two systems and let you know if it happens 
again. I have enabled --hash and we now know that the RAM is fine so 
we'd be a step forward if it does happen.

NB: typo on dar(1): "These hash files can be processes by md5sum" - 
s/processes/processed/

> it is partially the case in 2.8.x and is now fully the case in 2.9.x
> (this is the feature I was working on these last days). So now an
> isolated catalogue has all the needed information to operate the backup
> of reference and does not more need the first or last slice to know the
> slice-layout, archive format, compression algo, ciphering salt and other
> parameters of that sort.

Excellent addition.

>>>> 2. Byte sizes of non/last slices would allow basic integrity check but
>>>> (1) `file` doesn't show them; (2) dar doesn't show them in -l -0 mode
>>>> (so unable to extract from invalid file); (3) dar -l -q works on
>>>> isolated catalogue but only reports non-last slice size - given our
>>>> problem here is likely last slice being truncated, it doesn't really
>>>> help.
>
> dar has no need to show this information to work normally, either this
> is the expected slice or a warning shows to the users for they take the
> corrective action.

It already shows a lot of info (including sizes) in -l -q mode. Why not 
make it available in sequential read mode?

>>> The sar layer (which manages slices)
>>> processes a stream of bytes from upper layers (ciphering, compression,
>>> filtering, and so forth) which is generated on the fly from the
>>> filesystem to backup, the sar takes this flow and creates slices from it
>>> up to the time the flow dries up.
>>>
>>> What is stored at the beginning of *each slice* is the so called
>>> slice-layout:
>>> - first slice max size,
>>> - other slices max size,
>>> - slice header size
>>> - and slice format (2 formats so far).
>>>
>>> Only this could be shown in sequential-read mode, not what is displayed
>>> today in direct mode which adds to this the overall size of the archive
>>> and the size of the last slice (easy to get because in that mode this is
>>> the last slice that get read first).
>>
>> Okay, the ability to view the details of slice-layout in -0 mode would
>> be welcome. I'd say any header info available to dar should be made
>> visible on equal terms.
>
> I guess when you drive a car, you don't seen all the internal sensor
> values (there is a lot of them in today's car), but over the classical
> few indicators (oil temperature, speed, motor RPM,...) this is only when
> something wrong happens that these internal sensors trigger an alarm on
> your car dashboard in human understandable form, no?

This is why I'm always adding "in -0 mode". This mode is exactly for 
"when something wrong happens", right?

> For dar this is the same, first it would need a lot of work to expose
> everything and second it would lead to an unreadable output, dar is
> already too much talkative to my standpoint and I guess to the common
> opinion.

Verbose header info output is meant for specific modes (-l -q), not 
regular modus operandi.

> Another mistake you apparently do is to consider dar would manipulate a
> slice individually. Dar manipulate at once a set of slice, *by design*.
> Using the sequential-read mode to feed a arbitrary slice in the hope to
> get some information of it is just misuse of dar.

I never said I wanted to feed random slices to dar. I only said that 
"the ability to view the details of slice-layout in -0 mode would
be welcome". Currently, -l -q works but -l -q -0 doesn't, even if I feed 
it the first slice.

>>>> 3. Slice number is what I truly miss. According to my tests, messing
>>>> with non-last slices (removing, duplicating, reordering) is usually
>>>> detected by dar as different types of data inconsistency so even if
>>>> you have all slices perfectly good and well, just in the wrong order,
>>>> you would think the backup is botched, and those errors would only be
>>>> misleading.
>>>
>>> the slice number is expected to be in the filename and it is not
>>> expected that users would change/renumber it...
>>
>> Let me explain our storage strategy briefly. Backup files are held off
>> site for obvious reasons; the storage space is a set of buckets (could
>> be one or more per server but they're static) storing opaque blobs
>> whose only characteristics are acceptance date and data itself. Date
>> is stored for FIFO-like purging. Blobs can be moved or duplicated
>> elsewhere behind the scenes.
>>
>> I understand it may not be the most elegant or efficient solution but
>> that's what we have now. Allowing free form metadata like file names
>> would complicate blob management. For backup purposes, this is not a
>> problem because each backup location is tied to a particular bucket
>> where archives and slices go in chronological order so by inspecting
>> date and data size one can clearly divide entries into individual
>> archives even if there are no file names. We store additional data in
>> dar's user comment field, but it's obviously archive-wise.
>
> If by "bucket" you mean those of an S3 storage or Azure Blob Storage,
> why not using the metadata you can assign to these objects to record the
> slice information???
>
> This would pretty make sense, no?

We don't use S3, we use a homebrew system. I only used the term "bucket" 
as our system somewhat resembles an object storage.

>>> This is something unclear to me: you have this slice number information
>>> in the slice filename... isn't it the easiest way to know what slice
>>> number is a slice? And how much slice do one have? And also what is the
>>> latest slice?
>>
>> My point is that in some scenarios file names are unavailable. Even if
>> not for our storage system, file names may be missing as a result of
>> data recovery from a faulty drive, for example. Sure, that should not
>> normally happen, and if you use par2 then it would fix the names for
>> you as well, but we're talking about last-chance recovery here, and
>> that's why dar has the -0 mode, so why not take it a step further with
>> one or two simple new fields in the header?
>
> So to face some data corruption, you ask all applications to add the
> necessary redundant information that could be lost by the storage solution?

I merely propose to further recovery mode (-0) already exiting in dar. 
Why would dar add tape marks if it trusted the storage? Why would it add 
UUIDs into every slice header if you insist on storage integrity? Why 
implement -0 in the first place with this mindset?

> I think you have to take dar for what it is: a tool that expects slices
> names of that format <basename>.<slice number>.dar which is however
> flexible enough for those slices be fed through pipes or other
> mechanisms, like file regeneration based on blobs or other opaque data
> structure as you described.  But this is on the storage side to handle
> failures and provide the necessary environment to adapt to the tool
> which are expected to be used... not the inverse.

Why store UUID then? <basename> is expected to be unique, right? Why not 
use that, trust that the user will make sure to name all slices 
correctly (if you require that user maintains <slice number> himself, 
extending this requirement to <basename>.<slice number> is all the more 
logical) and forego any checks for slice correspondence?

> Moreover reading blobs metadata is very light, while instead fetching
> the many megabytes of a object/blob/slice to just pass it to a tool
> (file command or dar) that would tells you this is not the good slice is
> un-optimal at all and quite costly ($), no?

It is, but we avoid that by examining date of admission since there's a 
predictable time gap between incoming slices and archives. I use `file` 
after fetching the data only when investigating issues.

>>> What I don't understand is how you could
>>> accidentally miss the first slice, I mean how do you let dar accessing
>>> the slices as dar, even in sequential-read mode open a slice after a
>>> given filename and if it is not present it complains and waits.
>>
>> To deploy or even inspect a backup, its blobs should be pulled from
>> storage somewhere dar can run, then renamed or symlinked to form
>> arc.N.dar. Human factor makes such mistakes possible. If you forgot
>> one or more leading slices yet they're named arc.1.dar, arc.2.dar,
>> etc. - dar won't "complain and wait", it'd proceed straight to
>> throwing errors at you.
>
> same remark as above: the bucket/object/blob metadata is the logical
> place of the slice information you remove from filesystem metadata (=
> filename).
>
> You have to find a way to translate from filesystem metadata to
> S3/bucket/objet/blob metadata and vice versa, this is not the purpose of
> to do that very context specific task.

You seem to be constantly misinterpreting my words in spite of me 
explicitly stating that all my suggestions are concerning recovery mode, 
not normal operation. We have been using our (arguably weird) storage 
system along with dar for over a decade and it works just fine - without 
dar's built-in slice counters, extra UUIDs, etc. But our current problem 
highlighted what I believe is a shortcoming on dar's diagnostic/recovery 
side and I simply propose some improvements (very modest really). I even 
mentioned that it might come in handy for other cases (broken FAT with 
unavailable file names). Again, it seems very in-line with the existing 
tape marks and sequential reading mode.

>>> why only 4 bytes? This would limit the number of slices... even if today
>>> you think it is large enough, this is the same way of thinking that led
>>> to the year 2000 bug, the problem of year 2038, the upper and high
>>> memory in DOS some decades ago, the 2 GB files boundary (and need of
>>> large file support)... and so forth.
>>
>> I understand your point. But even if we take 1 KiB per slice, 2^32
>> slices would add up to 4.4 TiB of data... and more "realistic" 1 MiB
>> per slice would be 4.5 PiB... but if that is a concern, you could use
>> 64 bits, that would be (let me look that up on Wikipedia) around 2
>> exabytes of data if each slice only held 1 byte of payload... I'm
>> afraid we'd need more than quantum computers to cross that boundary...
>
> an so now, the smallest slice which is a few tens of bytes would becomes
> a few kilobytes... what a waste...

What's the practical reason for having over 4 billions of slices, each a 
few tens of bytes? Besides, the counter may be rotating, it's only a 
hint anyway.

>>> A better approach could be to add a random fixed size field at end of
>>> slice and have this same random value at the beginning of the next. But
>>> I don't see the use case as mentioned earlier about number of slices in
>>> filenames.
>>
>> If we're in the territory where 2^32 slices may not be enough, our
>> random numbers may not be random enough too, especially within
>> millions of slices... If we were choosing, I'd just use a fixed length
>> numeric field for the sake of simplicity. VLQ if you must, although
>> I'd grade that as an overkill...
>
> why a "fixed length numeric field" would be enough in the "2^32
> territory" while a "random number" (which size is still undefined) would
> not?

I assumed that both fields would be of the same size because a random 
number field that's wider than a fixed number field looks pointless to 
me - it takes more space, it doesn't guarantee uniqueness even within 
its range, it doesn't allow comparing positions, etc. What's the 
advantage of such a random field?

> what length (in byte, for example) would you give to this "simple
> current slice index"?

Practically, 2 bytes and allow it to overflow. I don't have statistics 
but I don't think 65k+ slices per archive are a norm, and even if this 
practice exists somewhere, a rotating 16-bit counter would make it 2^16 
times less likely that an incorrect ordering of slices would go 
undetected (and 2^16 times easier to determine the number of any given 
slice) compared to when there's no such counter at all. On dar's side, 
the check would simply go from 0 to 2^16, then overflow and repeat.