Re: [BackupPC-devel] [BackupPC-users] Verify full integrity of pool and backups

[Raoul B. <ra...@bh...>]

So, I am officially lost with additional errors popping up, i.e.

> bpc_path_refCountAll: can't read attrib file 
> /data/backup/backuppc/pc/aaa.bbb.ccc/8240/inode/03/attrib52_75b800923eee9d3484169624f5b691fa

Unfortunatly, I have no clue how to clean them up and/or force a true 
"Full" backup (means to truly check/recreate/retransfer everything).

 From what I understand right now, I have both bad/non-existing attrib 
cpool files,
as well as a few errors in the actualy data cpool files.


Any help would be appreciated, before I will have to fall back to a full 
re-initialization of my backups (which potentially means transferring 
500+ GB of data via painfully slow internet links).

Thanks,
Raoul

On 2020-08-29 07:50, Raoul Bhatia wrote:
> Hi,
> 
> another update from my side, for all who might be interested / future
> generations ;-)
> 
> I set $Conf{PoolNightlyDigestCheckPercent} = 100
> and ran BackupPC_refCountUpdate -m .
> 
> I get 104 ERRORs with md5sum d41d8cd98f00b204e9800998ecf8427e (= empty
> file) instead of the expected digest.  All of these files have 0
> length, see https://paste.ubuntu.com/p/RRBSsXvMWV/
> 
> My observations until now:
> 
> A. The timestamp of these files falls into 1 of 3 dates, see
> https://paste.ubuntu.com/p/FcY3gb5q8V/ (sorted via xargs ls -altr)
> If I disregard the 2 files from 2017, these files first started to
> appear when I upgraded rsync-bpc from 3.0.9.14 to 3.0.9.15
> (I usually initate a test backup right after an upgrade to validate
> everything is functioning correctly)
> 
>> /var/log/dpkg.log: 2020-08-12 22:46:22 upgrade rsync-bpc:amd64 
>> 3.0.9.14 3.0.9.15
> 
> 
> B. I verified that there are no other pool files with the same digest
> (so no digest* / digest + extension).
> 
> 
> C. At least some of these pool files are related to attrib files, i.e.
>> BackupPC_attribPrint 
>> aaa.aaa.aaa/8238/f%2fdata%2fmail%2f/fspool/attrib_f1811600a7b6f6e3f2fcafea644e005b
>> BackupPC_attribPrint: cannot read attrib file 
>> mail.bhatia.eu/8238/f%2fdata%2fmail%2f/fspool/attrib_f1811600a7b6f6e3f2fcafea644e005b 
>> (f1811600a7b6f6e3f2fcafea644e005b)
> 
> 
> D. Looking at the source of rsync-bpc, I find:
> https://github.com/backuppc/rsync-bpc/blob/3.0.9/backuppc/bpc_poolWrite.c#L348
> and
> https://github.com/backuppc/rsync-bpc/commit/3eb30d1b4fe844d0a1409f8a090c518e4d713f40#diff-11e5a98fabcaf3b4938189c693342209
> .  Could this play a role here?
> 
> 
> (I have to admit that I am not (yet?) able to wrap my head around the
> pooling, so please excuse any error in my assuptions.)
> 
> -->
> 1. To me, this reads in a way that backuppc might (have?) decide(d) to
> zero out a file / create an empty pool file?
> 
> 2. Would it make sense to handle this specific case (empty pool file)
> in BackupPC_refCountUpdate when checking the pool, i.e. at
> https://github.com/backuppc/backuppc/blob/master/bin/BackupPC_refCountUpdate#L821
> 
> 
> Any further help to debug this problem would be appreciated!
> Raoul
> PS. Cross-posting to backuppc-devel mailing list, just in case.
> 
> On 2020-08-27 21:58, Raoul Bhatia wrote:
>> Hi Craig, Guillermo, all.
>> 
>> A quick update from my side
>> 
>> I have a (complete?) list of (potentially) broken files as reported by
>> BTRFS read errors.
>> 
>> I then validated the md5sum from the (c)pool with a one-liner:
>>> for i in $(grep /cpool/ ~/broken_files.txt); do 
>>> M=$(/usr/share/backuppc/bin/BackupPC_zcat $i | md5sum | cut -d ' ' -f 
>>> 1); echo -n "$M: "; echo $i | grep --color $M || echo "$f ERR"; done
>> --> No error, perhaps I am lucky? :-)
>> 
>> 
>> This leaves only one file that might be damaged:
>> backuppc/pc/abc.def.ghi/1989/refCnt/poolCnt.1.16 ?
>> 
>> I am now experimenting with running
>> /usr/share/backuppc/bin/BackupPC_refCountUpdate -m
>> (Until now: No error; exit code 0)
>> 
>> 
>> 
>> Reading the source I have the following questions:
>> 
>> 1.
>> https://github.com/backuppc/backuppc/blob/master/bin/BackupPC_refCountUpdate#L819
>> does "rand(100) < $Conf{PoolNightlyDigestCheckPercent}"
>> 
>> My understanding would be that I won't get a predictable 1% check on
>> *each* run.  Perhaps on average, with a sufficiently large pool, over
>> a longer period of time, this might work out, but for a few runs of
>> BackupPC_refCountUpdate not.
>> 
>> --> Perhaps there would be another implementation that gets a more
>> predictable result?
>> (i.e. create an array of all the files to check, sort them, and then
>> take the first $Conf{PoolNightlyDigestCheckPercent} of entries, but at
>> least 1?)
>> 
>> 
>> 
>> 2. I also seem to have accumulated checksum errors over the past 
>> years.
>> 
>> --> How do I proceed with these files?  If they still exist on the
>> source, I'd like to re-sync them to the backup.
>> 
>> 
>> 3. For backuppc/pc/abc.def.ghi/1989/refCnt/poolCnt.1.16 , I do not
>> find a way to re-check only this one, old backup.
>> 
>> --> Would it be a good idea to add an option to add a "-n num" flag to
>> BackupPC_refCountUpdate to be able to operate on a particular backup?
>> 
>> 
>> 
>> Thanks for your guidance,
>> Raoul
>> 
>> On 2020-08-24 20:39, Raoul Bhatia wrote:
>>> Hi Craig and Guillermo,
>>> 
>>> On 2020-08-23 19:35, Craig Barratt via BackupPC-users wrote:
>>> 
>>>> $Conf{PoolNightlyDigestCheckPercent} is in percent, so you should 
>>>> set it to 100 to check all the pool file's MD5 digest against their 
>>>> file names.
>>>> 
>>>> As Guillermo mentions, to check the pool MD5 digests, you can set 
>>>> temporarily set $Conf{PoolNightlyDigestCheckPercent} to 100 and 
>>>> $Conf{PoolSizeNightlyUpdatePeriod} to 1.
>>> 
>>> When reading the documentation, I also came across these options.
>>> However, I didn't dare to run backuppc / BackupPC_nightly, because
>>> from the documentation:
>>> 
>>>    Overnight, when BackupPC_nightly next runs,
>>>    all the unused pool files will be deleted and
>>>    this will recover the disk space used by the client's backups.
>>> 
>>> I didn't want to end up with an empty pool...
>>> 
>>>> If you stop BackupPC, to check all the pool digests, run:
>>>> 
>>>>> BackupPC_refCountUpdate -m
>>>> If you want to also regenerate all the host reference counts (which 
>>>> will take a long time), you could run:
>>>> 
>>>>> BackupPC_refCountUpdate -m -F
>>> 
>>> Meanwhile, with the kind help of the btrfs community, I figured out a
>>> way to get the damaged files.  This process is not finished, yet,
>>> however, I have a first list:
>>> 
>>> /mnt/backuppc/pc/abc.def.ghi/1989/refCnt/poolCnt.1.16
>>> /mnt/backuppc/cpool/5c/c8/5cc9373a32e06baaa308a7b341db5ac9
>>> /mnt/backuppc/cpool/b2/62/b3629c46481cb038682aea248c45b89f
>>> /mnt/backuppc/cpool/20/c6/21c61013d40e644af734e28459df0a1a
>>> /mnt/backuppc/cpool/8a/f8/8af935bc53f7199ed75a5695bfd57f26
>>> 
>>> FYI: The most recent backup from host abc.def.ghi is 2291, so 1989 is
>>> quite far in the past.
>>> 
>>> How should I proceed when I have a list of broken files?
>>> Move them out from cpool and hope they will be re-synced by the next
>>> (full?) backup?
>>> 
>>> Thanks,
>>> Raoul
>>> 
>>>> Craig
>>>> 
>>>> On Sun, Aug 23, 2020 at 6:45 AM Guillermo Rozas 
>>>> <gui...@gm...> wrote:
>>>> 
>>>> Hi Raoul,
>>>> 
>>>> are you using BackupPC v4? If yes, you can use a modification of the 
>>>> script I posted here: 
>>>> https://sourceforge.net/p/backuppc/mailman/message/37032497/
>>>> 
>>>> In the latest version (4.4.0) you also have the config option 
>>>> $Conf{PoolNightlyDigestCheckPercent}, which checks the md5 digest of 
>>>> this fraction of the pool files each night. You can probably set it 
>>>> to 1 and wait a night for it to run.
>>>> 
>>>> Regards,
>>>> Guillermo
>>>> 
>>>> On Sun, Aug 23, 2020 at 5:38 AM Raoul Bhatia <ra...@bh...> 
>>>> wrote: Hi,
>>>> 
>>>> related to my previous email, it seems that the cause of my issues 
>>>> was a
>>>> file system corruption after a "power cut".
>>>> 
>>>> I managed to recover (most of?) the data and would now like to do a
>>>> thorough check of the data.
>>>> 
>>>> Is there any way to "fully verify" the integrity of my backuppc
>>>> installation, ideally in a nondestructive way ;-)
>>>> 
>>>> Thanks,
>>>> Raoul
>>>> 
>>>> PS. My backuppc process is stopped.
>>>> --
>>>> DI (FH) Raoul Bhatia MSc
>>>> E-Mail. ra...@bh...
>>>> Tel. +43 699 10132530
>> 
>> 
>> 
>> _______________________________________________
>> BackupPC-users mailing list
>> Bac...@li...
>> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki:    https://github.com/backuppc/backuppc/wiki
>> Project: https://backuppc.github.io/backuppc/

-- 
DI (FH) Raoul Bhatia MSc
E-Mail. ra...@bh...
Tel. +43 699 10132530