|
From: Raoul B. <ra...@bh...> - 2020-09-02 18:19:56
|
So, I am officially lost with additional errors popping up, i.e.
> bpc_path_refCountAll: can't read attrib file
> /data/backup/backuppc/pc/aaa.bbb.ccc/8240/inode/03/attrib52_75b800923eee9d3484169624f5b691fa
Unfortunatly, I have no clue how to clean them up and/or force a true
"Full" backup (means to truly check/recreate/retransfer everything).
From what I understand right now, I have both bad/non-existing attrib
cpool files,
as well as a few errors in the actualy data cpool files.
Any help would be appreciated, before I will have to fall back to a full
re-initialization of my backups (which potentially means transferring
500+ GB of data via painfully slow internet links).
Thanks,
Raoul
On 2020-08-29 07:50, Raoul Bhatia wrote:
> Hi,
>
> another update from my side, for all who might be interested / future
> generations ;-)
>
> I set $Conf{PoolNightlyDigestCheckPercent} = 100
> and ran BackupPC_refCountUpdate -m .
>
> I get 104 ERRORs with md5sum d41d8cd98f00b204e9800998ecf8427e (= empty
> file) instead of the expected digest. All of these files have 0
> length, see https://paste.ubuntu.com/p/RRBSsXvMWV/
>
> My observations until now:
>
> A. The timestamp of these files falls into 1 of 3 dates, see
> https://paste.ubuntu.com/p/FcY3gb5q8V/ (sorted via xargs ls -altr)
> If I disregard the 2 files from 2017, these files first started to
> appear when I upgraded rsync-bpc from 3.0.9.14 to 3.0.9.15
> (I usually initate a test backup right after an upgrade to validate
> everything is functioning correctly)
>
>> /var/log/dpkg.log: 2020-08-12 22:46:22 upgrade rsync-bpc:amd64
>> 3.0.9.14 3.0.9.15
>
>
> B. I verified that there are no other pool files with the same digest
> (so no digest* / digest + extension).
>
>
> C. At least some of these pool files are related to attrib files, i.e.
>> BackupPC_attribPrint
>> aaa.aaa.aaa/8238/f%2fdata%2fmail%2f/fspool/attrib_f1811600a7b6f6e3f2fcafea644e005b
>> BackupPC_attribPrint: cannot read attrib file
>> mail.bhatia.eu/8238/f%2fdata%2fmail%2f/fspool/attrib_f1811600a7b6f6e3f2fcafea644e005b
>> (f1811600a7b6f6e3f2fcafea644e005b)
>
>
> D. Looking at the source of rsync-bpc, I find:
> https://github.com/backuppc/rsync-bpc/blob/3.0.9/backuppc/bpc_poolWrite.c#L348
> and
> https://github.com/backuppc/rsync-bpc/commit/3eb30d1b4fe844d0a1409f8a090c518e4d713f40#diff-11e5a98fabcaf3b4938189c693342209
> . Could this play a role here?
>
>
> (I have to admit that I am not (yet?) able to wrap my head around the
> pooling, so please excuse any error in my assuptions.)
>
> -->
> 1. To me, this reads in a way that backuppc might (have?) decide(d) to
> zero out a file / create an empty pool file?
>
> 2. Would it make sense to handle this specific case (empty pool file)
> in BackupPC_refCountUpdate when checking the pool, i.e. at
> https://github.com/backuppc/backuppc/blob/master/bin/BackupPC_refCountUpdate#L821
>
>
> Any further help to debug this problem would be appreciated!
> Raoul
> PS. Cross-posting to backuppc-devel mailing list, just in case.
>
> On 2020-08-27 21:58, Raoul Bhatia wrote:
>> Hi Craig, Guillermo, all.
>>
>> A quick update from my side
>>
>> I have a (complete?) list of (potentially) broken files as reported by
>> BTRFS read errors.
>>
>> I then validated the md5sum from the (c)pool with a one-liner:
>>> for i in $(grep /cpool/ ~/broken_files.txt); do
>>> M=$(/usr/share/backuppc/bin/BackupPC_zcat $i | md5sum | cut -d ' ' -f
>>> 1); echo -n "$M: "; echo $i | grep --color $M || echo "$f ERR"; done
>> --> No error, perhaps I am lucky? :-)
>>
>>
>> This leaves only one file that might be damaged:
>> backuppc/pc/abc.def.ghi/1989/refCnt/poolCnt.1.16 ?
>>
>> I am now experimenting with running
>> /usr/share/backuppc/bin/BackupPC_refCountUpdate -m
>> (Until now: No error; exit code 0)
>>
>>
>>
>> Reading the source I have the following questions:
>>
>> 1.
>> https://github.com/backuppc/backuppc/blob/master/bin/BackupPC_refCountUpdate#L819
>> does "rand(100) < $Conf{PoolNightlyDigestCheckPercent}"
>>
>> My understanding would be that I won't get a predictable 1% check on
>> *each* run. Perhaps on average, with a sufficiently large pool, over
>> a longer period of time, this might work out, but for a few runs of
>> BackupPC_refCountUpdate not.
>>
>> --> Perhaps there would be another implementation that gets a more
>> predictable result?
>> (i.e. create an array of all the files to check, sort them, and then
>> take the first $Conf{PoolNightlyDigestCheckPercent} of entries, but at
>> least 1?)
>>
>>
>>
>> 2. I also seem to have accumulated checksum errors over the past
>> years.
>>
>> --> How do I proceed with these files? If they still exist on the
>> source, I'd like to re-sync them to the backup.
>>
>>
>> 3. For backuppc/pc/abc.def.ghi/1989/refCnt/poolCnt.1.16 , I do not
>> find a way to re-check only this one, old backup.
>>
>> --> Would it be a good idea to add an option to add a "-n num" flag to
>> BackupPC_refCountUpdate to be able to operate on a particular backup?
>>
>>
>>
>> Thanks for your guidance,
>> Raoul
>>
>> On 2020-08-24 20:39, Raoul Bhatia wrote:
>>> Hi Craig and Guillermo,
>>>
>>> On 2020-08-23 19:35, Craig Barratt via BackupPC-users wrote:
>>>
>>>> $Conf{PoolNightlyDigestCheckPercent} is in percent, so you should
>>>> set it to 100 to check all the pool file's MD5 digest against their
>>>> file names.
>>>>
>>>> As Guillermo mentions, to check the pool MD5 digests, you can set
>>>> temporarily set $Conf{PoolNightlyDigestCheckPercent} to 100 and
>>>> $Conf{PoolSizeNightlyUpdatePeriod} to 1.
>>>
>>> When reading the documentation, I also came across these options.
>>> However, I didn't dare to run backuppc / BackupPC_nightly, because
>>> from the documentation:
>>>
>>> Overnight, when BackupPC_nightly next runs,
>>> all the unused pool files will be deleted and
>>> this will recover the disk space used by the client's backups.
>>>
>>> I didn't want to end up with an empty pool...
>>>
>>>> If you stop BackupPC, to check all the pool digests, run:
>>>>
>>>>> BackupPC_refCountUpdate -m
>>>> If you want to also regenerate all the host reference counts (which
>>>> will take a long time), you could run:
>>>>
>>>>> BackupPC_refCountUpdate -m -F
>>>
>>> Meanwhile, with the kind help of the btrfs community, I figured out a
>>> way to get the damaged files. This process is not finished, yet,
>>> however, I have a first list:
>>>
>>> /mnt/backuppc/pc/abc.def.ghi/1989/refCnt/poolCnt.1.16
>>> /mnt/backuppc/cpool/5c/c8/5cc9373a32e06baaa308a7b341db5ac9
>>> /mnt/backuppc/cpool/b2/62/b3629c46481cb038682aea248c45b89f
>>> /mnt/backuppc/cpool/20/c6/21c61013d40e644af734e28459df0a1a
>>> /mnt/backuppc/cpool/8a/f8/8af935bc53f7199ed75a5695bfd57f26
>>>
>>> FYI: The most recent backup from host abc.def.ghi is 2291, so 1989 is
>>> quite far in the past.
>>>
>>> How should I proceed when I have a list of broken files?
>>> Move them out from cpool and hope they will be re-synced by the next
>>> (full?) backup?
>>>
>>> Thanks,
>>> Raoul
>>>
>>>> Craig
>>>>
>>>> On Sun, Aug 23, 2020 at 6:45 AM Guillermo Rozas
>>>> <gui...@gm...> wrote:
>>>>
>>>> Hi Raoul,
>>>>
>>>> are you using BackupPC v4? If yes, you can use a modification of the
>>>> script I posted here:
>>>> https://sourceforge.net/p/backuppc/mailman/message/37032497/
>>>>
>>>> In the latest version (4.4.0) you also have the config option
>>>> $Conf{PoolNightlyDigestCheckPercent}, which checks the md5 digest of
>>>> this fraction of the pool files each night. You can probably set it
>>>> to 1 and wait a night for it to run.
>>>>
>>>> Regards,
>>>> Guillermo
>>>>
>>>> On Sun, Aug 23, 2020 at 5:38 AM Raoul Bhatia <ra...@bh...>
>>>> wrote: Hi,
>>>>
>>>> related to my previous email, it seems that the cause of my issues
>>>> was a
>>>> file system corruption after a "power cut".
>>>>
>>>> I managed to recover (most of?) the data and would now like to do a
>>>> thorough check of the data.
>>>>
>>>> Is there any way to "fully verify" the integrity of my backuppc
>>>> installation, ideally in a nondestructive way ;-)
>>>>
>>>> Thanks,
>>>> Raoul
>>>>
>>>> PS. My backuppc process is stopped.
>>>> --
>>>> DI (FH) Raoul Bhatia MSc
>>>> E-Mail. ra...@bh...
>>>> Tel. +43 699 10132530
>>
>>
>>
>> _______________________________________________
>> BackupPC-users mailing list
>> Bac...@li...
>> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki: https://github.com/backuppc/backuppc/wiki
>> Project: https://backuppc.github.io/backuppc/
--
DI (FH) Raoul Bhatia MSc
E-Mail. ra...@bh...
Tel. +43 699 10132530
|