Re: [BackupPC-users] Missing pool files cause and remediation

[<bac...@ko...>]

Tim Herren wrote at about 19:36:39 +0200 on Monday, April 26, 2021:
 > Hi
 > 
 > A  day after a scheduled reboot of the server during a regular
 > maintenance we start noticing the "missing pool file" messages in the logs.
 > 
 > 2021-04-23 01:00:00 BackupPC_nightly now running BackupPC_refCountUpdate -m -s -c -P 7 -r 128-255
 > 2021-04-23 01:00:00 BackupPC_nightly now running BackupPC_refCountUpdate -m -s -c -P 7 -r 0-127
 > 2021-04-23 01:00:00  admin1 : __bpc_pidStart__ 18306
 > 2021-04-23 01:00:00  admin : __bpc_pidStart__ 18307
 > 2021-04-23 01:00:44  admin1 : BackupPC_refCountUpdate: missing pool file 81e5e7a28a0f857827a9c2de6099875601 count 1
 > 2021-04-23 01:01:18  admin1 : BackupPC_refCountUpdate: missing pool file 83b7b97de222ecb2f2a129233d63bad801 count 1
 > 2021-04-23 01:01:18  admin1 : BackupPC_refCountUpdate: missing pool file 82020853cb8c2718ac4ed58b2541c58c01 count 1
 > 
 > However the shutdown of the server during the maintenance was clean and
 > BackupPC exited with exit-code 0 and no apparent job running.
 > 
 > 2021-04-22 18:17:11 Got signal TERM... cleaning up (exit code = 0)
 > 2021-04-22 18:19:34 Reading hosts file
 > 2021-04-22 18:19:34 BackupPC 4.4.0 (Perl v5.16.3) started, pid 887
 > 2021-04-22 18:19:34 Next wakeup is 2021-04-22 20:00:00
 > 
 > In total I have 56 entries of missing pool files in my logs.
 > 

I had a similar problem with my BackupPC4 setup last Spring with about
40 pool files suddenly going missing.

That led me to finding several other "bugs" in BackuPC4 that Craig
helped nail down and correct. Though I don't believe we ever figured
out what caused the missing files that triggered this whole excursion.

I think at the time I chalked it up to perhaps there being an
improper server shutdown.

That being said it would be great if you could narrow down to when and how
this happened.

> With a quick script we managed to find the files that are missing from
 > the pool. The missing pool files all stem from 2 servers (out of 81) and
 > there from 3 different backups.

Are the files shared across 3 backups or from 3 separate backups on
the same client?
Did they all go missing the same day or did they go missing after each
of the 3 backups?

The more detail you can provide, the easier it will be try to find the mprolem.

 > All missing files are quite similar and are located within
 > 
 > /usr/lib/modules/4.18.0-240.22.1.el8_3.x86_64/kernel/drivers/
 > 
 > so somehow related.
 > 
 > Does anyone have an idea what could have led to those files being
 > missing from the pool?
 > Is there any way to add those files back to the pool to get rid of the
 > messages in the log. The files are still present on the servers.

If you have a copy of the pool, you can just copy the backed up pool
files into the pool. In my case, I had an old btrfs snapshot that I
could use.

If you don't, then the easiest thing to do would be to make a backup
of /usr/lib/modules/4.18.0-240.22.1.el8_3.x86_64/kernel/drivers/
That will force new pool files to be created to replace the missing ones.

 > 
 > Thanks for any insights.
 > 
 > Tim Herren
 > Systems Technician
 > Puzzle ITC GmbH
 > 
 > _______________________________________________
 > BackupPC-users mailing list
 > Bac...@li...
 > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:    https://github.com/backuppc/backuppc/wiki
 > Project: https://backuppc.github.io/backuppc/