Re: [BackupPC-users] poolfiles hardlinked between V3 and V4 pool

[<Mat...@gm...>]

Am Sonntag, dem 17.08.2025 um 16:58 +0100 schrieb G.W. Haywood:
> Hi there,
> 
> On Sun, 17 Aug 2025, Mat...@gm... wrote:
> 
> > Yes, I'm very sure I have no V3 backups anymore.
> > $Conf{PoolV3Enabled} is still enabled. I'll disable it soon.
> 
> When you do that, the only change to operation should be that BackupPC
> V4 will no longer look for anything in the V3 pool.
> 
> > $Conf{BackupPCNightlyPeriod} = 16, since 2 days. Before it was 1.
> 
> Am I right in thinking that BackupPC has been running with the value
> set to 1 for some days after you converted all your V3 backups to V4?
Yes, some weeks
> 
> > Yesterday I've already deleted in ../*pool/?/:
> >    1. all files with only 1 hardlink
> >    2. all files which are only hardlinked with other files in pool or cpool
> > I didn't remove any file in ../*pool/??/ and I still have a list of all (2) deleted files and
> > where
> > they was linked too. So I'm able to restore any of them.
> > Now I don't have a single file left in ../*pool/?/??
> > 
> > Tonight BackupPC_nightly run and have no errors.
> 
> :)
> 
> > Just 1.800 "admin : BackupPC_refCountUpdate: missing pool file". But this only affects files
> > that
> > are a few years old. I'm still investigating.
> 
> Have you tried BackupPC_fsck?
Yes and I'll run it once again, after analysis before repairing "manually".
> 
> This might (possibly) help:
> 
> https://sourceforge.net/p/backuppc/mailman/message/58712645/
I already found this script. But I have more than 11,000 missing poolfiles. because that I decide to
build another script, using:
   BackupPC_ls -R... | grep -B1 "bpc_attrib_dirRead: can't open bpc_attrib_dirRead: can't open "
for each host, to scan all backups and find the backups containing this missing pool files. Than I
can try to restore them from an backup of my old V3 data and migrate it again to V4 or delete this
backups.

BTW1:
 * is there a place where I could provide this (bash) script?
 * sometimes I'm developing patches to BackupPC perl scripts (applyable with patch -p0 --dry-run --
   reject-file=oops.rej <file.patch) and better documented than below. Is there a place where I
   could provide them?
    - BackupPC_backupDuplicate can need some time and the client could go to sleep in the meantime. A
      patch to  BackupPC_dump move the call of DumpPreUserCmd before execution of
      BackupPC_backupDuplicate and a users DumpPreUserCmd can disable hibernation on client side.
    - BackupPC_restore could need some time between calculation of $lastNum and using it for
      RestoreInfo and RestoreLOG. A patch move the calculation behind the call of the
      RestorePostUserCmd and if someone, like me😁️, is calling BackupPC_restore from a programm
      several times in parallel for different shares and dirs of a host, each single call can use
      another $lastNum.

BTW2:
 * "BackupPC_ls -R -h stella-pc -n 69 -s BACKUP4U /" is listing the files in this backup as well as
   some missing pool files:
   :
   -rw-r--r-- 1014/513 13133281 2014-06-15 09:01:46 /.bmr/volWBP.img.gz (9f927123ffec03873252eea3bff234e2)
   /bin:
   bpc_attrib_dirRead: can't open /var/lib/backuppc/pool/76/e6/76e64ae4bb7e74082523819cd8c01fb5
   /etc:
   -rw-rw-r-- 1014/544 33338 2008-11-08 00:31:46 /etc/Backup4U-License.txt (c52061746e223ec3a1a5ef4b495c071d)
   :
drwxrwx--- 1014/544 0 2011-10-09 20:45:32 /home/Backup4U/OBS/
/home/Backup4U/.ssh:
-rwxrwx--- 1014/544 415 2008-12-14 14:12:10 /home/Backup4U/.ssh/authorized_keys
(e86ae4879765b4579bb3deee0626e88b)
bpc_attrib_dirRead: can't open /var/lib/backuppc/pool/74/fa/74faf6dde97ccc2439ee042541197853
/home/Backup4U/OBS:
-rwxrwx--- 1014/544 5 2008-12-22 23:29:12 /home/Backup4U/OBS/.bash_history
(f4b524261fce06c1fbd10b4681ad0b97)
:

I understand the first bpc_attrib_dirRead, which means "/bin" is empty.
But I don't understand the second one because "/home/Backup4U/.ssh/authorized_keys" exists.

> 
> See also 'Extensions and Scripts' in the Wiki:
> 
> https://github.com/backuppc/backuppc/wiki/
Thank you, good to know 😊️
> 
> > One question: Is it really true that backuppc V4 still need the old ../*pool/?/ directories or
> > only
> > until $Conf{PoolV3Enabled} is disabled?
> 
> If $Conf{PoolV3Enabled} is set to a true value, BackupPC version 4
> uses the files in the V3 pool if it needs to.  If you *successfully*
> converted all the V3 backups to V4 backups, then after the requisite
> number of executions of BackupPC_nightly (this number depends on the
> value of $Conf{BackupPCNightlyPeriod}) then theoretically it should
> have deleted the V3 pool files by itself and it will no longer need
> anything from the V3 pool.  The only files that should remain in the
> V3 pool are *singly-linked* attrib files, which you can delete if you
> wish because they are not used if they only have a single link but the
> documentation is clear that you need to check carefully.  Also in this
> situation, you can set $Conf{PoolV3Enabled} to false (whether you have
> deleted the V3 attrib files or not).
Great - so I'll still wait a week or two for disabling  $Conf{PoolV3Enabled}
> 
> The script which converts V3 backups to the V4 equivalent has not had
> anything like as much exercise as it could have had.  In circumstances
> where there are 'issues' it could conceivably fail to do what it is
> intended to do (and perhaps do things which it is not intended to do).
> In such a case you probably do not want to set $Conf{PoolV3Enabled} to
> a false value until you have made sure that no V3 pool file is needed.
> At this late stage in the life of BackupPC V3 I think in the future it
> is likely that very few people will want to migrate V3 systems to V4.
> So it's unlikely that the conversion script will ever benefit from the
> sort of extensive testing which it would need to make it bullet-proof.
I think the main reason of this corruptions in my system was an insufficient maximum number of open
file descriptors.
As soon as I recognized this and set "ulimit -n 10000" all remaining migrations went well.
Probably the aborts before, because of insufficient maximum number of open file descriptors, create
some wrong links.
However - I cleaned it up😊️
> 
> It seems likely that your conversion of V3 backups to V4 backups did
> not go very well.  You said that there were 'issues', and that now the
> count of missing pool files is nearly two thousand.  (1.800 - is that
> right? in the UK we use comma , not decimal point . as the thousands
> separator).  You are right to want to investigate.  Are you able to
> recover a few files successfully?  Perhaps choose some at random, and
> some because they're big/small/valuable/new/old?
1,800 missing pool files found last night, in 1/16 of the pool. In total it is more than 11,000 out
of 30,000,000. So only a few files are affected😁️.
Probably because of filesystem corrections made by e2fsck in the past or because of some aborted
migrations, mentioned above.
> 
> > Have a great day
> 
> U2. :)
> 
best regards 😊️