backuppc-users — General list for user discussion, questions and support

Re: [BackupPC-users] Cache Poisoning- solution?

[Christian V. <cvo...@kn...>]

Hi& thanjks for the addition.

So I assume your script works (more or less) similar to the nocache utility.

I figured out there are some edge cases where the nocache does not work 
for 100% and the kernel leaves some pages as active.
To prevent this it is recommended to use the "--no-mmap" parameter for 
rsync. It might be avery little bit slower but interacts in a better way 
with nocache.

Configured my rsync parameters with the --no-mmap now for all hosts 
where I use the nocache utility.

Any furhter hints?

/KNEBB


Am 11.08.25 um 10:58 schrieb Andreas Schnederle-Wagner - Futureweb GmbH:
> Hi all,
>
> interesting discussion – we’ve observed similar effects with the page cache in our environment as well, especially when backing up dozens of terabytes of data across many servers.
> It’s not that these effects caused any serious issues on the systems, but in the statistics it was clearly visible when backups were running and the existing cache got replaced.
>
> For that reason, we’ve been successfully using the nocache.sh script (https://raw.githubusercontent.com/virtuozzo/docs_downloads/main/nocache.sh) from Virtuozzo for several years now on our servers running Virtuozzo 7.5.
> With it, the existing cache is no longer evicted during backups, and we have never experienced any negative impact on the backups themselves – which is exactly what you would expect from a technical point of view.
>
> Why it works (Virtuozzo specific):
> nocache.sh runs the backup process inside its own memory cgroup and applies a tight page-cache limit (default 256 MB) to that cgroup. This means the backup can only use a small cache working set, preventing it from pushing out valuable cached data from the rest of the system.
> It does not globally disable caching – metadata (directory entries, inode info) is not part of the page cache and remains available for fast lookups, which is particularly helpful for subsequent incremental backups.
> The script uses cgroup v1 memory controller features and Virtuozzo-specific knobs like memory.cache.limit_in_bytes and memory.disable_cleancache, so it works out of the box on Virtuozzo 7.5.
> On systems using pure cgroup v2 (like stock Debian 12), these interfaces aren’t available by default.
>
> Hopefully these insights are useful to others.
>
> Best regards from Tyrol
> Andreas
>
>
> -----Ursprüngliche Nachricht-----
> Von: Christian Völker via BackupPC-users <bac...@li...>
> Gesendet: Samstag, 9. August 2025 21:18
> An: bac...@li...
> Cc: Christian Völker <cvo...@kn...>
> Betreff: Re: [BackupPC-users] Cache Poisoning- solution?
>
> Hi,
>
> this was the type of discussion I wanted to have. Thanks!
>
> I did not think about the metadata and the other shares, indeed. So according to your comments I took a deeper dive to nocache:
>
> It marks the pages used by the kernel for reading the data as „do not need“. So after reading a chunk the kernel frees the memory immediately.
> So for the next chunk it will be re-used. So the existing page cache does not get polluted by backup.
>
> But metadata and directory entries will still be cached- as they are not in page cache anyways they are not affected. So I guess for subsequent incremental backups this will be perfect.
>
> Looks like my initial thoughts were fine and because of metadata still caching it is really a very good option to improve setup.
> However, for running databases this is indeed not suitable- just as no backups are…
>
> Grüsse
> Christian Völker
>
>> Am 09.08.2025 um 17:00 schrieb G.W. Haywood <ba...@ju...>:
>>
>> Hi there,
>>
>>> On Sat, 9 Aug 2025, Christian V?lker wrote:
>>>
>>> I noticed ... high CPU usage 20% ... throwing away "old" filesystem
>>> cache blocks to cache ... directory ... 400GB ... blocks are read only once ...
>>> thrown away a couple of hours later ...
>>> ...
>>> ... "RsyncClientPath" ... to "/usr/bin/nocache /usr/bin/rsync".
>>>
>>> Now the blocks read during backup are not going into the cache ...
>>>
>>> So what do you think? Is this a good solution? Or are there any pitfalls?
>> Does using 20% CPU for a couple of hours actually matter?
>>
>> Presumably by doing this you'll be defeating _all_ caching of anything
>> you're backing up, not just caching of the 400GB directory.  That will
>> include directory contents as well as file contents, and may possibly
>> be particularly relevant to incremental backups.  You're obviously
>> well in tune with your systems so I guess you'll probably notice any
>> untoward effects.  I'd expect significantly increased disc activity
>> because of directory reads which are no longer cached.  That might be
>> more relevant for the life expectancy of spinning rust than for SSDs.
>> It might mean that the backups take significantly longer.  I'm sure
>> we'll all be interested in your reports, if you find such effects.
>>
>> In a case like this as the data you're backing up is relatively static
>> I think I'd be tempted to put the 400GB directory in a share all of
>> its own, and back it up on a different schedule.  But the 'nocache'
>> utility was created expressly for your use case so I don't think it's
>> likely that there will be any serious downsides unless there's some
>> terrible flaw in the implementation which I guess can't be ruled out.
>> I'd be reluctant to do this to my backups for that reason.
>>
>> FWIW where I have for example large database directories - of course
>> not necessarily static data - I don't let BackupPC anywhere near them.
>> Instead I use the DB facilities to create data dumps at intervals (via
>> cron), and then back up the data dumps using rsync (again via cron).
>>
>> --
>>
>> 73,
>> Ged.
>>
>>
>> _______________________________________________
>> BackupPC-users mailing list
>> Bac...@li...
>> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki:    https://github.com/backuppc/backuppc/wiki
>> Project: https://backuppc.github.io/backuppc/
>
>
> _______________________________________________
> BackupPC-users mailing list
> Bac...@li...
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:    https://github.com/backuppc/backuppc/wiki
> Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] Cache Poisoning- solution?

[Andreas Schnederle-W. - F. G. <sch...@fu...>]

Hi all,

interesting discussion – we’ve observed similar effects with the page cache in our environment as well, especially when backing up dozens of terabytes of data across many servers.
It’s not that these effects caused any serious issues on the systems, but in the statistics it was clearly visible when backups were running and the existing cache got replaced.

For that reason, we’ve been successfully using the nocache.sh script (https://raw.githubusercontent.com/virtuozzo/docs_downloads/main/nocache.sh) from Virtuozzo for several years now on our servers running Virtuozzo 7.5.
With it, the existing cache is no longer evicted during backups, and we have never experienced any negative impact on the backups themselves – which is exactly what you would expect from a technical point of view.

Why it works (Virtuozzo specific):
nocache.sh runs the backup process inside its own memory cgroup and applies a tight page-cache limit (default 256 MB) to that cgroup. This means the backup can only use a small cache working set, preventing it from pushing out valuable cached data from the rest of the system.
It does not globally disable caching – metadata (directory entries, inode info) is not part of the page cache and remains available for fast lookups, which is particularly helpful for subsequent incremental backups.
The script uses cgroup v1 memory controller features and Virtuozzo-specific knobs like memory.cache.limit_in_bytes and memory.disable_cleancache, so it works out of the box on Virtuozzo 7.5. 
On systems using pure cgroup v2 (like stock Debian 12), these interfaces aren’t available by default.

Hopefully these insights are useful to others.

Best regards from Tyrol
Andreas


-----Ursprüngliche Nachricht-----
Von: Christian Völker via BackupPC-users <bac...@li...> 
Gesendet: Samstag, 9. August 2025 21:18
An: bac...@li...
Cc: Christian Völker <cvo...@kn...>
Betreff: Re: [BackupPC-users] Cache Poisoning- solution?

Hi,

this was the type of discussion I wanted to have. Thanks!

I did not think about the metadata and the other shares, indeed. So according to your comments I took a deeper dive to nocache:

It marks the pages used by the kernel for reading the data as „do not need“. So after reading a chunk the kernel frees the memory immediately. 
So for the next chunk it will be re-used. So the existing page cache does not get polluted by backup. 

But metadata and directory entries will still be cached- as they are not in page cache anyways they are not affected. So I guess for subsequent incremental backups this will be perfect. 

Looks like my initial thoughts were fine and because of metadata still caching it is really a very good option to improve setup. 
However, for running databases this is indeed not suitable- just as no backups are…

Grüsse
Christian Völker 

> Am 09.08.2025 um 17:00 schrieb G.W. Haywood <ba...@ju...>:
> 
> Hi there,
> 
>> On Sat, 9 Aug 2025, Christian V?lker wrote:
>> 
>> I noticed ... high CPU usage 20% ... throwing away "old" filesystem 
>> cache blocks to cache ... directory ... 400GB ... blocks are read only once ...
>> thrown away a couple of hours later ...
>> ...
>> ... "RsyncClientPath" ... to "/usr/bin/nocache /usr/bin/rsync".
>> 
>> Now the blocks read during backup are not going into the cache ...
>> 
>> So what do you think? Is this a good solution? Or are there any pitfalls?
> 
> Does using 20% CPU for a couple of hours actually matter?
> 
> Presumably by doing this you'll be defeating _all_ caching of anything 
> you're backing up, not just caching of the 400GB directory.  That will 
> include directory contents as well as file contents, and may possibly 
> be particularly relevant to incremental backups.  You're obviously 
> well in tune with your systems so I guess you'll probably notice any 
> untoward effects.  I'd expect significantly increased disc activity 
> because of directory reads which are no longer cached.  That might be 
> more relevant for the life expectancy of spinning rust than for SSDs.
> It might mean that the backups take significantly longer.  I'm sure 
> we'll all be interested in your reports, if you find such effects.
> 
> In a case like this as the data you're backing up is relatively static 
> I think I'd be tempted to put the 400GB directory in a share all of 
> its own, and back it up on a different schedule.  But the 'nocache'
> utility was created expressly for your use case so I don't think it's 
> likely that there will be any serious downsides unless there's some 
> terrible flaw in the implementation which I guess can't be ruled out.
> I'd be reluctant to do this to my backups for that reason.
> 
> FWIW where I have for example large database directories - of course 
> not necessarily static data - I don't let BackupPC anywhere near them.
> Instead I use the DB facilities to create data dumps at intervals (via 
> cron), and then back up the data dumps using rsync (again via cron).
> 
> --
> 
> 73,
> Ged.
> 
> 
> _______________________________________________
> BackupPC-users mailing list
> Bac...@li...
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:    https://github.com/backuppc/backuppc/wiki
> Project: https://backuppc.github.io/backuppc/



_______________________________________________
BackupPC-users mailing list
Bac...@li...
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

[BackupPC-users] Could I remove ../pool/0-f after migration to V4

[<Mat...@gm...>]

Hi,

Can I remove ../pool/0 till ../pool/f directories after migration to V4?

As far as I understand we use ../pool/00 till ../pool/fe instead the old single number ones
and the pool directories with single character names are obsolete now.

Thanks in advance
Matthias

Re: [BackupPC-users] Cache Poisoning- solution?

[Christian V. <cvo...@kn...>]

Hi,

this was the type of discussion I wanted to have. Thanks!

I did not think about the metadata and the other shares, indeed. So according to your comments I took a deeper dive to nocache:

It marks the pages used by the kernel for reading the data as „do not need“. So after reading a chunk the kernel frees the memory immediately. 
So for the next chunk it will be re-used. So the existing page cache does not get polluted by backup. 

But metadata and directory entries will still be cached- as they are not in page cache anyways they are not affected. So I guess for subsequent incremental backups this will be perfect. 

Looks like my initial thoughts were fine and because of metadata still caching it is really a very good option to improve setup. 
However, for running databases this is indeed not suitable- just as no backups are…

Grüsse 
Christian Völker 

> Am 09.08.2025 um 17:00 schrieb G.W. Haywood <ba...@ju...>:
> 
> Hi there,
> 
>> On Sat, 9 Aug 2025, Christian V?lker wrote:
>> 
>> I noticed ... high CPU usage 20% ... throwing away "old" filesystem cache
>> blocks to cache ... directory ... 400GB ... blocks are read only once ...
>> thrown away a couple of hours later ...
>> ...
>> ... "RsyncClientPath" ... to "/usr/bin/nocache /usr/bin/rsync".
>> 
>> Now the blocks read during backup are not going into the cache ...
>> 
>> So what do you think? Is this a good solution? Or are there any pitfalls?
> 
> Does using 20% CPU for a couple of hours actually matter?
> 
> Presumably by doing this you'll be defeating _all_ caching of anything
> you're backing up, not just caching of the 400GB directory.  That will
> include directory contents as well as file contents, and may possibly
> be particularly relevant to incremental backups.  You're obviously
> well in tune with your systems so I guess you'll probably notice any
> untoward effects.  I'd expect significantly increased disc activity
> because of directory reads which are no longer cached.  That might be
> more relevant for the life expectancy of spinning rust than for SSDs.
> It might mean that the backups take significantly longer.  I'm sure
> we'll all be interested in your reports, if you find such effects.
> 
> In a case like this as the data you're backing up is relatively static
> I think I'd be tempted to put the 400GB directory in a share all of
> its own, and back it up on a different schedule.  But the 'nocache'
> utility was created expressly for your use case so I don't think it's
> likely that there will be any serious downsides unless there's some
> terrible flaw in the implementation which I guess can't be ruled out.
> I'd be reluctant to do this to my backups for that reason.
> 
> FWIW where I have for example large database directories - of course
> not necessarily static data - I don't let BackupPC anywhere near them.
> Instead I use the DB facilities to create data dumps at intervals (via
> cron), and then back up the data dumps using rsync (again via cron).
> 
> --
> 
> 73,
> Ged.
> 
> 
> _______________________________________________
> BackupPC-users mailing list
> Bac...@li...
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:    https://github.com/backuppc/backuppc/wiki
> Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] Cache Poisoning- solution?

[G.W. H. <ba...@ju...>]

Hi there,

On Sat, 9 Aug 2025, Christian V?lker wrote:

> I noticed ... high CPU usage 20% ... throwing away "old" filesystem cache
> blocks to cache ... directory ... 400GB ... blocks are read only once ...
> thrown away a couple of hours later ...
> ...
> ... "RsyncClientPath" ... to "/usr/bin/nocache /usr/bin/rsync".
>
> Now the blocks read during backup are not going into the cache ...
>
> So what do you think? Is this a good solution? Or are there any pitfalls?

Does using 20% CPU for a couple of hours actually matter?

Presumably by doing this you'll be defeating _all_ caching of anything
you're backing up, not just caching of the 400GB directory.  That will
include directory contents as well as file contents, and may possibly
be particularly relevant to incremental backups.  You're obviously
well in tune with your systems so I guess you'll probably notice any
untoward effects.  I'd expect significantly increased disc activity
because of directory reads which are no longer cached.  That might be
more relevant for the life expectancy of spinning rust than for SSDs.
It might mean that the backups take significantly longer.  I'm sure
we'll all be interested in your reports, if you find such effects.

In a case like this as the data you're backing up is relatively static
I think I'd be tempted to put the 400GB directory in a share all of
its own, and back it up on a different schedule.  But the 'nocache'
utility was created expressly for your use case so I don't think it's
likely that there will be any serious downsides unless there's some
terrible flaw in the implementation which I guess can't be ruled out.
I'd be reluctant to do this to my backups for that reason.

FWIW where I have for example large database directories - of course
not necessarily static data - I don't let BackupPC anywhere near them.
Instead I use the DB facilities to create data dumps at intervals (via
cron), and then back up the data dumps using rsync (again via cron).

-- 

73,
Ged.

[BackupPC-users] Cache Poisoning- solution?

[Christian V. <cvo...@kn...>]

Hi,

I noticed during a rsync based backup on one of my larger Debian12 
BaclupPC client host a high CPU usage 20%) of the kswapd. After digging 
deeper I noticed it is doing some sort of throwing away "old" filesystem 
cache blocks to cache the recent read blocks.

In this scenarion, this does not make sense as th directory in question 
is a very stable and very less read directory- mostly just read by 
BackupPC. Unfortunately I is veryl large in size (around 400GB). So when 
BackupPC is doing a backup the kswapd throws away the current cache to 
free it for the to-be-backed-up directory. The blocks are read only once 
and after this thrown away a couple of hours later when the default 
workload kicks in.

So it does not make sense to keep this in cache and  waste thememory for 
the real cache. As you can not (afaik) disable caching for a directory I 
figured out there is a program which might be helpful. On Debian 12 it 
is simply called "nocache".

So I edited my "RsyncClientPath" and set it to "/usr/bin/nocache 
/usr/bin/rsync".

Now the blocks read during backup are not going into the cache and the 
cache does not get polluted and is keept for proper workload.

So what do you think? Is this a good solution? Or are there any pitfalls?

Greetings

/CV

Re: [BackupPC-users] How to repair "Unexpected magic number read from attrib-file"

[<Mat...@gm...>]

Solving the other issue, "error while loading shared libraries: libdl.so.2", with ulimit -u 10000
leads to finish the migration successfull.
Probably the "old" attrib files are corrupt and a restore couldn't restore correct attributes like
owner and access rights. But this is a minor issue for me 😊️

Best regards
Matthias

Am Dienstag, dem 29.07.2025 um 19:29 +0200 schrieb Mat...@gm...:
> Dear all,
> 
> I tried to migrate a  very old V3 backup to V4:
>    sudo -u backuppc /usr/share/backuppc/bin/BackupPC_migrateV3toV4 -h firewall -v -n 978
> During migration I get a lot of messages like:
>    BackupPC_migrateV3toV4: can't read attribute file ./f%2f/fvar/fstate/attribbpc_attrib_dirRead:
> can't open ./f%2f/fvar/fstate/attrib
>    
> It looks like attrib-file is defect or corrupt.
> sudo -u backuppc /usr/share/backuppc/bin/BackupPC_attribPrint
> /var/lib/backuppc/pc/firewall/978.old/f%2f/fvar/fstate/attrib delivers:
>    Unexpected magic number 0xd7da130f read from
> /var/lib/backuppc/pc/firewall/978.old/f%2f/fvar/fstate/attrib
>    $attrib = {
>        'dhcp' => {
>           'compress' => 1,
>           'digest' => '',
>           'gid' => 0,
>           'inode' => 0,
>           'mode' => 16877,
>           'mtime' => 1280460863,
>           'name' => 'dhcp',
>           'nlinks' => 0,
>           'size' => 0,
>           'type' => 5,
>           'uid' => 0
>        },
>    };
> Is it possible to repair attrib files or ignore attributes but migrate the backup to V4 anyhow?
> 
> Thanks in advance
> Matthias

Re: [BackupPC-users] BackupPC_migrateV3toV4: /usr/bin/perl: error while loading shared libraries: libdl.so.2 - solved

[<Mat...@gm...>]

Sorry - as Ged already answered in another post:
   ulimit -n 10000
solved this issue.

Br
Matthias

Am Dienstag, dem 29.07.2025 um 19:29 +0200 schrieb Mat...@gm...:
> During migration of very old V3 backups to V4, I got a strange error:
>    :
>    bpc_attrib_dirRead: can't open ./f%2f/fvar/fspool/attrib
>    BackupPC_migrateV3toV4: can't read attribute file ./f%2f/fvar/fstate/attrib
>    bpc_attrib_dirRead: can't open ./f%2f/fvar/fstate/attrib
>    bpc_poolRefFileWrite: can't open/create pool delta file name
> /var/lib/backuppc/pc/firewall/978.v4/refCnt/tpoolCntDelta_0_-1_0_2013339 (errno 24)
>    bpc_poolRefRequestFsck: can't open/create fsck request file
> /var/lib/backuppc/pc/firewall/978.v4/refCnt/needFsck2013339 (errno 24)
>    firewall #978: 93.0%
>    BackupPC_migrateV3toV4: converted backup in /var/lib/backuppc/pc/firewall/978; removing
> /var/lib/backuppc/pc/firewall/978.old
>    /usr/bin/perl: error while loading shared libraries: libdl.so.2: cannot open shared object
> file: Error 24
>    BackupPC_migrateV3toV4: BackupPC_refCountUpdate exited with errors
>    BackupPC_migrateV3toV4: got 1041 errors
> 
> This is strange because a lot of backups are migrated without this error. Only backups with
> corrupt attrib-files finished with this message and my$LD_LIBRARY_PATH is pointing to libdl.so.2
> 
> Thanks in advance
> Matthias
> 
> 

[BackupPC-users] How to repair "Unexpected magic number read from attrib-file"

[<Mat...@gm...>]

Dear all,

I tried to migrate a  very old V3 backup to V4:
   sudo -u backuppc /usr/share/backuppc/bin/BackupPC_migrateV3toV4 -h firewall -v -n 978
During migration I get a lot of messages like:
   BackupPC_migrateV3toV4: can't read attribute file ./f%2f/fvar/fstate/attribbpc_attrib_dirRead: can't
   open ./f%2f/fvar/fstate/attrib
   
It looks like attrib-file is defect or corrupt.
sudo -u backuppc /usr/share/backuppc/bin/BackupPC_attribPrint
/var/lib/backuppc/pc/firewall/978.old/f%2f/fvar/fstate/attrib delivers:
   Unexpected magic number 0xd7da130f read from
   /var/lib/backuppc/pc/firewall/978.old/f%2f/fvar/fstate/attrib
   $attrib = {
      'dhcp' => {
         'compress' => 1,
         'digest' => '',
         'gid' => 0,
         'inode' => 0,
         'mode' => 16877,
         'mtime' => 1280460863,
         'name' => 'dhcp',
         'nlinks' => 0,
         'size' => 0,
         'type' => 5,
         'uid' => 0
      },
   };
Is it possible to repair attrib files or ignore attributes but migrate the backup to V4 anyhow?

Thanks in advance
Matthias

[BackupPC-users] BackupPC_migrateV3toV4: /usr/bin/perl: error while loading shared libraries: libdl.so.2

[<Mat...@gm...>]

During migration of very old V3 backups to V4, I got a strange error:
   :
   bpc_attrib_dirRead: can't open ./f%2f/fvar/fspool/attrib
   BackupPC_migrateV3toV4: can't read attribute file ./f%2f/fvar/fstate/attrib
   bpc_attrib_dirRead: can't open ./f%2f/fvar/fstate/attrib
   bpc_poolRefFileWrite: can't open/create pool delta file name
   /var/lib/backuppc/pc/firewall/978.v4/refCnt/tpoolCntDelta_0_-1_0_2013339 (errno 24)
   bpc_poolRefRequestFsck: can't open/create fsck request file
   /var/lib/backuppc/pc/firewall/978.v4/refCnt/needFsck2013339 (errno 24)
   firewall #978: 93.0%
   BackupPC_migrateV3toV4: converted backup in /var/lib/backuppc/pc/firewall/978; removing
   /var/lib/backuppc/pc/firewall/978.old
   /usr/bin/perl: error while loading shared libraries: libdl.so.2: cannot open shared object file:
   Error 24
   BackupPC_migrateV3toV4: BackupPC_refCountUpdate exited with errors
   BackupPC_migrateV3toV4: got 1041 errors

This is strange because a lot of backups are migrated without this error. Only backups with corrupt
attrib-files finished with this message and my$LD_LIBRARY_PATH is pointing to libdl.so.2

Thanks in advance
Matthias

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[G.W. H. <ba...@ju...>]

Hi there,

On Sun, 27 Jul 2025, bac...@ko... wrote:

> Love to get some feedback on it first...

Well since you ask...

1. The month in the 'Version' line is wrong:

8<----------------------------------------------------------------------
# Version 0.2 June 2025 
# 
# CHANGELOG: 
# 0.1 (June 2025) 
# 0.2 (July 2025) 
8<----------------------------------------------------------------------

2. You might need to tell people what to do with this:

8<----------------------------------------------------------------------
use lib "/usr/share/backuppc/lib"; 
8<----------------------------------------------------------------------

3. and possibly this:

8<----------------------------------------------------------------------
my $bpc = BackupPC::Lib->new('/var/lib/backuppc', undef, undef, 1)
8<----------------------------------------------------------------------

4. When you output "ServerConnect failed" it could be useful to say
why you think it failed.

5. This could be a command-line option or something:

8<----------------------------------------------------------------------
my $lookback_days = 365; # Configurable lookback period for recent backups
8<----------------------------------------------------------------------

6. I find the left justification of all the numbers not very readable,
and I'm not wild about the commas in the numbers but I can live with it.
Configuration options?

7. I'd be inclined to put a decimal digit in the TIME column numbers.
Configuration option?

8. It could be helpful to be able to set (e.g. command line options)
ratios which define what "differ substantially" means for MAX_FILES/
MAX_SIZE and the current values, and to flag any outliers found.

9. It could be really useful to get dates for when the MAX_FILES and
MAX_SIZE values were found.  For example, although I've a pretty good
idea why one of my servers was up to 904,000 files at some point but
it's down at around 270,000 now - with hardly any difference in the
SIZE - I'd like to check that I'm right.  Maybe a 'verbose' option to
log all that stuff?

10. Generally it's a great piece of work, nicely laid out, very
readable code, and it's probably worth a mention in Github issue #365.

11. I ran it a few times, it seems to work but I didn't stress-test it.

*Now* will you put it in the wiki? :)

-- 

73,
Ged.

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[<bac...@ko...>]

Love to get some feedback on it first...
G.W. Haywood wrote at about 13:48:52 +0100 on Saturday, July 26, 2025:
 > Hi there,
 > 
 > On Sat, 26 Jul 2025, bac...@ko... wrote:
 > 
 > > FYI, here is a perl script I recently wrote to summarize my backups...
 > 
 > How about adding it to the wiki?
 > 
 > https://github.com/backuppc/backuppc/wiki
 > 
 > -- 
 > 
 > 73,
 > Ged.
 > 
 > 
 > _______________________________________________
 > BackupPC-users mailing list
 > Bac...@li...
 > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:    https://github.com/backuppc/backuppc/wiki
 > Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[G.W. H. <ba...@ju...>]

Hi there,

On Sat, 26 Jul 2025, bac...@ko... wrote:

> FYI, here is a perl script I recently wrote to summarize my backups...

How about adding it to the wiki?

https://github.com/backuppc/backuppc/wiki

-- 

73,
Ged.

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[<bac...@ko...>]

FYI, here is a perl script I recently wrote to summarize my backups at
one glance so I can identify missed backups, errors, missed files etc.
It organizes a lot of data in a simple text file.
I call the script daily from cron.
It doesn't *need* to be run as 'backuppc' (or root) though it works
better if it does.

It's not fully polished but it works well
--------------------------------------------------------------------------------
#!/usr/bin/perl
#========================================================================
#
# BackupPC_summarizeBackups.pl
#                       
#
# DESCRIPTION

#   Summarize status of latest backups (full and incremental) with one
#   (long) tabular line per host.
#
#   Provides the following columns per host
#     General Status:
#       HOST             Name of host
#       STATUS           Current status
#                          Idle - if idle
#                          NNNNm - Active time in minutes if backing up
#                          Fail - if last backup failed
#                          Man - if set to manual backups (BackupsDisable = 1)
#                          Disab - if backups disabled (BackupsDisable = 2)
#      LAST              Fractional days since last backup (full or incremental)
#
#    Full Backup Status:
#      FULL              Fractional days since last full backup
#      FILES             Number of files in last full
#      SIZE              Size of last full
#      TIME              Time to complete last full (in minutes)
#      ERRS/BAD          Number of errors/bad files in last full
#
#    Incremental  Backup Status:
#      INCR              Fractional days since last incremental backup
#      FILES             Number of files in last incremental
#      SIZE              Size of last incremental
#      TIME              Time to complete last incremental (in minutes)
#      ERRS/BAD          Number of errors/bad files in last incremental
#
#    Sanity Checks (worry if current numbers differ substantially from this)
#      MAX_FILES         Maximum files in past $lookback_days (default 365)
#      MAX_SIZE          Max backup size in past $lookback_days (default 365)
#
# AUTHOR
#   Jeff Kosowsky
#
# COPYRIGHT
#   Copyright (C) 2025  Jeff Kosowsky
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#
#========================================================================
#
# Version 0.2 June 2025
#
# CHANGELOG:
# 0.1 (June 2025)
# 0.2 (July 2025)
#
#========================================================================
use strict;
use warnings;
use lib "/usr/share/backuppc/lib";
use BackupPC::Lib;
use POSIX qw(strftime);

# Configuration
my $lookback_days = 365; # Configurable lookback period for recent backups
my $host_width = 20; #Max width of host name

# Initialize BackupPC
my $bpc = BackupPC::Lib->new('/var/lib/backuppc', undef, undef, 1)
    or die "Failed to initialize BackupPC: $@\n";
my $Conf = $bpc->ConfigDataRead()
    or die "Failed to load config: $@\n";
my $now = time;

# Get Status and Info
my $Status = {};
my $Info = {};
my $err = $bpc->ServerConnect();
if (!$err) { # Read dynamically via Server_Message (requires backuppc or root privilege)
    my $reply = $bpc->ServerMesg("status hosts");
    if (!defined $reply) {
        warn "ServerMesg for hosts returned undef\n";
    } else {
        my %Status;
        eval($reply);
        if ($@) {
            warn "Eval error for hosts: $@\n";
        } else {
            $Status = \%Status;
        }
    }
    $reply = $bpc->ServerMesg("status info"); # Get system info including pool stats
    if (!defined $reply) {
        warn "ServerMesg for info returned undef\n";
    } else {
        my %Info;
        eval($reply);
        if ($@) {
            warn "Eval error for info: $@\n";
        } else {
            $Info = \%Info;
        }
    }
} else { # Read from 'logDir/status.pl' using StatusDataRead
    warn "ServerConnect failed, falling back to status.pl\n";
    my $log_dir = $bpc->LogDir
        or die "LogDir not defined\n";
    my $status_file = "$log_dir/status.pl";
    if (!(-f $status_file && -r $status_file)) {
        warn "$status_file not found or not readable\n";
    } else { # Read from logDir/status.pl  using StatusDataRead
        ($Status, $Info) = $bpc->{storage}->StatusDataRead();
        if (!defined $Status ||  ref($Status) ne 'HASH') {
            warn "Failed to read status from $status_file\n"
		unless defined($Status);
            $Status = {};
        }
        if (!defined $Info || ref($Info) ne 'HASH') {
            warn "Failed to read valid Info from $status_file\n";
            $Info = {};
        }
    }
}

# Check if BackupPC is running
my $backuppc_running = defined($Info->{pid});

# Print warning if BackupPC is not running
print "***WARNING*** BackupPC not running!\n" unless $backuppc_running;

# Print header
printf "%-*s %-8s %-7s | %-7s %-10s %-10s %-7s %-10s | %-7s %-10s %-10s %-7s %-10s | %-10s %-10s\n",
    $host_width + 1,
    "HOST", "STATUS", "LAST",
    "FULL", "FILES", "SIZE", "TIME", "ERRS/BAD",
    "INCR", "FILES", "SIZE", "TIME", "ERRS/BAD", 
    "MAX_FILES", "MAX_SIZE";

# Get host list
my $hosts = $bpc->HostInfoRead()
    or die "Failed to read hosts: $@\n";
my @host_data;
my @host_nobackups;

foreach my $host (keys %$hosts) {
    my $status = "Idle";
    my $days_since_last = 999999;
    my $days_since_full = undef;
    my $days_since_incr = undef;
    my $files_full = '-';
    my $size_full = '-';
    my $time_full = '-';
    my $errs_bad_full = '-';
    my $files_incr = '-';
    my $size_incr = '-';
    my $time_incr = '-';
    my $errs_bad_incr = '-';
    my $max_files = 0;
    my $max_size = 0;
    my $has_recent_backup = 0;
    my $has_valid_backup = 0;
    my $last_backup_time = 0;

    # Get current status
    my $host_status = $Status->{$host} // {};
    if (defined $host_status->{job} && $host_status->{job} eq "Backup" &&
        defined $host_status->{reason} && $host_status->{reason} eq "Reason_backup_in_progress") {
        my $start_time = $host_status->{startTime} // $now;
        $status = sprintf "%dm", ($now - $start_time) / 60;
    } elsif (defined $host_status->{backoffTime} && $host_status->{backoffTime} > $now) {
        $status = sprintf "[%.1fh]", ($host_status->{backoffTime} - $now) / 3600;
    } elsif (defined $host_status->{reason} && $host_status->{reason} eq "Reason_backup_queued") {
        $status = "QUEUE";
    } elsif (defined $host_status->{reason} && $host_status->{reason} eq "Reason_host_disabled") {
        $status = "Disab";
    } elsif (defined $host_status->{error}) {
        $status = "Fail";
    } else {
        my $backups_disable = $Conf->{BackupsDisable} // 0; #General config.pl configuration
        my $host_conf = $bpc->ConfigDataRead($host); #Host-specific configuration in confDir/pc
        $backups_disable = $host_conf->{BackupsDisable} if $host_conf && defined $host_conf->{BackupsDisable};
        if ($backups_disable == 1 && $status eq "Idle") {
            $status = "Man";
        } elsif ($backups_disable == 2 && $status eq "Idle") {
            $status = "Disab";
        }
    }

    # Get backups
    my @backups = $bpc->BackupInfoRead($host);

    foreach my $backup (@backups) {
        my $type = $backup->{type} // '';
        my $backup_time = $backup->{startTime} || 0;
        next if !$type || $type eq "partial";

        my $n_files = $backup->{nFiles} || 0;
        my $size = $backup->{size} || 0; # Bytes
        my $backup_duration = ($backup->{endTime} && $backup->{startTime}) ? 
                              commify(int(($backup->{endTime} - $backup->{startTime}) / 60 + 0.5)) : '-';

        my $xfer_errs = $backup->{xferErrs} || 0;
	my $xfer_bad =  $backup->{xferBadFile} || 0;

        if ($type eq "active" && $backup_time) {
            $status = sprintf "%dm", ($now - $backup_time) / 60; # Time since backup start in minutes
            $has_valid_backup = 1;
        }

        if ($type eq "full" || $type eq "incr") {
            $has_valid_backup = 1;
            if ($backup_time > $last_backup_time) {
                $last_backup_time = $backup_time;
                $days_since_last = sprintf "%.1f", ($now - $backup_time) / 86400;
            }
        }

        if ($type eq "full" && (!defined $days_since_full || $backup_time >= ($now - ($days_since_full * 86400)))) {
            $days_since_full = sprintf "%.1f", ($now - $backup_time) / 86400;
            $files_full = commify($n_files);
            $size_full = commify(int($size / 1048576)); # MiB
            $time_full = $backup_duration;
            $errs_bad_full = sprintf "%d/%d", $xfer_errs, $xfer_bad;
        }

        if ($type eq "incr" && (!defined $days_since_incr || $backup_time >= ($now - ($days_since_incr * 86400)))) {
            $days_since_incr = sprintf "%.1f", ($now - $backup_time) / 86400;
            $files_incr = commify($n_files);
            $size_incr = commify(int($size / 1048576)); # MiB
            $time_incr = $backup_duration;
            $errs_bad_incr = sprintf "%d/%d", $xfer_errs, $xfer_bad;
        }

        if ($backup_time > $now - $lookback_days * 86400) {
            $has_recent_backup = 1;
            $max_files = $n_files if $n_files > $max_files;
	    $max_size= $size if $size > $max_size;
        }
    }

    if (!$has_valid_backup) {
        push @host_nobackups, {
	    host => $host, 
	    status => $status,
	};
        next;
    }

    my $max_files_output = $has_recent_backup ? commify($max_files) : '-';
    my $max_size_output = $has_recent_backup ? commify(int($max_size / 1048576)) : '-';

    push @host_data, {
        host            => $host,
        days_since_last => $days_since_last,
        status          => $status,
        days_since_full => $days_since_full // '-',
        files_full      => $files_full,
        size_full       => $size_full,
        time_full       => $time_full,
        errs_bad_full   => $errs_bad_full,
        days_since_incr => $days_since_incr // '-',
        files_incr      => $files_incr,
        size_incr       => $size_incr,
        time_incr       => $time_incr,
        errs_bad_incr   => $errs_bad_incr,
        max_files       => $max_files_output,
        max_size        => $max_size_output,
	recent_backup   => $has_recent_backup,
    };
}

# Sort by LAST column (ascending), then hostname
@host_data = sort {
    $a->{days_since_last} <=> $b->{days_since_last} ||
    $a->{host} cmp $b->{host}
} @host_data;

# Print hosts with backups
my $last_had_star = 0;
foreach my $data (@host_data) {
    if (!$data->{recent_backup} && !$last_had_star) {
        print "\n";
    }
    printf "%s%-*s %-8s %-7s | %-7s %-10s %-10s %-7s %-10s | %-7s %-10s %-10s %-7s %-10s | %-10s %-10s\n",
	$data->{recent_backup} ? ' ' : '*',
	$host_width,
        substr($data->{host}, 0, $host_width),
        $data->{status},
        $data->{days_since_last} == 999999 ? '-' : $data->{days_since_last},
        $data->{days_since_full},
        $data->{files_full},
        $data->{size_full},
        $data->{time_full},
        $data->{errs_bad_full},
        $data->{days_since_incr},
        $data->{files_incr},
        $data->{size_incr},
        $data->{time_incr},
        $data->{errs_bad_incr},
        $data->{max_files},
        $data->{max_size};
    $last_had_star = !$data->{recent_backup};
}

# Print hosts with no backups
print "\n";
foreach my $data (sort { $a->{host} cmp $b->{host} } @host_nobackups) {
    printf "%-*s %-8s\n", $host_width+1, "*" . substr($data->{host}, 0, $host_width), $data->{status};
}

# Print pool statistics
print "\nPool Statistics:\n";
printf "%-15s %-15s %-15s\n", "", "Pool", "CPool";
printf "%-15s %-15s %-15s\n", " " x 15, "-" x 15, "-" x 15;
printf "%-15s %-15s %-15s\n", "Files:", 
    commify($Info->{poolFileCnt} || 0), 
    commify($Info->{cpool4FileCnt} || 0);
printf "%-15s %-15s %-15s\n", "Size (GiB):", 
    commify(sprintf("%.2f", ($Info->{poolKb} || 0) / (1024**2))), 
    commify(sprintf("%.2f", ($Info->{cpool4Kb} || 0) / (1024**2)));
printf "%-15s %-15s %-15s\n", "Max Links:", commify($Info->{poolFileLinkMax} || 0), commify($Info->{cpool4FileLinkMax} || 0);
printf "%-15s %-15s %-15s\n", "Removed Files:", commify($Info->{poolFileCntRm} || 0), commify($Info->{cpool4FileCntRm} || 0);

# Add commas to numbers
sub commify {
    my $num = shift;
    return $num if $num eq '-';  # special case you added

    my ($int, $dec) = $num =~ /^(-?\d+)(\.\d+)?$/;
    return $num unless defined $int;

    $int = reverse $int;
    $int =~ s/(\d{3})(?=\d)/$1,/g;
    $int = reverse $int;

    return defined $dec ? "$int$dec" : $int;
}

--------------------------------------------------------------------------------

G.W. Haywood wrote at about 14:08:11 +0100 on Friday, July 25, 2025:
 > Hi there,
 > 
 > On Fri, 25 Jul 2025, Matthew Pounsett wrote:
 > 
 > > .. what I'm more interested in doing is setting up something to
 > > interpret BackupPC_serverMesg output so that I can get backup status
 > > into my actual monitoring system.
 > 
 > Two things.
 > 
 > First, there's a script here
 > 
 > https://github.com/moisseev/BackupPC_report
 > 
 > which you might find useful.  I've run it a couple of times and it
 > seems to do what it's supposed to do but I can't say more than that.
 > If it doesn't actually do what you want at least it will be a start.
 > 
 > Second, please take a look at github issue #365 and please feel free
 > to comment on the sort of things that you'd find useful in an API.
 > 
 > -- 
 > 
 > 73,
 > Ged.
 > 
 > 
 > _______________________________________________
 > BackupPC-users mailing list
 > Bac...@li...
 > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:    https://github.com/backuppc/backuppc/wiki
 > Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[G.W. H. <ba...@ju...>]

Hi there,

On Fri, 25 Jul 2025, Matthew Pounsett wrote:

> .. what I'm more interested in doing is setting up something to
> interpret BackupPC_serverMesg output so that I can get backup status
> into my actual monitoring system.

Two things.

First, there's a script here

https://github.com/moisseev/BackupPC_report

which you might find useful.  I've run it a couple of times and it
seems to do what it's supposed to do but I can't say more than that.
If it doesn't actually do what you want at least it will be a start.

Second, please take a look at github issue #365 and please feel free
to comment on the sort of things that you'd find useful in an API.

-- 

73,
Ged.

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[Matthew P. <ma...@co...>]

On Wed, Jul 23, 2025 at 10:08 AM G.W. Haywood <ba...@ju...>
wrote:

>
> The first is that investigation can be started relatively painlessly.
> If you look around line 1760 in .../bin/BackupPC you should find sub
> HostSortCompare.  Here are the comments at its head:
>

Thanks for the pointers.  I'll hold on to this in case I have time to dig
more deeply into what was going on (or if I have occasion to run into this
again).


>
> If you don't have the time either, then maybe raise an issue on Github
> (WHAT AM I SAYING????) and put a link to this thread in there.
>
>
I had decided there was no value in doing that until I saw the email that
development was being picked up again. :)   Now it seems like opening an
issue might actually be useful.


> > 4) Enable monitoring so that hosts with backups that are too old are
> > notified so that you can resolve the issue, instead of finding out later
> > after some potential failure
> > ...
>
> +1
>
> In my config.pl I've changed $Conf{EMailNotifyOldBackupDays} from the
> default of 7 days to 1.2.  I'm thinking about increasing it to 1.5. :)
> This notification system does work.  I have several hosts which don't
> get switched on very often and I get almost daily emails about them.
>
> But it sounds like this notification didn't happen for Mr. Pounsett
> even though the host was not backed up for two weeks, is that right?
> If so, this needs attention.
>

I did get the email notification after about a week... I don't remember the
precise timing now, but if that turned up on a weekend I'd likely not have
noticed it for a few days either.  After that I spent a few days watching
it to try to figure out why it wasn't getting backed up, and then started
scheduling backups by hand.

Changing the notification threshold is probably a good choice, but what I'm
more interested in doing is setting up something to interpret
BackupPC_serverMesg output so that I can get backup status into my actual
monitoring system.

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[Matthew P. <ma...@co...>]

On Tue, Jul 22, 2025 at 10:06 PM Adam Goryachev via BackupPC-users <
bac...@li...> wrote:

>
> This is not what I saw.   I let the problem host get up to 8 days without
> a backup before I realized we had a problem and I had to start manually
> queueing a backup every couple days.  Backuppc appeared to be grabbing
> whatever was at the top of the queue, not whatever had the least-recent
> last backup.
>
> Now that I'm no longer on MaxBackups=1, most backups are
> completed early in the day so by the time I get to my desk the Current
> Queue is mostly empty, and harder to observe what its behaviour is.
>
> I think you can validate the behavior easily by viewing the queue page. At
> each wakeup time, all hosts are added to the queue, seemingly in
> alphabetical order.
>
> If there is a spare "backup slot" ie, MaxBackups is > currently in
> progress, then a host is taken from the top of the queue, and evaluated
> whether a backup should start or not (blackout period, last backup time,
> etc)
>
> Repeat the above, until backups in progress == MaxBackups, then pause and
> do not continue to process the queue until a backup completes
>

Yes, this is pretty much what I saw.


> So, if your host is sorted later in the list, and you are unable to
> complete all backups during the backup window, then the host will indeed
> never be backed up.
>

I'd like to reiterate that the "backup window" for most hosts is 24 hours
long.  There was just the one host with a narrow window.,


> There is a valid reason to sort the queue for "oldest" backup, however,
> this may simply break someone else's workflow, for example, schedule the
> important host to have a backup every hour, and some other host to backup
> once a week, so the "oldest" is the once a week, even though it isn't
> really due.
>
Yeah, this is why I suggested the other option, which is to skip but not
remove hosts from the queue that are due for backup but blocked by their
blackout period.  They'd filter to the top of the list by the time their
blackout period ends, and then immediately get backed up.

 In your case, the suggestions would be:

> 1) Make sure all backups can be completed within the backup window allowed.
>
They can be.


> 2) Consider adding more wake up times, including at or close to midnight
>
I'm using the default: 23 wakeups per day.


> 3) Consider adding blackout windows to your other hosts to allow this one
> to complete at the 1am window first, and other hosts can only start at the
> next window after 1am.
>
That's not a bad workaround, actually... make it so that nothing else can
be backed up during the first two wakeups.


> 4) Enable monitoring so that hosts with backups that are too old are
> notified so that you can resolve the issue, instead of finding out later
> after some potential failure
>
Yeah... if I thought I'd have this problem regularly that'd be near the top
of my list.  As it is, it's already on my list, just waaay down there below
all the actual fires. :)   That said, it would help if BackupPC_serverMesg
was documented a little more completely.  There are some basics in the
docs, but it takes reading the code to know what it's actually able to do.

Thanks for the reply.

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[G.W. H. <ba...@ju...>]

Hi there,

On Tue, 22 Jul 2025, Matthew Pounsett wrote:

> On Fri, Jul 18, 2025 at 2:59?PM G.W. Haywood wrote:
>>
>> Are you saying that during the period in which you allow backups there
>> should be enough time to perform a backup for every client, but that a
>> client wasn't getting backed up despite that?
>
> Yes.
> ...
> ...

>> Since February 2020 I have had MaxBackups set at 1, and the blackout
>> period for all the clients set at Monday-Saturday from 07:30 to 23:30.
>
> No, that's not identical.  If all of your hosts have the same blackout
> period, then none of them can stomp on the blackout periods of other
> hosts.

Sorry, I somehow lost sight of the fact that the problem host had a
shorter blackout than the others.

Even though the problem has been resolved there are two reasons for
writing now.

The first is that investigation can be started relatively painlessly.
If you look around line 1760 in .../bin/BackupPC you should find sub
HostSortCompare.  Here are the comments at its head:

8<----------------------------------------------------------------------
# Compare function for host sort.  Hosts with errors go first,
# sorted with the oldest errors first.  The remaining hosts
# are sorted so that those with the oldest backups go first.
8<----------------------------------------------------------------------

If things aren't working this way then maybe something needs fixing.

When hosts are added to the queue, sub HostSortCompare is called by
sub QueueAllPCs (which is at around line 1875).

In sub QueueAllPCs there's this bit of code

8<----------------------------------------------------------------------
     foreach my $host ( sort HostSortCompare keys(%$Hosts) ) {
         $nSkip++ if ( QueueOnePC($host, $host, 'BackupPC', 'bg', 'auto') == 2 );
     }
     foreach my $dhcp ( @{$Conf{DHCPAddressRanges}} ) {
         for ( my $i = $dhcp->{first} ; $i <= $dhcp->{last} ; $i++ ) {
             my $ipAddr = "$dhcp->{ipAddrBase}.$i";
             $nSkip++ if ( QueueOnePC($ipAddr, $ipAddr, 'BackupPC', 'bg', 'dhcpPoll') == 2 );
         }
     }
8<----------------------------------------------------------------------

I don't have time to investigate this now.  If you want to dig further
then I suggest that when you next set MaxBackups to '1', immediately
before the first occurrence of the "$nSkip++ if ( ..." you could add a
log line such as

print(LOG $bpc->timeStamp, "Attempting to queue host $host\n");

and immedaitely begore the second occurrence  a line something like

print(LOG $bpc->timeStamp, "Attempting to queue IP $ipAddr\n");

You could obviously extend this to do more complex logging with a bit
of Perl tinkering.

If you don't have the time either, then maybe raise an issue on Github
(WHAT AM I SAYING????) and put a link to this thread in there.

The second and I think more important reason for writing now is

On Wed, 23 Jul 2025, Adam Goryachev wrote:

> ...
> 4) Enable monitoring so that hosts with backups that are too old are
> notified so that you can resolve the issue, instead of finding out later
> after some potential failure
> ...

+1

In my config.pl I've changed $Conf{EMailNotifyOldBackupDays} from the
default of 7 days to 1.2.  I'm thinking about increasing it to 1.5. :)
This notification system does work.  I have several hosts which don't
get switched on very often and I get almost daily emails about them.

But it sounds like this notification didn't happen for Mr. Pounsett
even though the host was not backed up for two weeks, is that right?
If so, this needs attention.

-- 

73,
Ged.

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[Adam G. <mai...@we...>]

On 23/7/2025 00:35, Matthew Pounsett wrote:
>
>
> On Fri, Jul 18, 2025 at 5:51 PM Christian Völker via BackupPC-users 
> <bac...@li...> wrote:
>
>
>     As far as I can say (and I see it here in my v4 setup) is: there
>     is an
>     order in the host list to be backed up. If a scheduled time occurs,
>     BackupPC takes the host for the next backup where the last backup
>     date
>     is the oldest. Of course, if checks then for some blackout
>     periods. So
>     it might skip this host. But then it'll grab the next from the
>     ordered
>     list of "last backup".
>
>
> This is not what I saw.   I let the problem host get up to 8 days 
> without a backup before I realized we had a problem and I had to start 
> manually queueing a backup every couple days.  Backuppc appeared to be 
> grabbing whatever was at the top of the queue, not whatever had the 
> least-recent last backup.
>
> Now that I'm no longer on MaxBackups=1, most backups are 
> completed early in the day so by the time I get to my desk the Current 
> Queue is mostly empty, and harder to observe what its behaviour is.

I think you can validate the behavior easily by viewing the queue page. 
At each wakeup time, all hosts are added to the queue, seemingly in 
alphabetical order.

If there is a spare "backup slot" ie, MaxBackups is > currently in 
progress, then a host is taken from the top of the queue, and evaluated 
whether a backup should start or not (blackout period, last backup time, 
etc)

Repeat the above, until backups in progress == MaxBackups, then pause 
and do not continue to process the queue until a backup completes

So, if your host is sorted later in the list, and you are unable to 
complete all backups during the backup window, then the host will indeed 
never be backed up.

There is a valid reason to sort the queue for "oldest" backup, however, 
this may simply break someone else's workflow, for example, schedule the 
important host to have a backup every hour, and some other host to 
backup once a week, so the "oldest" is the once a week, even though it 
isn't really due.

Admittedly, it would get skipped pretty quickly, since there is no 
backup due.

However, probably the better sort would be based on when the next backup 
is due, but I expect the "overhead" of performing this analysis is not 
worth the effort for implementation.

Possibly, hosts could be added to the queue based on when next backup is 
due, (ie, instead of being evaluated each time a host needs to be 
retrieved from the queue, it is evaluated once when adding the hosts to 
the queue), but again... probably not worth it.


In your case, the suggestions would be:

1) Make sure all backups can be completed within the backup window allowed.

2) Consider adding more wake up times, including at or close to midnight

3) Consider adding blackout windows to your other hosts to allow this 
one to complete at the 1am window first, and other hosts can only start 
at the next window after 1am.

4) Enable monitoring so that hosts with backups that are too old are 
notified so that you can resolve the issue, instead of finding out later 
after some potential failure


Either way, sounds like you have it resolved now, but always helpful to 
understand more about how BPC is working.

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[Matthew P. <ma...@co...>]

On Fri, Jul 18, 2025 at 5:51 PM Christian Völker via BackupPC-users <
bac...@li...> wrote:

>
> As far as I can say (and I see it here in my v4 setup) is: there is an
> order in the host list to be backed up. If a scheduled time occurs,
> BackupPC takes the host for the next backup where the last backup date
> is the oldest. Of course, if checks then for some blackout periods. So
> it might skip this host. But then it'll grab the next from the ordered
> list of "last backup".
>

This is not what I saw.   I let the problem host get up to 8 days without a
backup before I realized we had a problem and I had to start manually
queueing a backup every couple days.  Backuppc appeared to be grabbing
whatever was at the top of the queue, not whatever had the least-recent
last backup.

Now that I'm no longer on MaxBackups=1, most backups are completed early in
the day so by the time I get to my desk the Current Queue is mostly empty,
and harder to observe what its behaviour is.

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[Matthew P. <ma...@co...>]

On Fri, Jul 18, 2025 at 2:59 PM G.W. Haywood <ba...@ju...>
wrote:

>
> Is there anything in the logs which might shed any light?
>

Nothing that looked meaningful to me.


>
> Are you saying that during the period in which you allow backups there
> should be enough time to perform a backup for every client, but that a
> client wasn't getting backed up despite that?
>

Yes.


>
> If there was enough time then something seems wrong, but if the 8.1
> hours is too short I'd guess one way or another you'll have a problem.
> Buut I'd call it a configuration problem.  (BTW I don't think 23.9 in
> the blackout times means 23:59.  More like 23:54.)
>

Yeah, I was approximating. :)

The 8-ish hours is not enough time to back up all hosts, but only the one
host has that restriction.  With MaxBackups=1, I think all hosts were
getting backed up within about 16 hours.

The behaviour I was seeing is that there was a low probability that the
host with the blackout period would end up near the top of the queue
outside its blackout period.   It's possible the problem is compounded by
the fact that the end of its blackout period coincides with the first
wakeup.  (blackout period ends at ~midnight, but there is no 0h wakeup by
default... 1st wakeup is at 01:00).

What I was seeing is that the first 8 hours of the day, when this host
could be backed up, the queue was concentrating on other hosts.  Then
there'd be a long period at the end of the day where nothing was being
backed up.. because everything _except_ the host with the blackout period
had been done by then, and the blacked out host was.. well.. blacked out.


>
> Since February 2020 I have had MaxBackups set at 1, and the blackout
> period for all the clients set at Monday-Saturday from 07:30 to 23:30.
> This seems more or less identical to your setup yet I've never seen
> the symptoms you describe.  It was a fresh V4 install.
>

No, that's not identical.  If all of your hosts have the same blackout
period, then none of them can stomp on the blackout periods of other
hosts.  And obviously your entire collection of hosts can be backed up in
that period of time.  Mine can't.

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[Christian V. <cvo...@kn...>]

As far as I can say (and I see it here in my v4 setup) is: there is an 
order in the host list to be backed up. If a scheduled time occurs, 
BackupPC takes the host for the next backup where the last backup date 
is the oldest. Of course, if checks then for some blackout periods. So 
it might skip this host. But then it'll grab the next from the ordered 
list of "last backup".

So I am really surprised your host does not get backed up. It should 
start with the one if there have been no backups for a couple of days.

/KNEBB



Am 18.07.25 um 20:57 schrieb G.W. Haywood:
> Hi there,
>
> On Fri, 18 Jul 2025, Matthew Pounsett wrote:
>
>> For I/O preserving reasons that aren't really relevant to the list, I've
>> temporarily had my backup server on MaxBackups = 1 for a couple of 
>> weeks.
>> I also have a backup client that is on a tightly restricted 
>> BlackoutPeriod
>> of 8.0 through 23.9 (allowing backups from 23:59 through 08:00).
>>
>> The combination of these two things has resulted in the blackout server
>> never receiving an automatically queued backup for two straight weeks.
>> ...
>> ...
>> This is my guess about what's going on ...
>> ...
>> Thoughts?  Am I completely off base?
>
> Is there anything in the logs which might shed any light?
>
> Are you saying that during the period in which you allow backups there
> should be enough time to perform a backup for every client, but that a
> client wasn't getting backed up despite that?
>
> If there was enough time then something seems wrong, but if the 8.1
> hours is too short I'd guess one way or another you'll have a problem.
> Buut I'd call it a configuration problem.  (BTW I don't think 23.9 in
> the blackout times means 23:59.  More like 23:54.)
>
> Since February 2020 I have had MaxBackups set at 1, and the blackout
> period for all the clients set at Monday-Saturday from 07:30 to 23:30.
> This seems more or less identical to your setup yet I've never seen
> the symptoms you describe.  It was a fresh V4 install.
>
> The host status page is on a tab in my browser.  I keep it permanently
> open, and most mornings the first thing I'll do when I switch on my
> screen is check the 'Last Backup (days)' column on the host status
> page.  It's very unusual for me to see anything unexpected there, but
> if I do it's usually a network problem, a host down, or something like
> that.  I think the only times I have seen BackupPC fail to start the
> backup for a host is if I've dropped multiple gigabytes of junk in a
> silly place, so that a share that should take a few minutes to back up
> takes a couple of days, and nothing else gets a chance to be queued.
>
> I wonder if there's something else perhaps, er, unusual in your config
> which might have a bearing on the issue.  ISTR that you converted your
> backups from V3 to V4 quite recently.  I'm not sure how thoroughly the
> convert process has been exercised and I wonder if it's relevant. We
> probably should look at the entire config.  I guess it's not an issue
> for you if you now have MaxBackups at some higher number, but if you
> have any qualms you could always install a V4 system from scratch on a
> spare machine, making a bare minimum of configuration changes so that
> if thigs go awry it's easier to nail down.
>

Re: [BackupPC-users] Scheduler edge case (design flaw?)

[G.W. H. <ba...@ju...>]

Hi there,

On Fri, 18 Jul 2025, Matthew Pounsett wrote:

> For I/O preserving reasons that aren't really relevant to the list, I've
> temporarily had my backup server on MaxBackups = 1 for a couple of weeks.
> I also have a backup client that is on a tightly restricted BlackoutPeriod
> of 8.0 through 23.9 (allowing backups from 23:59 through 08:00).
>
> The combination of these two things has resulted in the blackout server
> never receiving an automatically queued backup for two straight weeks.
> ...
> ...
> This is my guess about what's going on ...
> ...
> Thoughts?  Am I completely off base?

Is there anything in the logs which might shed any light?

Are you saying that during the period in which you allow backups there
should be enough time to perform a backup for every client, but that a
client wasn't getting backed up despite that?

If there was enough time then something seems wrong, but if the 8.1
hours is too short I'd guess one way or another you'll have a problem.
Buut I'd call it a configuration problem.  (BTW I don't think 23.9 in
the blackout times means 23:59.  More like 23:54.)

Since February 2020 I have had MaxBackups set at 1, and the blackout
period for all the clients set at Monday-Saturday from 07:30 to 23:30.
This seems more or less identical to your setup yet I've never seen
the symptoms you describe.  It was a fresh V4 install.

The host status page is on a tab in my browser.  I keep it permanently
open, and most mornings the first thing I'll do when I switch on my
screen is check the 'Last Backup (days)' column on the host status
page.  It's very unusual for me to see anything unexpected there, but
if I do it's usually a network problem, a host down, or something like
that.  I think the only times I have seen BackupPC fail to start the
backup for a host is if I've dropped multiple gigabytes of junk in a
silly place, so that a share that should take a few minutes to back up
takes a couple of days, and nothing else gets a chance to be queued.

I wonder if there's something else perhaps, er, unusual in your config
which might have a bearing on the issue.  ISTR that you converted your
backups from V3 to V4 quite recently.  I'm not sure how thoroughly the
convert process has been exercised and I wonder if it's relevant.  We
probably should look at the entire config.  I guess it's not an issue
for you if you now have MaxBackups at some higher number, but if you
have any qualms you could always install a V4 system from scratch on a
spare machine, making a bare minimum of configuration changes so that
if thigs go awry it's easier to nail down.

-- 

73,
Ged.

[BackupPC-users] Scheduler edge case (design flaw?)

[Matthew P. <ma...@co...>]

For I/O preserving reasons that aren't really relevant to the list, I've
temporarily had my backup server on MaxBackups = 1 for a couple of weeks.
 I also have a backup client that is on a tightly restricted BlackoutPeriod
of 8.0 through 23.9 (allowing backups from 23:59 through 08:00).

The combination of these two things has resulted in the blackout server
never receiving an automatically queued backup for two straight weeks.
We're back to regular MaxBackups now, and the other server will return to
unrestricted hours soon, but I thought I'd bring this up in case there's a
design problem with the scheduler that can be addressed.

This is my guess about what's going on ... I assume someone in here knows
more than I do about the scheduler and can correct or confirm my
assumptions.

When BackupPC does a wakeup it adds every host to the queue, and doesn't
appear to apply any kind of sorting (is it explicitly random order?).  When
the server gets under MaxBackups and decides to start a new backup, it
appears to iterate over the current queue in the order listed, discarding
hosts that are not due for backups at all as it finds them.  The first host
it finds that is due for a backup of any kind, it starts the backup and
removes that host from the queue.  If it encounters a host that is due for
backup but in blackout, it discards/removes that host as if it were not due
for backup.

The end result is that a server that has a blackout period might never get
a backup, because the server will only execute a backup for it if, outside
of its blackout period, there are no servers ahead of it in the queue that
are due for backup.  The problem does not require that there be too many
backups to complete in a single BackupNightlyPeriod.  All it needs is for
there to be enough servers for there to be a low probability of a server
with a blackout period randomly appearing near the top of the queue outside
its blackout hours.

Assuming I'm right about the mechanisms in play, there are two fixes I can
think of (one can be changed in two places):
1a. hosts are added to the queue in reverse order of their "Last Backup
(days)" value from the Host Summary (larger numbers first).
1b. backuppc processes the queue in Last Backup order, larger numbers first.
2. when picking from the queue, hosts that are due for backup but in
blackout are skipped without being removed, leaving them in their queue
position

Thoughts?  Am I completely off base?

Re: [BackupPC-users] Problem accessing v3 backups

[Dave B. <dav...@ho...>]

It seems that I was the one who missed the obvious as I had expected that BackupPC_migrateV3toV4 would have allowed me to access the backups via the GUI.

Much thanx for continuing my education!

________________________________
From: Adam Goryachev via BackupPC-users <bac...@li...>
Sent: Tuesday, July 15, 2025 06:16
To: bac...@li... <bac...@li...>
Cc: Adam Goryachev <mai...@we...>
Subject: Re: [BackupPC-users] Problem accessing v3 backups



On 14/7/2025 07:45, Dave Bachmann wrote:
It's obvious that I didn't fully understand how to use BackupPC_ls and had moved on to try other techniques. I had expected that one or the other would have eventually led to having a GUI interface that no longer said Error: Directory XXXXX is empty, but instead listed the files available to be restored. Now that it seems that I'll be working at the command line exclusively, I'm in the process of trying to understand the usage and options of BackupPC_restore and BackupPC_zcat and expect that I'll have further questions coming.

I appreciate your patience with one who, in this case at least, knows what he doesn't know - even if he uses a word that he first learned in boot camp many years ago to describe his ignorance.


This might be silly of me, but in V3 you should be able to just access the files from the CLI under the "pc" directory....

For me, that is /var/lib/backuppc/pc/hostname/backup-num/f%2f/fetc/....


The path/filename is slightly munged, but you should be able to work it out, special chars/spaces etc will be "escaped" so "f%2f" is actually "/" and "fetc" is "etc".

When you find the right file, use BackupPC_zcat to get the original content.


Hope that helps, or I've missed something really obvious.....


Dave
________________________________
From: G.W. Haywood <ba...@ju...><mailto:ba...@ju...>
Sent: Sunday, July 13, 2025 06:15
To: bac...@li...<mailto:bac...@li...> <bac...@li...><mailto:bac...@li...>
Subject: Re: [BackupPC-users] Problem accessing v3 backups

Hello again,

On Sun, 13 Jul 2025, Dave Bachmann wrote:

> I'm not sure that I understand how I would know which digests are associated with a file that I was searching for. This, for example, is a typical result:
>    $ ./BackupPC_zcat /srv/backuppc/cpool/aa/aa/abaa09668efeb993980a19a1f83e52ad
>    VSS
>        Detector.so????????   ???>%???=??{~??$

> I assume that this is one chunk of a file named Detector.so, but ...

I was always told that 'assume' makes an 'ass' out of 'u' and 'me'. :)

No, it isn't one chunk.  It's the content of the whole file (and its
name very likely isn't 'Detector.so').

> clearly not the way to find and recreate all of the chunks of that file.

No need for anything like that.  The files are not stored in chunks,
they're stored whole, but usually (hopefully) in a compressed form as
you've seen.

> To repeat, what I *think* I need to do is to recreate the pool from
> the cpool as the first step in order to be able to find a file
> named, for example, "/home/common/data/tunes/A/Allman
> Brothers/Allman Brothers Band/Dreams.mp3" on a specific host. Is
> there a BackupPC_? script/function/setting/whatever to do that?

Well if you put it like that ... to repeat:

8<----------------------------------------------------------------------
On Thu, 10 Jul 2025, G.W. Haywood wrote:
> On Thu, 10 Jul 2025, Dave Bachmann wrote:
>
>> I am trying to see if I can recover a file that may have been
>> deleted years ago and *may* be on a very old backup disk. ...
>> ...
>
> Have you tried using the 'BackupPC_ls' script?
8<----------------------------------------------------------------------

You could do worse than read the V4 documentation, particularly the
part which begins "Here is a more detailed discussion:". :)

> Much thanx for your continued help!

You're very welcome.  In these exchanges I often learn as much as - if
not more than - the person I'm helping.

--

73,
Ged.


_______________________________________________
BackupPC-users mailing list
Bac...@li...<mailto:Bac...@li...>
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/





_______________________________________________
BackupPC-users mailing list
Bac...@li...<mailto:Bac...@li...>
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/