Re: [BackupPC-users] How does --checksum work in v4?

[Guillermo R. <gui...@gm...>]

I've attached the script I'm using. It's very rough, so use at your own risk!

I run it daily checking 4 folders of the pool per day, sequentially,
so it takes 32 days to check them all. You can modify the external
loop to change this. The last checked folder is saved in an auxiliary
file.

The checksum is done uncompressing the files in the pool using
zlib-flate (line 25), but it can be changed to pigz or BackupPC_zcat.
On my severely CPU-limited server (Banana Pi) both pigz and zlib-flate
are much faster than BackupPC_zcat, they take around a quarter of the
time to check the files (pigz is marginally faster than zlib-flate).
On the other hand, BackupPC_zcat puts the lowest load on the CPU,
zlib-flate's load is 30-35% higher, and pigz's is a whooping 80-100%
higher.

However, as BackupPC_zcat produces slightly modified gzip files, there
is a (very) small chance that a BackupPC_zcat compressed file is not
properly uncompressed by the other two (line 28 in the script). If
that happens, you need to re-check every zlib-flate or pigz failure
with BackupPC_zcat before calling it a real error. I think this gets
the best balance between load on the system and time spent checking
the pool (at least for my server and pool...).

Best regards,
Guillermo


On Mon, Jun 8, 2020 at 1:28 PM <bac...@ko...> wrote:
>
> Good point...
> Craig - would it make sense to add a parameter to BackupPC_nightly
> that would check a user-settable percentage of the files each night,
> say NightlyChecksumPercent. So if set to 3%, the pool would be checked
> (sequentially) over the period of ~1 month
>
> Guillermo Rozas wrote at about 11:12:39 -0300 on Monday, June 8, 2020:
>  > Yes, I wouldn't worry about collisions by chance.
>  >
>  > However, there is a second aspect that is not covered here: if you
>  > rely only on saved checksums in the server, it will not check again
>  > unmodified pool files. This risks you missing file system corruption
>  > or bit rot in the backup files that were previously caught by the V3
>  > behaviour (which periodically checksummed the pool files).
>  >
>  > Two solutions:
>  > - put the pool in a file system with checksum verification included
>  > - use a script to periodically traverse the pool and chesum the files
>  >
>  > Best regards,
>  > Guillermo
>  >
>  >
>  >
>  > On Mon, Jun 8, 2020 at 10:58 AM G.W. Haywood via BackupPC-users
>  > <bac...@li...> wrote:
>  > >
>  > > Hi there,
>  > >
>  > > On Mon, 8 Jun 2020, Jeff Kosowsky wrote:
>  > >
>  > > > ... presumably a very rare event ...
>  > >
>  > > That's putting it a little mildly.
>  > >
>  > > If it's really all truly random, then if you tried random collisions a
>  > > million times per picosecond you would (probably) need of the order of
>  > > ten trillion years to have a good chance of finding one...
>  > >
>  > > $ echo ' scale=2; 2^128 / 10^6 / 10^12 / 86400 / 365 / 10^12 ' | bc
>  > > 10.79
>  > >
>  > > I think it's safe to say that it's not going to happen by chance.
>  > >
>  > > If it's truly random.
>  > >
>  > > --
>  > >
>  > > 73,
>  > > Ged.
>  > >
>  > >
>  > > _______________________________________________
>  > > BackupPC-users mailing list
>  > > Bac...@li...
>  > > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
>  > > Wiki:    http://backuppc.wiki.sourceforge.net
>  > > Project: http://backuppc.sourceforge.net/
>  >
>  >
>  > _______________________________________________
>  > BackupPC-users mailing list
>  > Bac...@li...
>  > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
>  > Wiki:    http://backuppc.wiki.sourceforge.net
>  > Project: http://backuppc.sourceforge.net/
>
>
> _______________________________________________
> BackupPC-users mailing list
> Bac...@li...
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:    http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/