From: Craig B. <cba...@us...> - 2013-07-01 18:54:33
|
BackupPC community, I'm pleased to announce that BackupPC 4.0.0alpha1 has been released on SourceForge at: https://sourceforge.net/projects/backuppc/files/backuppc-beta/4.0.0alpha1/ BackupPC 4.0.0alpha1 contains a few bug fixes since BackupPC 4.0.0alpha0 was released just over a week ago. If you are testing BackupPC 4.0.0alpha0, I recommend you upgrade. Each of the three packages in the release has been updated. You should upgrade all three packages: - BackupPC-4.0.0alpha1.tar.gz: the usual BackupPC release tar ball. - BackupPC-XS-0.10.tar.gz: a perl XS module with C code that replaces several BackupPC perl libraries for improved performance. - rsync-bpc-3.0.9.1.tar.gz: a modified rsync that runs on the server that has a shim layer that interfaces directly to the BackupPC file system. As before, I do not recommend using this release in any production environment. Having some people try it out in a sandbox in their environment would be very helpful I'm attaching some short notes on the build/install steps for each package. Craig BackupPC-XS-0.10.tar.gz: tar zxvf BackupPC-XS-0.10.tar.gz cd BackupPC-XS-0.10 perl Makefile.PL make make test make install rsync-bpc-3.0.9.1.tar.gz: tar zxvf rsync-bpc-3.0.9.1.tar.gz cd rsync-bpc-3.0.9.1 ./configure.sh make make install BackupPC-4.0.0alpha1.tar.gz: tar zxvf BackupPC-4.0.0alpha1.tar.gz cd BackupPC-4.0.0alpha1 ./configure.pl The last step for each will need to be run as a privileged user. If you want to install rsync_bpc in /usr/local/bin (default might be /usr/bin), then you should add the --prefix option to configure.sh: ./configure.sh --prefix=/usr/local |
From: <bac...@ko...> - 2013-07-01 20:38:48
|
Craig Barratt wrote at about 11:53:57 -0700 on Monday, July 1, 2013: > BackupPC community, > > I'm pleased to announce that BackupPC 4.0.0alpha1 has been released on > SourceForge at: > Awesome... It would be great to hear feedback from early adopters... both to understand any potential issues and to build confidence and excitement from real world users... Maybe just keep such comments on the devel list if this is not interesting to all users. |
From: Craig B. <cba...@us...> - 2013-07-10 21:35:43
|
Jean, I should have been clearer in my explanation. This isn't a user feature in 4.0.0 that allows users to delete any backup. What I was trying to say was that 4.0.0 includes the ability to delete any backup, while maintaining the consistency of the reverse deltas. That simplifies the deletion logic. In contrast, 3.x doesn't allow a backup to be deleted that others depend on for the deltas. It's not a user-level feature though. Craig On Wed, Jul 10, 2013 at 12:36 AM, Ghislain <ga...@aq...> wrote: > hi, > > this is great news to see v4 in the radar, we love this wonderfull piece > of software and seing a new version is exciting :) > > I saw : Any backup can be deleted (deltas are merged into next older > backup if it is not filled). > > in the changelogs. Can this possibility be turned off or restricted on a > user/host/general basis in the web Gui ? > > One of the thing i love with backuppc is that even an hacked account > cannot be wiped out, limiting the famous "fired employee that logs and > wipe all including backup" one minute before leaving problem :) > > we encountered the issue once and the fact that backuppc was read only > saved the day. > > regards, > Jean > > |
From: Tomasz C. <ma...@wp...> - 2013-07-13 06:26:57
|
Any ideas how to get this compiled? This is on Debian 7. $ tar xpf BackupPC-XS-0.10.tar.gz $ cd BackupPC-XS-0.10 $ perl Makefile.PL Checking if your kit is complete... Looks good MakeMaker (v6.57_05) Warning (non-fatal): Target 'dynamic' depends on targets in skipped section 'dynamic_lib' Warning (non-fatal): Target 'static' depends on targets in skipped section 'static_lib' Writing Makefile for BackupPC::XS::md5 Writing MYMETA.yml MakeMaker (v6.57_05) Warning (non-fatal): Target 'dynamic' depends on targets in skipped section 'dynamic_lib' Warning (non-fatal): Target 'static' depends on targets in skipped section 'static_lib' Writing Makefile for BackupPC::XS::zlib Writing MYMETA.yml Writing Makefile for BackupPC::XS Writing MYMETA.yml root@srv4:~/BackupPC-XS-0.10# make ./configure.sh configure.sh: Configuring rsync 3.0.9 checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking whether to include debugging symbols... yes (...) checking whether to support ACLs... running tests: checking for acl_get_file in -lacl... no checking for ACL support... no checking ACL test results... No ACL support found checking whether to support extended attributes... Using Linux xattrs configure.sh: creating ./config.status config.status: creating config.h rsync 3.0.9 configuration successful cp lib/BackupPC/._XS.pm blib/lib/BackupPC/._XS.pm cp lib/._BackupPC blib/lib/._BackupPC cp lib/BackupPC/XS.pm blib/lib/BackupPC/XS.pm make[1]: Entering directory `/root/BackupPC-XS-0.10/md5' cc -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"\" -DXS_VERSION=\"\" -fPIC "-I/usr/lib/perl/5.14/CORE" ._md5.c ._md5.c:1:1: warning: null character(s) ignored [enabled by default] ._md5.c:1:2: error: stray ‘\5’ in program ._md5.c:1:2: error: stray ‘\26’ in program ._md5.c:1:2: error: stray ‘\7’ in program ._md5.c:1:5: warning: null character(s) ignored [enabled by default] ._md5.c:1:2: error: stray ‘\2’ in program ._md5.c:1:7: warning: null character(s) ignored [enabled by default] ._md5.c:1:9: error: unknown type name ‘Mac’ ._md5.c:1:16: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘X’ ._md5.c:1:17: warning: null character(s) ignored [enabled by default] ._md5.c:1:16: error: stray ‘\2’ in program ._md5.c:1:27: warning: null character(s) ignored [enabled by default] ._md5.c:1:35: warning: null character(s) ignored [enabled by default] ._md5.c:1:16: error: stray ‘\254’ in program ._md5.c:1:39: warning: null character(s) ignored [enabled by default] ._md5.c:1:16: error: stray ‘\2’ in program ._md5.c:1:43: warning: null character(s) ignored [enabled by default] ._md5.c:1:16: error: stray ‘\336’ in program ._md5.c:1:47: warning: null character(s) ignored [enabled by default] ._md5.c:1:89: warning: null character(s) ignored [enabled by default] ._md5.c:1:16: error: stray ‘\336’ in program ._md5.c:1:97: warning: null character(s) ignored [enabled by default] ._md5.c:1:16: error: stray ‘\230’ in program ._md5.c:1:101: warning: null character(s) ignored [enabled by default] ._md5.c:1:105: warning: null character(s) ignored [enabled by default] ._md5.c:1:16: error: stray ‘\1’ in program ._md5.c:1:121: warning: null character(s) ignored [enabled by default] ._md5.c:1:16: error: stray ‘\230’ in program ._md5.c:1:125: warning: null character(s) ignored [enabled by default] ._md5.c:1:129: warning: null character(s) ignored [enabled by default] ._md5.c:1:16: error: stray ‘\25’ in program ._md5.c:1:152: warning: null character(s) ignored [enabled by default] ._md5.c:1:160: error: invalid suffix "d1931d" on integer constant ._md5.c:1:160: error: expected identifier or ‘(’ before numeric constant ._md5.c:1:160: error: stray ‘\’ in program ._md5.c:1:169: error: unknown type name ‘Google’ ._md5.c:1:194: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘-’ token ._md5.c:1:195: error: invalid suffix "B59" on integer constant ._md5.c:1:200: error: invalid suffix "B" on floating constant ._md5.c:1:210: error: invalid suffix "D94FFFC0866" on integer constant ._md5.c:1:222: warning: null character(s) ignored [enabled by default] make[1]: *** [._md5.o] Error 1 make[1]: Leaving directory `/root/BackupPC-XS-0.10/md5' make: *** [subdirs] Error 2 -- Tomasz Chmielewski http://www.ptraveler.com |
From: Tomasz C. <ma...@wp...> - 2013-07-13 06:36:06
|
On Sat, 13 Jul 2013 14:26:40 +0800 Tomasz Chmielewski <ma...@wp...> wrote: > Any ideas how to get this compiled? This is on Debian 7. (...) > cc -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fstack-protector > -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE > -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"\" -DXS_VERSION=\"\" > -fPIC "-I/usr/lib/perl/5.14/CORE" ._md5.c ._md5.c:1:1: warning: > null character(s) ignored [enabled by default] ._md5.c:1:2: error: > stray ‘\5’ in program ._md5.c:1:2: error: stray ‘\26’ in > program ._md5.c:1:2: error: stray ‘\7’ in program ._md5.c:1:5: > warning: null character(s) ignored [enabled by default] ._md5.c:1:2: > error: stray ‘\2’ in program ._md5.c:1:7: warning: null character(s) > ignored [enabled by default] ._md5.c:1:9: error: unknown type name > ‘Mac’ ._md5.c:1:16: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or > ‘__attribute__’ before ‘X’ ._md5.c:1:17: warning: null character(s) Ah, there are some bogus files in the archive (some binary garbage with "Mac OS X" string in it?) - this seems to do the trick: find -name ._\* | xargs rm -- Tomasz Chmielewski http://www.ptraveler.com |
From: Craig B. <cba...@us...> - 2013-07-13 06:59:27
|
Yes - you have the right solution. These files shouldn't be in the release. I was using a new MacOSX machine, and I botched the final tarballs for alpha1 by using the default bsdtar on MacOSX (instead of gnu tar), and it added the MacOSX AppleDouble files, which contain metadata which is useless in this case. Craig |
From: Tomasz C. <ma...@wp...> - 2013-07-14 00:13:28
|
> - An rsync "full" backup now uses --checksum (instead of > --ignore-times), which is much more efficient on the server side - > the server just needs to check the full-file checksum computed by the > client, together with the mtime, nlinks, size attributes, to see if > the file has changed. If you want a more conservative approach, you > can change it back to --ignore-times, which requires the server to > send block checksums to the client. --checksum is quite paranoid and causes substantial IO on both sides. Basically, it would cause md5sum calculated for each and every file, on both sides. If the archive is dozens of gigabytes or more - it means: - extra CPU used, - extra IO used, - since we have to read all the files, anything the server had in cache/buffers, will be purged from there (newly read files go to cache/buffers instead, but it's not very useful there, since the backup will be most likely made daily or less often). All above cause visible slowdown. I'm pretty sure that my clients don't secretly change the file content, while preserving their size and timestamp. In that case - will BackupPC still work correctly if I change the default: $Conf{RsyncFullArgsExtra} = [ '--checksum', ]; to: $Conf{RsyncFullArgsExtra} = [ '', ]; ? -- Tomasz Chmielewski http://wpkg.org |
From: Craig B. <cba...@us...> - 2013-07-14 00:43:23
|
On Sat, Jul 13, 2013 at 5:13 PM, Tomasz Chmielewski <ma...@wp...> wrote: > > --checksum is quite paranoid and causes substantial IO on both sides. > Basically, it would cause md5sum calculated for each and every file, on > both sides. > If the archive is dozens of gigabytes or more - it means: > > - extra CPU used, > > - extra IO used, > No, that is not true. On the server side in 4.x, so long as that file was previously backed up with the same path for that client, the full-file md5sum is stored in the file attributes, so it takes no more effort to check the md5sum that comparing other attributes like the mtime and size. (Of course, if the file isn't in the pool already, the server needs to compress and write it to the pool, which is an expensive operation.) On the client side you are correct - the entire file needs to be read. But isn't that the point of a full backup? The client-side load is similar to version 3.x, where a full backup requires the client to also read the entire contents of every file. There are two (typically one-time) cases where the client might do a bit more work compared to 3.x: - If the file isn't in the pool at all, the server will send empty block checksums (ie: nothing to match), and the client will then send the whole file verbatim. That requires two reads of the file on the client (first to get the md5sum, and second to send the file). But these are close in time and the second read will probably be cached. - If the file is anywhere in the pool (even if it's a first backup of a brand new client), the right pool file will be a likely match (based on md5sum), and block digests will be sent to the client. If it is a match, very little network traffic is needed, but the client needs to re-read the whole file to be sure. So that also requires two reads of the file on the client, and one on the server. Again, the two client reads are close in time and the second read will probably be cached. - since we have to read all the files, anything the server had in > cache/buffers, will be purged from there (newly read files go to > cache/buffers instead, but it's not very useful there, since the > backup will be most likely made daily or less often). That's a good point. Igor Sverkos recently pointed out the fadvise patch to rsync that gives hints via the posix_fadvise call whether to cache certain files or not. That patch significantly reduces the impact you describe. However, in the steady state, rsync_bpc on the 4.x server isn't reading entire files very often. I think the patch is likely to help the client side more than the server, but I haven't tried it. I'm pretty sure that my clients don't secretly change the file content, > while preserving their size and timestamp. > In that case - will BackupPC still work correctly if I change the > default: > > $Conf{RsyncFullArgsExtra} = [ > '--checksum', > ]; > > > to: > > $Conf{RsyncFullArgsExtra} = [ > '', > ]; If you are comfortable with mtime, size etc catching all changes to files, then rather than disabling --checksum in $Conf{RsyncFullArgsExtra}, I recommend you only ever do incremental backups. 4.x supports that. Set $Conf{FullPeriod} to a very large value. If you are doing incremental-only, you should set $Conf{FillCycle} to, say, 7. This will make sure every 7th backup is filled. The "Full" backup delete settings (eg: "FullKeepCnt") actually mean "Filled" backups in 4.x, so they control how many of those filled backups to keep, including the exponential expiry option. Note that, by default, $Conf{FillCycle} is 0, which keeps fulls filled, and incrementals not filled (except for the most recent backup which is always filled), so the delete settings work as you would expect. I grappled with whether I should rename "FullKeepCnt" etc to "FillKeepCnt", but I thought that would be more confusing for existing users (and for old 3.x backups, FillKeepCnt would really still mean FullKeepCnt -- ugh!). Plus it's risky to change all the per-PC client configs to rename the variables. Suggestions are welcome. Craig |
From: Craig B. <cba...@us...> - 2013-07-14 00:57:05
|
I should also mentioned you can just replace --checksum in $Conf{RsyncFullArgsExtra} with --ignore-times if you want the "full" semantics to closely match 3.x. In this case, 4.x will have more server load than 3.x since it has to read all the file contents to get the block checksums, while in 3.x they are cached at the end of the compressed file. On the other hand, 4.x is running native C code, and 3.x is mostly in perl, so it's hard to predict that actual difference. I don't plan to implement block checksum caching in 4.x since I think --checksum is a better approach. Also, rsync's 30 generations of protocols (all backward compatible!) has 3 flavors of block checksums, so supporting all 3 for older versions is extra work, tricky, and takes a lot more testing. (3.x had to support two of the three.) Craig |
From: Kenneth P. <sh...@se...> - 2013-09-14 00:07:49
|
--On Monday, July 01, 2013 12:53 PM -0700 Craig Barratt <cba...@us...> wrote: > I'm attaching some short notes on the build/install steps for each > package. Has anyone created a spec file to build an RPM? I'm figuring I can adapt the one from CentOS but don't want to re-invent the wheel if someone else has already done it. |