From: Paul G. D. <pdg...@go...> - 2010-04-29 11:58:22
|
Dear BackupPC users, I'm doing a bit of a review of my BackupPC configuration, largely because I originally set it up with Tar, and when my server decided it wanted to backup my laptop using Tar over WiFi (when I was away from my desk), it was rather painful. So now it's using Rsync and all seems well there. However, looking into the matter of rsync vs tar again lead me to realise that rsync now supports hard-links, and all-round does a better job than tar at detecting changes, and now I'm trialling rsync for the local backup my server does of itself. Now I've added checksum caching to speed things up, but I was wondering about whether RsyncP supports the --whole-file option, as bandwidth is clearly not an issue, and all that checksuming seems a waste of CPU. Furthermore, if it is possible to use the --whole-file option, how does that affect checksum caching? Would it still help? I imagine it would still save time calculating whole-file checksums, but I'm not sure. Or am I making a huge mistake chosing rsync over tar for localhost backup? So, does anyone have a clue? Paul |
From: Chris A. <chr...@mc...> - 2013-09-24 01:51:41
|
I'm using backuppc to backup > 14TB of data on a local machine. I'm using the rsyncd option since tar does not detect deleted files in incrementals. I was trying to see if the -whole-file option in the rsync options avoided the delta xfer algorithm that is not important since it is not transferring over a network. I had a look in the RsyncP code and it ignores the -whole-file option. Could this be implemented in RsyncP since the backup rsync processes are very CPU bound and this slows down backups, particularly full backups. Chris Adamson. Dr Chris Adamson Research Officer, Developmental Imaging, Murdoch Childrens Research Institute Murdoch Childrens Research Institute Royal Children's Hospital Flemington Road, Parkville, Victoria 3052, Australia www.mcri.edu.au<http://www.mcri.edu.au/> E chr...@mc... T 03 9936 6780 ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ |
From: Till H. <hof...@gm...> - 2013-09-24 17:10:09
|
I think (correct me if I'm wrong) that the rsync method doesn't ignore the --whole-file option. Is there a reason why you use rsyncd instead of rsync? On Tue, Sep 24, 2013 at 3:51 AM, Chris Adamson <chr...@mc...>wrote: > I’m using backuppc to backup > 14TB of data on a local machine. I’m > using the rsyncd option since tar does not detect deleted files in > incrementals. I was trying to see if the –whole-file option in the rsync > options avoided the delta xfer algorithm that is not important since it is > not transferring over a network. I had a look in the RsyncP code and it > ignores the –whole-file option. Could this be implemented in RsyncP since > the backup rsync processes are very CPU bound and this slows down backups, > particularly full backups.**** > > ** ** > > Chris Adamson.**** > > ** ** > > *Dr **Chris Adamson > *Research Officer, Developmental Imaging, *Murdoch Childrens Research > Institute > *Murdoch Childrens Research Institute > Royal Children’s Hospital > Flemington Road, Parkville, Victoria 3052, Australia > www.mcri.edu.au**** > > E chr...@mc... > T 03 9936 6780 **** > > ** ** > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > ______________________________________________________________________ > > > ------------------------------------------------------------------------------ > October Webinars: Code for Performance > Free Intel webinars can help you accelerate application performance. > Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most > from > the latest Intel processors and coprocessors. See abstracts and register > > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk > _______________________________________________ > BackupPC-users mailing list > Bac...@li... > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > > |
From: Chris A. <chr...@mc...> - 2013-09-24 23:57:26
|
I don't think RsyncP supports -whole-file for receiving. I initially went over to rsyncd because I thought this was the only way to remove ssh, I know different now but the problem is RsyncP, which is used in both transfer methods. From: Till Hofmann [mailto:hof...@gm...] Sent: Wednesday, 25 September 2013 3:09 AM To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] RsyncP and --whole-file I think (correct me if I'm wrong) that the rsync method doesn't ignore the --whole-file option. Is there a reason why you use rsyncd instead of rsync? On Tue, Sep 24, 2013 at 3:51 AM, Chris Adamson <chr...@mc...<mailto:chr...@mc...>> wrote: I'm using backuppc to backup > 14TB of data on a local machine. I'm using the rsyncd option since tar does not detect deleted files in incrementals. I was trying to see if the -whole-file option in the rsync options avoided the delta xfer algorithm that is not important since it is not transferring over a network. I had a look in the RsyncP code and it ignores the -whole-file option. Could this be implemented in RsyncP since the backup rsync processes are very CPU bound and this slows down backups, particularly full backups. Chris Adamson. Dr Chris Adamson Research Officer, Developmental Imaging, Murdoch Childrens Research Institute Murdoch Childrens Research Institute Royal Children's Hospital Flemington Road, Parkville, Victoria 3052, Australia www.mcri.edu.au<http://www.mcri.edu.au/> E chr...@mc...<mailto:chr...@mc...> T 03 9936 6780<tel:03%209936%206780> ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk _______________________________________________ BackupPC-users mailing list Bac...@li...<mailto:Bac...@li...> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com If you have any question, please contact MCRI IT Helpdesk for further assistance. ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ |
From: Les M. <les...@gm...> - 2013-09-24 18:03:23
|
On Mon, Sep 23, 2013 at 8:51 PM, Chris Adamson <chr...@mc...> wrote: > I’m using backuppc to backup > 14TB of data on a local machine. I’m using > the rsyncd option since tar does not detect deleted files in incrementals. I > was trying to see if the –whole-file option in the rsync options avoided the > delta xfer algorithm that is not important since it is not transferring over > a network. I had a look in the RsyncP code and it ignores the –whole-file > option. Could this be implemented in RsyncP since the backup rsync processes > are very CPU bound and this slows down backups, particularly full backups. Where are you finding that in the RsyncP code? I see it always does whole-file when sending (restores) but don't see anything about not passing it to the other side for backups (but didn't look much beyond a grep for 'whole'...). -- Les Mikesell les...@gm... |
From: Chris A. <chr...@mc...> - 2013-09-24 23:58:21
|
This is /usr/lib/perl5/File/RsyncP.pm 115 # 116 # Since the exclude arguments are no longer needed (they are 117 # passed via the socket, not the command-line args), update 118 # $rs->{rsyncOpts} 119 # 120 @{$rs->{rsyncArgs}} = @ARGV; 121 122 # 123 # Now process the rest of the arguments we care about 124 # 125 return if ( !$p->getoptions($rs->{rsyncOpts}, 126 "block-size=i", 127 "devices|D", 128 "from0|0", 129 "group|g", 130 "hard-links|H", 131 "ignore-times|I", 132 "links|l", 133 "numeric-ids", 134 "owner|o", 135 "perms|p", 136 "protocol=i", 137 "recursive|r", 138 "relative|R", 139 "timeout", 140 "verbose|v+", 141 ) ); Note that whole-file is not included in the list of options that it looks for, so it is ignored by RsyncP. This is 0.68 but it is the same deal in 0.70. -----Original Message----- From: Les Mikesell [mailto:les...@gm...] Sent: Wednesday, 25 September 2013 4:03 AM To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] RsyncP and --whole-file On Mon, Sep 23, 2013 at 8:51 PM, Chris Adamson <chr...@mc...> wrote: > I'm using backuppc to backup > 14TB of data on a local machine. I'm > using the rsyncd option since tar does not detect deleted files in > incrementals. I was trying to see if the -whole-file option in the > rsync options avoided the delta xfer algorithm that is not important > since it is not transferring over a network. I had a look in the > RsyncP code and it ignores the -whole-file option. Could this be > implemented in RsyncP since the backup rsync processes are very CPU bound and this slows down backups, particularly full backups. Where are you finding that in the RsyncP code? I see it always does whole-file when sending (restores) but don't see anything about not passing it to the other side for backups (but didn't look much beyond a grep for 'whole'...). -- Les Mikesell les...@gm... ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk _______________________________________________ BackupPC-users mailing list Bac...@li... List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com If you have any question, please contact MCRI IT Helpdesk for further assistance. ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ |
From: Les M. <les...@gm...> - 2013-09-26 04:01:00
|
On Tue, Sep 24, 2013 at 6:58 PM, Chris Adamson <chr...@mc...> wrote: > This is /usr/lib/perl5/File/RsyncP.pm > > 115 # > 116 # Since the exclude arguments are no longer needed (they are > 117 # passed via the socket, not the command-line args), update > 118 # $rs->{rsyncOpts} > 119 # > 120 @{$rs->{rsyncArgs}} = @ARGV; > 121 > 122 # > 123 # Now process the rest of the arguments we care about > 124 # > 125 return if ( !$p->getoptions($rs->{rsyncOpts}, > 126 "block-size=i", > 127 "devices|D", > 128 "from0|0", > 129 "group|g", > 130 "hard-links|H", > 131 "ignore-times|I", > 132 "links|l", > 133 "numeric-ids", > 134 "owner|o", > 135 "perms|p", > 136 "protocol=i", > 137 "recursive|r", > 138 "relative|R", > 139 "timeout", > 140 "verbose|v+", > 141 ) ); > > Note that whole-file is not included in the list of options that it looks for, so it is ignored by RsyncP. This is 0.68 but it is the same deal in 0.70. > I don't see --one-file-system in that list either, but I know it is honored because I use it everywhere. This must be the list that is processed on the receiving side. Wouldn't whole-file be passed to the sender and handled there? -- Les Mikesell les...@gm... |
From: Chris A. <chr...@mc...> - 2013-09-28 11:56:27
|
Les, The only difference I can see is that, as you point out, --one-file-system is purely handled by the sender and doesn't need to be "supported" as such by the receiver. I haven't as yet figured out how to get rsyncd to print out the final set of options it is using when backuppc establishes a connection to it so I don't know whether whole-file is being used directly. However, the full backups are CPU bound for both the rsync and BackupPC_dump processes and if I do a iftop to look at the traffic on the loopback network connection there is hardly anything being transferred over it. This suggests that only checksums are being transferred over the connection rather than the data itself. I will look into getting rsyncd to print out more verbose log files so that I can see if it is indeed using the delta xfer algorithm even though I specify --whole-file. -----Original Message----- From: Les Mikesell [mailto:les...@gm...] Sent: Thursday, 26 September 2013 2:01 PM To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] RsyncP and --whole-file On Tue, Sep 24, 2013 at 6:58 PM, Chris Adamson <chr...@mc...> wrote: > This is /usr/lib/perl5/File/RsyncP.pm > > 115 # > 116 # Since the exclude arguments are no longer needed (they are > 117 # passed via the socket, not the command-line args), update > 118 # $rs->{rsyncOpts} > 119 # > 120 @{$rs->{rsyncArgs}} = @ARGV; > 121 > 122 # > 123 # Now process the rest of the arguments we care about > 124 # > 125 return if ( !$p->getoptions($rs->{rsyncOpts}, > 126 "block-size=i", > 127 "devices|D", > 128 "from0|0", > 129 "group|g", > 130 "hard-links|H", > 131 "ignore-times|I", > 132 "links|l", > 133 "numeric-ids", > 134 "owner|o", > 135 "perms|p", > 136 "protocol=i", > 137 "recursive|r", > 138 "relative|R", > 139 "timeout", > 140 "verbose|v+", > 141 ) ); > > Note that whole-file is not included in the list of options that it looks for, so it is ignored by RsyncP. This is 0.68 but it is the same deal in 0.70. > I don't see --one-file-system in that list either, but I know it is honored because I use it everywhere. This must be the list that is processed on the receiving side. Wouldn't whole-file be passed to the sender and handled there? -- Les Mikesell les...@gm... ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk _______________________________________________ BackupPC-users mailing list Bac...@li... List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com If you have any question, please contact MCRI IT Helpdesk for further assistance. ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ |
From: Till H. <hof...@gm...> - 2013-09-28 12:31:15
|
Doesn't rsync calculate checksums even if you set --whole-file? It doesn't calculate block checksums but it still calculates file checksums - and if they match it skips the file (which only happens if you set --checksum-seed because otherwise it uses the current time as checksum-seed) On Sat, Sep 28, 2013 at 1:56 PM, Chris Adamson <chr...@mc...>wrote: > Les, > > The only difference I can see is that, as you point out, --one-file-system > is purely handled by the sender and doesn't need to be "supported" as such > by the receiver. I haven't as yet figured out how to get rsyncd to print > out the final set of options it is using when backuppc establishes a > connection to it so I don't know whether whole-file is being used directly. > However, the full backups are CPU bound for both the rsync and > BackupPC_dump processes and if I do a iftop to look at the traffic on the > loopback network connection there is hardly anything being transferred over > it. This suggests that only checksums are being transferred over the > connection rather than the data itself. I will look into getting rsyncd to > print out more verbose log files so that I can see if it is indeed using > the delta xfer algorithm even though I specify --whole-file. > > -----Original Message----- > From: Les Mikesell [mailto:les...@gm...] > Sent: Thursday, 26 September 2013 2:01 PM > To: General list for user discussion, questions and support > Subject: Re: [BackupPC-users] RsyncP and --whole-file > > On Tue, Sep 24, 2013 at 6:58 PM, Chris Adamson <chr...@mc...> > wrote: > > This is /usr/lib/perl5/File/RsyncP.pm > > > > 115 # > > 116 # Since the exclude arguments are no longer needed (they are > > 117 # passed via the socket, not the command-line args), update > > 118 # $rs->{rsyncOpts} > > 119 # > > 120 @{$rs->{rsyncArgs}} = @ARGV; > > 121 > > 122 # > > 123 # Now process the rest of the arguments we care about > > 124 # > > 125 return if ( !$p->getoptions($rs->{rsyncOpts}, > > 126 "block-size=i", > > 127 "devices|D", > > 128 "from0|0", > > 129 "group|g", > > 130 "hard-links|H", > > 131 "ignore-times|I", > > 132 "links|l", > > 133 "numeric-ids", > > 134 "owner|o", > > 135 "perms|p", > > 136 "protocol=i", > > 137 "recursive|r", > > 138 "relative|R", > > 139 "timeout", > > 140 "verbose|v+", > > 141 ) ); > > > > Note that whole-file is not included in the list of options that it > looks for, so it is ignored by RsyncP. This is 0.68 but it is the same deal > in 0.70. > > > > I don't see --one-file-system in that list either, but I know it is > honored because I use it everywhere. This must be the list that is > processed on the receiving side. Wouldn't whole-file be passed to the > sender and handled there? > > -- > Les Mikesell > les...@gm... > > > ------------------------------------------------------------------------------ > October Webinars: Code for Performance > Free Intel webinars can help you accelerate application performance. > Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most > from the latest Intel processors and coprocessors. See abstracts and > register > > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk > _______________________________________________ > BackupPC-users mailing list > Bac...@li... > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > > If you have any question, please contact MCRI IT Helpdesk for further > assistance. > ______________________________________________________________________ > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > ______________________________________________________________________ > > > ------------------------------------------------------------------------------ > October Webinars: Code for Performance > Free Intel webinars can help you accelerate application performance. > Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most > from > the latest Intel processors and coprocessors. See abstracts and register > > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk > _______________________________________________ > BackupPC-users mailing list > Bac...@li... > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > |
From: Les M. <les...@gm...> - 2013-09-28 15:26:17
|
On Sat, Sep 28, 2013 at 7:30 AM, Till Hofmann <hof...@gm...> wrote: > Doesn't rsync calculate checksums even if you set --whole-file? It doesn't > calculate block checksums but it still calculates file checksums - and if > they match it skips the file (which only happens if you set --checksum-seed > because otherwise it uses the current time as checksum-seed) > Yes, --whole-file still shouldn't send matching files. If you ran over ssh you would be able to use ps to see what options were passed on the remote command line. If you are poking around in the code, maybe you could find a way to turn off the automatically added --ignore-times that is added to full runs. That would make it go as fast as incrementals, at the expense of a little safety. -- Les Mikesell les...@gm... |
From: Chris A. <chr...@mc...> - 2013-09-30 11:33:17
Attachments:
backuppc-ignore-times.patch
|
The idea of using --whole-file is that rsync behaves like tar and I thought that I could set --whole-file so that the rsync method would behave like the tar method and compare byte by byte rather than using checksums. I had a look both in the rsyncp and BackupPC code (RsyncFileIO.pm). The use of checksums is intrinsic in the way that the rsync transfer method works. So it uses the full file MD4 checksums to determine whether a file has changed at all. Whether it uses the whole file option for transferring changed files is unimportant since 99% of our files don't generally change between fulls. After poking around it seems that the --whole-file option is specific to the actual rsync tool and the BackupPC code along with RsyncP send a customized set of commands to the rsync server. So it appears that not all of the rsync "program" options are applicable. Maybe we already knew this but yeah this concludes my investigation of --whole-file. I have attached a patch file that doesn't add the --ignore-times argument to a full backup. It is based on backuppc 3.2.1. -----Original Message----- From: Les Mikesell [mailto:les...@gm...] Sent: Sunday, 29 September 2013 1:26 AM To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] RsyncP and --whole-file On Sat, Sep 28, 2013 at 7:30 AM, Till Hofmann <hof...@gm...> wrote: > Doesn't rsync calculate checksums even if you set --whole-file? It > doesn't calculate block checksums but it still calculates file > checksums - and if they match it skips the file (which only happens if > you set --checksum-seed because otherwise it uses the current time as > checksum-seed) > Yes, --whole-file still shouldn't send matching files. If you ran over ssh you would be able to use ps to see what options were passed on the remote command line. If you are poking around in the code, maybe you could find a way to turn off the automatically added --ignore-times that is added to full runs. That would make it go as fast as incrementals, at the expense of a little safety. -- Les Mikesell les...@gm... ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk _______________________________________________ BackupPC-users mailing list Bac...@li... List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com If you have any question, please contact MCRI IT Helpdesk for further assistance. ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ |
From: Les M. <les...@gm...> - 2013-09-30 16:48:36
|
On Mon, Sep 30, 2013 at 6:33 AM, Chris Adamson <chr...@mc...> wrote: > The idea of using --whole-file is that rsync behaves like tar and I thought that I could set --whole-file so that the rsync method would behave like the tar method and compare byte by byte rather than using checksums. I had a look both in the rsyncp and BackupPC code (RsyncFileIO.pm). The use of checksums is intrinsic in the way that the rsync transfer method works. So it uses the full file MD4 checksums to determine whether a file has changed at all. Whether it uses the whole file option for transferring changed files is unimportant since 99% of our files don't generally change between fulls. After poking around it seems that the --whole-file option is specific to the actual rsync tool and the BackupPC code along with RsyncP send a customized set of commands to the rsync server. So it appears that not all of the rsync "program" options are applicable. Maybe we already knew this but yeah this concludes my investigation of --whole-file. I have attached a patch file that doesn't add the --ignore-times argument to a full backup. It is based on backuppc 3.2.1. > I think --whole-file is passed to, and handled by, the remote sender, but it is only going to make a difference if you have large files with small changes that take more time to merge than to send the whole thing. With your --ignore-times patch you will lose the 'safety factor' of checking the content (and underlying media) periodically, which may be more important in a de-duping environment where only one backup copy of the content will exist. You might want to back it out for occasional weekend runs or something like that. -- Les Mikesell les...@gm... |
From: Tyler J. W. <ty...@to...> - 2010-04-29 12:45:03
|
On Thursday 29 April 2010 13:58:12 Paul Gideon Dann wrote: > Or am I making a huge mistake chosing rsync over tar for localhost backup? I use local rsync for backing up the server itself (indeed, for about 70 servers), and it has always been fine for me. I don't think the checksumming really makes much difference, as once it is copied (to itself) it will still have to be checksummed to go into the pool. Is the checksum caching rsync does the same as the checksum that is stored in the pool? Regards, Tyler -- "Violence, naked force, has settled more issues in history than has any other factor, and the contrary opinion is wishful thinking at its worst." -- Robert A. Heinlein, Starship Troopers, 1959 |
From: Paul G. D. <pdg...@go...> - 2010-04-29 13:12:41
|
On Thursday 29 April 2010 13:44:52 Tyler J. Wagner wrote: > I use local rsync for backing up the server itself (indeed, for about 70 > servers), and it has always been fine for me. I don't think the > checksumming really makes much difference, as once it is copied (to > itself) it will still have to be checksummed to go into the pool. Yeah, but the rsync protocol's chunk checksumming is designed to reduce network traffic, and since in the case of a localhost backup the bandwidth is however many orders of magnitude higher than over a network, it's just going to waste CPU cycles, isn't it? I know that when used to sync two local directories, rsync proper will enable --whole-file by default for this reason. However, because RsyncP is involved, I doubt rsync will be aware that it's a local transfer, so the option would have to be specified explicitly. > Is the checksum caching rsync does the same as the checksum that is stored > in the pool? No, it's a different hash. My understanding is that under defaut RsyncP conditions, BackupPC will do two pooled file decompressions, plus a checksum for each chunk of the file, plus a whole-file checksum, for each file. Checksum caching prevents all of this CPU overhead, but does not affect BackupPC's pool- related checksumming. BackupPC will still perform file content hashing and filename mangling regardless. Here are the docs on checksum caching for reference: http://backuppc.sourceforge.net/faq/BackupPC.html#rsync_checksum_caching Paul |