Re: [BackupPC-users] Copying in a file instead of backing up?

[Jeffrey J. K. <bac...@ko...>]

Adam Goryachev wrote at about 14:20:40 +1100 on Wednesday, January 14, 2009:
 > -----BEGIN PGP SIGNED MESSAGE-----
 > Hash: SHA1
 > 
 > Jeffrey J. Kosowsky wrote:
 > > Christian Völker wrote at about 22:13:24 +0100 on Tuesday, January 13,
 > 2009:
 > >  > But now there is a 4GB file on the remote box- and the broadband
 > >  > connection is dropped every 24h hours (f!ck Telekom), so any running
 > >  > backup process will fail.
 > >  >
 > >  > Whatever, because I have this file locally as well (same file, md5sum
 > >  > and times are the same!) can I "insert" tthis file somehow into my
 > >  > backupPC under /var/lib so it recognizes it as "already there"and does
 > >  > not try to backup all the time?
 > >  >
 > >  > Thanks for further ideas.
 > >  >
 > > The following "kludge" should work.
 > >
 > > 1. Create a new temporary backuppc host that points to your local
 > >    machine. Set up the corresponding 'share' so it only backs up the
 > >    4GB file you have locally. This will give you a valid (c)pool
 > >    backup of that file.
 > >
 > > 2. Then run backuppc on your remote machine and it will create a
 > >    hard-link to the now already existing pool file.
 > >
 > > 3. Delete the temporary host config and the corresponding tree under
 > >    TopDir.
 > >
 > > Alternatively, you could cobble together a routine for directly
 > > injecting a file into the pool or cpool (by calculating the md5sum
 > > name and compressing as appropriate). Be sure to either add an extra
 > > temporary link to it or keep BackupPC_nightly from running until you
 > > do a real backup of the remote machine or otherwise the pool entry
 > > will be dropped.
 > >  
 > 
 > Even if the file exists in the (c)pool, it will still be transferred
 > by rsync/backuppc during the backup, unless it existed in the previous
 > backup of the same share/host.

OK. I can see now why this is true. But it seems like one could
rewrite the backuppc rsync protocol to check the pool for a file with
same checksum  before syncing. This could give some real speedup on
long files. This would be possible at least for the cpool where the
rsync checksums (and full file checksums) are stored at the end of
each file.

 > 
 > One thing I found when backing up (or copying) very large files with
 > rsync is that copying the first portion of the file, then stopping the
 > transfer, preserving the portion transferred (--partial), and then
 > re-starting the transfer can have a huge impact on the amount of data
 > rsync will transfer.
 > 
 > This can easily seen like this:
 > 1 Create a file of 1000M with the letter 'a' on a remote machine
 > 2 run rsync --partial --progress to sync the remote file to the local file
 > 3 After noting the throughput/etc (wait for a few MB), hit CTRL-C, and
 > then restart the transfer.
 > 4 Notice the transfer is much quicker now
 > 
 > I'd suggest rename the remote 4G file, copy the first 50M (dd
 > if=filerename of=fileorig bs=1k count=10k), exclude the renamed copy
 > of the full file, run a full backup, then rename the full file back,
 > re-run another full backup.
 > 
 > If it doesn't help, then you may just need to incrementally increase
 > the size of the file so that each chunk can be added to the end of the
 > file in a single run....
 > 
 > Also check you have enabled ssh compression (-C) to improve bandwidth
 > usage...
 > 
 > Hope that helps,
 > 
 > Regards,
 > Adam
 > -----BEGIN PGP SIGNATURE-----
 > Version: GnuPG v1.4.6 (GNU/Linux)
 > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
 > 
 > iD8DBQFJbVoIGyoxogrTyiURAiacAKCC4WI0JWHb5rV/qRHrerW1m6o10QCfRKYi
 > GvvxfwVIl5M7VVXomSJBjBo=
 > =KDyx
 > -----END PGP SIGNATURE-----
 > 
 > 
 > ------------------------------------------------------------------------------
 > This SF.net email is sponsored by:
 > SourcForge Community
 > SourceForge wants to tell your story.
 > http://p.sf.net/sfu/sf-spreadtheword
 > _______________________________________________
 > BackupPC-users mailing list
 > Bac...@li...
 > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:    http://backuppc.wiki.sourceforge.net
 > Project: http://backuppc.sourceforge.net/