[BackupPC-users] How I back up 500GB of BackupPC Pool

[Christian V. <chr...@kn...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
as lots of other people I had serious issues transferring the pool to a
remote site. Rsync needs days to sync the hardlinks. dump/restore
doesn't work properly (or dumps too much data, don't remember). drbd
needs a proxy through small lines (has to be purhcased). Full image
backup is too much data for the line or needs manual intervention (bring
the USB drive manually to remote office).

So I created a small script which will backup my pool to an remote
office through the line. It'll split the image in 512MB chunks, compress
them and store them on the USB drive. After the backup is done, I'll use
rsync to transfer the relatively small chunks to remote office.

What doese it do so?
Reads the disk image in 512MB chunk and copies them to ramdisk /dev/shm
(you'll need enough ram though!). It'll calculate the md5sum of the
chunk and compare to the existing one. If they differ, it'll gzip the
chunk and copy it to the USB and update the md5.
If md5 are same, it'll just remove the chunk from ramdisk.

Disadvantage is the script has to read the entire device every time is
runs. Another disadvantge might be the sizing for the chunks. If they
are to large a single bit change in a block will force the transfer of
the whole chunk. Too small chunks might reduce performance (not tested!).
It scales nearly linear with thesize of disk. If no data has changed
it'll need double time on double size.

It's not perfect at the moment, but it works. My initial backup now runs
 for 3 hrs and has transferred 100G (compressed to 94G). Mostly waiting
for gzip to finish- so if you prefer, just skip compression.

Any comments and suggestions are welcome.


#!/bin/bash
date
SRC=/dev/sdb1
DST=/srv/backuppc
TMP_FILE=/dev/shm/tmp
DST_FILE=$DST/backuppc.gz.iso
service backuppc stop
mount /srv || exit
if [ ! -e /srv/backuppc ] ; then mkdir /srv/backuppc; fi
rm -fr /dev/shm/*
sync
rm -f $DST/backuppc.log

i=0
loop=`losetup -f`

bs=$[64*1024] # blocksize 64k
fs=$[1024*1024*512] # filesize 512M
count=$[$fs/$bs] # block count per file (16384)
devlen=`df | grep $SRC| awk '{print $2}'` # #of 1k blocks
umount /var/lib/BackupPC
devlen=$[$devlen*1024] # #of bytes
devbsblocks=$[$devlen/$bs] # #of 64k blocks
# In case device does not perfectly fit into 64 blocks
if [ $[$devbsblocks*$bs] -ne $devlen ] ; then
devbsblocks=$[$devbsblocks+1] ; fi

files=$[$devbsblocks/$count] # #of blocks per dest file
if [ $[$files*$count] -ne $devbsblocks ] ; then files=$[$files+1] ; fi

while [ $i -le $files ] ; do
        losetup -o $[$i*$count*$bs] $loop $SRC
        free=$[`df /dev/shm | grep shm|awk '{print $4}'`*1024]
        while [ $free -lt $fs ] ; do
                sleep 30s
                free=$[`df /dev/shm | grep shm|awk '{print $4}'`*1024]
        done
        #dd if=$loop bs=$bs count=$count | tee $TMP_FILE.$i | md5sum >
$TMP_FILE.$i.md5
        dd if=$loop bs=$bs count=$count of=$TMP_FILE.$i
        md5sum $TMP_FILE.$i > $TMP_FILE.$i.md5
        losetup -d $loop
        if [ -e $DST_FILE.$i.md5 ] ; then
                echo "diff $DST_FILE.$i.md5 $TMP_FILE.$i.md5"
                diff $DST_FILE.$i.md5 $TMP_FILE.$i.md5
                if [ $? -ne 0 ] ; then
                        echo "$i is different" >> $DST/backuppc.log
                        ( cat $TMP_FILE.$i | gzip -2 >  $DST_FILE.$i ;
rm -f $TMP_FILE.$i ) &
                        mv $TMP_FILE.$i.md5 $DST_FILE.$i.md5
                else
                        rm -f $TMP_FILE.$i $TMP_FILE.$i.md5
                fi
        else
                ( cat $TMP_FILE.$i | gzip -2 >  $DST_FILE.$i ; rm -f
$TMP_FILE.$i ) &
                mv $TMP_FILE.$i.md5 $DST_FILE.$i.md5
        fi

        i=$[$i+1]
done
umount /srv
service backuppc start
date


Greetings Christian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with CentOS - http://enigmail.mozdev.org/

iD8DBQFLD8Ih0XNIYlAXmzsRAs9MAJsFi2H/pxWQOBQohPcXRXtXYUqjagCfUM3c
VHc9dbKXsVZP+d1s9KPn6xI=
=E6yw
-----END PGP SIGNATURE-----