Re: [BackupPC-users] Split share into several shares

[dan <dan...@gm...>]

WLAN is going to slow you down because of latency.  rsync does quite a bit
of 2 way communication.  in fact it does it for every single file it
inspects as it sends the checksum back and forth.  wireless is a killer
here, it often has latencies that are longer than a remote DSL site!

The other thing is that you should measure your backup performance by
filecount with backuppc as filecount has as much or more effect of backup
time as file size, especially when dealing with small files.  the double I/O
expense(I/O on client AND server) and bandwidth use of the rsync checksum,
as well as the time of the file list generation and transfer can be much
much higher that the cost of simply transfering a small file.

you can do a bit to spead these things up with some added rsync flags.  for
instance, if you set your --block-size for the checksum to something less
than but close to your MTU, that will help make sure you dont send checksums
inefficiently. for instance, if MTU is 1500(thats bytes) and you are sending
a file that is 7500 bytes, it is much much better to have block size=1500 so
there are only 3 chunks sent rather than have the blocksize=1000, which
would require 8 chunks, or blocksize=2000, which would require 6 chunks.

if you use -u you will skip any files that are the same size and have the
same timestamp(or the backuppc file is newer, but i wouldnt expect that).
that can be an issue if you are backing up a system that is not diligent
with timestamps.  this may help a little as it wont even checksum those
files, saving I/O.

you could also do the option -W if your network connection is faster than
your I/O, as it will skip the incrementing algorythm and just transfer the
whole file.  This will save a ton of I/O but will use up proportionaly more
bandwidth.  I have this running on a few hosts on one of my bpc servers
because the clients are on gigabit, so i have no network bandwidth
concerns.  a windows XP client with 15GB of data on takes about 5 minutes to
transfer the data, and a minute or so for the file list which is about 3
times faster than using the incremeting algorythm but is immensly bandwidth
intensive.  you still have to deal with the checksum locally for the
hardlinking algorythm.

Alternatively, backup using smb on the local network.  no bandwidth savings
at all and you also loose access to some files and file permissions.

On Sat, Jun 7, 2008 at 1:24 AM, Hendrik Friedel <hen...@we...> wrote:

>  Hello,
>
> interesting. Thanks for your reply.
>
> My backups take longer. Much longer. My last incremental took nearly 280
> minutes for about 600 MB. This is over a 54 Mbit Wlan connection. But even
> over this connection 600MB are transferable in a fraction of 280 minutes...
> I really don't know what's the problem.
> Have you checked, how much time you save by splitting the backup?
>
> Greetings,
> Hendrik
>
>  ------------------------------
> *Von:* dan...@gm... [mailto:dan...@gm...]
> *Gesendet:* Samstag, 7. Juni 2008 03:55
> *An:* Hendrik Friedel
> *Cc:* BackupPC User List
> *Betreff:* Re: [BackupPC-users] Split share into several shares
>
> I actually do this on my tru64 unix and my sco unix machines.  This works
> quite well.  It also uses up way less RAM with rsync, which is good because
> some of the tru64 machines don't have a lot of extra RAM.
>
> I have one specific tru64 alpha 4cpu machine that I have 4 hosts setup for
> and I trigger all 4 hosts for backup at the same time with cron as they are
> 1Ghz cpus and I can peg all 4 and complete backupps much quicker as my scsi
> disk array is much much faster that the cpus can handle.  I have one backup
> the / with exclusions for each directory I will backup with one of the other
> backups, then the other machines are backing up all the other directories.
> I just did some guess work on how to split it up efficiently by comparing
> file count and then filesize.  This works very well and I can get an
> incremental backup done in about 20 minutes and a full in under an hour.
> its about 12GB worth of data, but an alpha es45 is not exactly the fastest
> machine around these days.  I also dont have many files that change, maybe
> 100, but a few of them are 1GB+ each day.
>
> On Fri, Jun 6, 2008 at 3:14 PM, Hendrik Friedel <hen...@we...> wrote:
>
>> Hi,
>>
>> My backups often don't complete due to time-outs (or whatever).
>> Would it make sense, to split the rsyncd share into several shares (for
>> example: /pictures /data /[rest]), in order to make it more likely, that a
>> backup finishes, and that I don't have to start from scratch if the whole
>> backup was not complete?
>>
>> How would I do this?
>>
>> Greetings,
>> Hendrik
>>
>>
>> -------------------------------------------------------------------------
>> Check out the new SourceForge.net Marketplace.
>> It's the best place to buy or sell services for
>> just about anything Open Source.
>> http://sourceforge.net/services/buy/index.php
>> _______________________________________________
>> BackupPC-users mailing list
>> Bac...@li...
>> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki:    http://backuppc.wiki.sourceforge.net
>> Project: http://backuppc.sourceforge.net/
>>
>
>