Re: [BackupPC-users] How scalable is backuppc?

[Chris R. <cro...@gc...>]

Mike Dresser wrote:
> Matthias Meyer wrote:
>   
>> Dear all,
>>
>> How scalable is backuppc?
>> Where are the limits or what can produce performance bottlenecks?
>>
>> I've heard about hardlinks which can be a problem if theire are millions of
>> it. Is that true?
>>   
>>     
> The file system can become... interesting to fix or backup when you get 
> a few million hard links, especially if you're using XFS.

Amen to that...

[root@archive-1 trash]# df -i /dev/sdb1
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb1            6442438528 55813203 6386625325    1% /data
[root@archive-1 trash]# mount |grep /dev/sdb1
/dev/sdb1 on /data type xfs 
(rw,noatime,nodiratime,nobarrier,logbufs=8,sunit=512,swidth=6656,logbsize=262144)
[root@archive-1 trash]# cat /etc/redhat-release
CentOS release 5.2 (Final)
[root@archive-1 trash]# uname -srvmp
Linux 2.6.18-92.1.18.el5 #1 SMP Wed Nov 12 09:19:49 EST 2008 x86_64 x86_64

>   There _appears_ to be some bugs in Debian etch's xfs tools, last time I had to 
> run an xfs_repair -n on etch it took 6 days.. after upgrading to lenny 
> it takes about 10 minutes or so.  I'm guessing there were some major 
> improvements in the xfs tools from etch to lenny with regards to memory 
> usage.
>   
>> Anyone would share his experience about CPU usage in relation to bandwidth
>> usage? Particularly by using rsync?
>>   
>>     
> I see about 2-15m/s on our rsync backups   CPU usage is pretty low, it's 
> mostly disk i/o that holds it up.  The newer faster machines back up 
> faster, so there's a bottleneck on the clients as well.
>   

Same story here.  CPU is, to a close approximation, never a problem.  
For an added twist, most of my clients are on the far end of a satellite 
link.  After the initial backups, even running 16 backups in parallel, I 
hardly peak above 4Mbit/sec (per iftop).

>> I have a 2x1GHz Server with 2GB RAM and 4x500GB SATA Disks in Software
>> Raid5.
>>   
>>     
> server here is a 2x1.8ghz opteron 265, 5GB ram, 8x1TB with 3ware raid5.  
> Backup window is set from 18:00 to 23:00, it generally finishes all the 
> backups within that 5 hours.  Full's every 8 days, incr every day.
>
>     * Pool is 1852.64GB comprising 8466234 files and 4369 directories
>       (as of 3/12 02:26),
>     * Pool hashing gives 9654 repeated files with longest chain 85,
>     * Nightly cleanup removed 7189 files of size 35.62GB (around 3/12
>       02:26),
>     * Pool file system was recently at 44% (3/12 12:05), today's max is
>       44% (3/12 00:00) and yesterday's max was 44%.
>
> There are 42 hosts that have been backed up, for a total of:
>
>     * 789 full backups of total size 10819.30GB (prior to pooling and
>       compression),
>     * 284 incr backups of total size 913.00GB (prior to pooling and
>       compression).

For comparison, I have a Xeon X3320, 8GB RAM and 16 Seagate ES.2 1 TB 
drives (ST31000340NS) on a Adaptec 51645 using a RAID6 setup 
(effectively 13 data drives, 2 parity, 1 hot spare).  Fulls every 7 
days, incremental every day.  I Started off keeping 4 weeks of fulls and 
two weeks of incrementals, and cleaning the whole pool nightly.  All 
(~125) backups completed within the 20:00-07:00 time frame and 
BackupPC_nightly finished in under 5 hours (with a concurrence of 8).  
Then I doubled retention on both fulls and incrementals and have been 
fighting steadily worsening performance problems since.  Dropping the 
retention back to initial levels is taking a while (BackupPC_trashClean 
seems to finally be gaining some ground).

    * Pool is 587.09GB comprising 31187909 files and 4369 directories
      (as of 2009-03-11 14:46),
    * Pool hashing gives 2294 repeated files with longest chain 944,
    * Nightly cleanup removed 1648979 files of size 50.98GB (around
      2009-03-11 14:46),
    * Pool file system was recently at 11% (2009-03-12 11:00), today's
      max is 11% (2009-03-12 07:00) and yesterday's max was 11%.

There are 125 hosts that have been backed up, for a total of:

    * 495 full backups of total size 2384.27GB (prior to pooling and
      compression),
    * 855 incr backups of total size 380.11GB (prior to pooling and
      compression).

Chris