From: dan <dan...@gm...> - 2008-03-03 19:44:48
|
amen Les no need to have just 1 backup server! with 8GB of ram, I would give a rough estimate that you can have up to 500,000,000 files in flight at one time as far as ram is concerned! that includes ALL hosts that would be backed up simultaniously. I doubt RAM will be an issue for you. Probably hard disk speed will be the big hurdle here. You have some pretty limited options here because you will spend BIG BIG money getting a very fast RAID5/6 hardware array while regular desktop drives will perform pretty poorly with a software RAID5/6 because having the CPU do the checksums adds a lot of latency with = poor write speed for large numbers of small files aka hard links and directories aka 90% of what backuppc does. just take Les' advice, split up the backup job among a few servers instead of one BIG one. On Mon, Mar 3, 2008 at 10:08 AM, Les Mikesell <les...@gm...> wrote: > Christopher Derr wrote: > > > > I'm a new backuppc user for a college academic department. I have a > > moderately sized disk array (3 TB, RAID 5, Areca RAID) backing up the > > data on various servers. > > I think that's the first time I've heard someone call 3 TB "moderately > sized", but I guess times change... > > > The backup server has 8 GB of memory and is > > currently running a backup of 350 GB user directories on a Windows 2003 > > server with rsyncd. Here's what TOP shows on the backup server: > > > > top - 07:59:11 up 17 days, 42 min, 2 users, load average: 1.06, 0.76, > 0.35 > > Tasks: 95 total, 2 running, 93 sleeping, 0 stopped, 0 zombie > > Cpu(s): 12.9%us, 2.8%sy, 0.0%ni, 77.9%id, 6.3%wa, 0.0%hi, 0.1%si, > > 0.0%st > > Mem: 8189252k total, 7891284k used, 297968k free, 21104k buffers > > Swap: 19800072k total, 88k used, 19799984k free, 6361760k cached > > > That doesn't look bad, but were you running the max number of backup > processes at the time? If you have the sysstat package installed you > can use 'sar' to see snapshots of the system state at 10 minute intervals. > > > And here's the main process: > > > > 22277 backuppc 18 0 211m 159m 2340 D 24 2.0 1:45.67BackupPC_dump > > The server process itself doesn't do all the work. > > > And the host summary for this host: > > > > Host: UserServer > > #full: 2 > > Full Age: 1.2 > > Full Size: 345.75 GB > > Speed: 7.29 MB/s > > #incr: 6 > > > > I haven't fiddled with the basic configuration at all (write caches, no > > of fulls vs no of incr, etc). How do these stats look to you for what > > I'm doing? There's a Gigabit connection between the backuppc server and > > the Windows box. Last full backup of the UserServer's 345 GB took 787.9 > > minutes (13 hours). It is, of course, lots of smaller files. The time > > to back it all up isn't unacceptable at this point, though the > > UserServer will be bumped to 4 TB soon, and I expect usage to climb over > > 1 TB by the end of the year (if you build it, they will come). > > Running rsync you should see a big difference between full and > incremental runs. > > > What do the experts think about the memory usage and the speed of the > > backup? Any suggestions for improving anything? > > One issue is that rsync wants to transfer a complete directory listing > before starting, so it needs a lot of RAM to hold it. You might be able > to break the backups up into smaller subdirectory runs to help with > that, but your 8 gigs may be fine anyway. Another common scaling > problem is with the disk seek speed when dealing with all the hardlinks > between directories. At some point it may be easier to set up a 2nd > backup server than to fight this battle with more expensive hardware or > specialized tuning. > > -- > Les Mikesell > les...@gm... > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > BackupPC-users mailing list > Bac...@li... > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > |