Re: [BackupPC-users] Update Re: How to speed up backup process?

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Thanks for the suggestion. Regarding the network transfer, why do I 
still need to transfer all the files? The new full backup is following a 
previous full backup. So all the files are already in "pool" and do not 
need to transfer again. It should only transfer the new files, which I 
do not have any. 8 hours is too long to just generate file list and 
transfer almost no file over network. So, what BackupPC is doing?

I understand that 6MB/s network transfer may be close to "as good as I 
can get". If I do compress with rsync (that is, if I can be convinced 
that I need to transfer all 170GB every time I do full backup), when 
will the original pool files be deleted (so to free up space)? If later 
decide not to do the compression, when will the cpool files will be deleted?

I do plan to put an internal 500GB hard drive in the server. This way, I 
can have a backup on the server and a second backup on external usb 
drive which can be put away offline. It will also give me some test if 
usb is the main reason to slow things down.

Gigabit upgrade seems quite expensive. Maybe later...

Jinshi

dan wrote:
> I'm glad your speed has improved.  Now that your CPU is not the 
> bottleneck, you have 2 remaining bottlenecks which are the USB hard 
> drive and the 100Mb/s network.  Immediately I suspect that the network 
> is the bottleneck because of the 6MB/s which is pretty typical of 
> 100Mb/s networking.  An inexpensive network card or switch will compound 
> the issue but really, you can only expect 6-8MB/s over TCP/IP because of 
> protocol overhead and network collisions.
> 
> At this point, I suggest you fix the networking issue.  Now, that 
> doesn't necessarily mean going gigabit though that would help.  I 
> suggest you compress with rsync.  you will likely be able to improve how 
> much data is pushed over the link though your network interface will 
> still report 6MB/s.  You should see 20-50% low backup times, depending 
> on the compressibility of the data.
> 
> I made the jump to gigabit on my LAN at work primarily because of 
> backups.  I run all cisco equipment so I dropped some fat cash on 
> gigabit but it was worth it.
> 
> good luck
> 
> On Jan 7, 2008 10:51 AM, Jinshi < ji...@gm... 
> <mailto:ji...@gm...>> wrote:
> 
>     Thank you all again for your responses. I setup the backup server on a
>     better computer now: Pentium 4 2.4GHz with 760M ram. The system hard
>     drive interface is also faster (ATA133, instead of ATA66 on old
>     computer). The backup files are still on the same USB drive. The data
>     files are on a new computer with Q6600 2.4GHz + 4GB ram running Vista
>     x64. Total backup files are about 170GB (over 500k files).All hash files
>     are still on the backup server. So there should be no major file
>     transfer over network (is this correct?).
> 
>     I did a full backup via rsyncd. It takes about 1 hour for the client
>     computer to prepare the file list before the transfer started. Total
>     backup time is a little less than 8 hours and average transfer speed
>     more than 6MB/sec. Big boost than before. I think the CPU is making the
>     big difference and rsync is doing a lot of calculation. Can someone
>     explain in detail what rsync is doing? I do not use any compression.
> 
>     8 hours is long but tolerable for once a week. I suppose incremental
>     backup will be faster. I don't think I will do any upgrade on the hard
>     drive/usb/network since I believe the CPU/memory is the limiting step.
>     Any thought on fine tune /configure the BackupPC to make it faster?
> 
>     Thanks.
> 
>     PS: the USB drive still shows as 40.000MB/s in dmesg. Is this a FreeBSD
>     thing? I suppose it should be 60MB for usb2.0 (480Mbps). This may not be
>     important here since the file transfer speed is far below 40MB/s.
> 
> 
>     Les Mikesell wrote:
>      > Jinshi wrote:
>      >> Thank you all for your reply. Apparently everyone consider this
>     is too
>      >> slow since the difference between usb2.0 and usb1.1 is quite
>     big. But I
>      >> do tested all other usb ports. The dmesg says the transfer speed is
>      >> either 40.000MB or 1.000MB/s. So, sorry everyone, I still insist
>     I have
>      >> usb2.0 here :)
>      >>
>      >> I didn't reply sooner because I want to wait and see how long
>     the backup
>      >> is. Now, total file about 170GB (I moved some files from other
>     computer.
>      >> Those files were backed up before, so the hash files are
>     available on
>      >> the backup computer). There is almost no new files (details
>     below). And
>      >> full backup by SMB transfer takes 32 hours (only 7.3MB new
>     files), by
>      >> rsyncd transfer takes 26 hours (1.8GB new files since my wife
>     uploaded
>      >> more photos during SMB backup time).
>      >
>      > Was the rsync run a full or incremental?  A mostly unchanged
>     incremental
>      > should be much faster than a full because the full does a block
>     checksum
>      > comparison even on files where the timestamp and length match.
>      >
>      >> So, rsyncd helps, but not a lot, just like Serge predicted. Average
>      >> transfer speed is 1.5 and 1.8 MB/sec, respectively.
>      >>
>      >> Now, back to my original question: where is the bottleneck? I am
>     going
>      >> to upgrade the computer to P4. If that still doesn't help much,
>     I don't
>      >> know what to do next?!
>      >
>      > Rsync transfers the entire directory listing before starting so
>     there is
>      > a certain amount of RAM needed per file - if you have less the
>     server
>      > might start swapping and slow down.  Depending on your directory
>     layout,
>      > it might be possible to break the target up into several smaller
>     chunks
>      > to help with this.  Also, a slightly more extreme approach would
>     be to
>      > make the subdirectory runs appear to be different hosts using the
>     alias
>      > feature.  That way you could stagger the full and incremental runs so
>      > you don't have to complete a full in one day.
>      >
> 
> 
>     -------------------------------------------------------------------------
>     Check out the new SourceForge.net Marketplace.
>     It's the best place to buy or sell services for
>     just about anything Open Source.
>     http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
>     _______________________________________________
>     BackupPC-users mailing list
>     Bac...@li...
>     <mailto:Bac...@li...>
>     List:     https://lists.sourceforge.net/lists/listinfo/backuppc-users
>     Wiki:    http://backuppc.wiki.sourceforge.net
>     Project: http://backuppc.sourceforge.net/
> 
> 
> 
> ------------------------------------------------------------------------
> 
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> BackupPC-users mailing list
> Bac...@li...
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:    http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/