Re: [bu-users] Possible to backup to another machine?

[Vincent S. <bu...@hi...>]

On Fri, Jan 12, 2007 at 09:54:51PM +0100, al...@ge... wrote:
> Hello all.
> 
> Right now I don't have a disk that I can mount on my machine.  I have
> access through ssh/scp to another machine with large backup disks, but
> the computer manager does not want that they be mounted on my
> computer.  Is it possible to use bu to backup to that machine?  I'm
> afraid it's not, but in this case would it be possible to extend bu so
> it could do it?  If so, would it be very difficult or extremely
> difficult :-)
> 
> Regards,
> Alain

Off the top of my head, I suspect it would be possible by streaming the
files to something like rsync rather than pax.  It would take some
research and experimentation.

I am sure it could be done using a separate call to [rs]cp for each file
but I moved away from doing that with bu because incremental backups are
up to 7 times faster by directly statting the dates and streaming the
names of the newer files to a single pax process.  That was with direct
file access from a mounted nfs/local FS.  With rcp, for incremental
backups, it would be even slower because it would have to do a separate
remote execution of the shell stat command to get the file date for each
file as well.  Of course the stat issue would be the same with rsync
unless I ran rsync in update mode and let it take care of choosing what
has changed.  Of course this takes some of the control away from bu.
It would take some thought and tests to determine if I could retain all
of bu's features like detailed logging, etc.  I should still be able to
use the bu Exclude patterns because rsync does have the ability to pass
exclude patterns to it.

I am not sure it is something I will likely have time to look into
extensively very soon.  I still have some other features I have been
promising for bu for some time, that I need to implement first.
However, my priorities could change because it would be a nice feature
for backing up to a remote machine over the Internet.  Something we have
begun to need from time to time ourselves.

If anybody has any ideas for other methods of implementation I am open 
to hearing them.

I am not sure how rsync does it without a special daemon on the remote
end.  If the rsh/ssh API libraries provide the ability stat, and copy
multiple files while holding the same connection open, it would be worth
looking into.  I did find there is a perl rsync module as well as a perl
ssh module that would need to be researched.  Or there may be the option
of writing something in C.

Currently, for example, the way I keep our remote web server backed up
to our local machines is by running bu on the webhost to a local backup
directory.  Then I periodically run rsync on our local machine to backup
new files in the remote webhost:/backup directory to the local machine
using a script with a command something like this:

    rsync --rsh=ssh -auv root@webhost:/backup/ /backups/webhost


If your computer manager refuses to allow NFS because of security
concerns, you might want to suggest they look into AFS.

    http://en.wikipedia.org/wiki/Andrew_file_system
    http://www.openafs.org/

Of course, it would not surprise me if their concern is stability if they
are running Linux.  Linux has had NFS stability issues for years.  It
was _one_ of the main issues that forced us over to BSD a long time ago,
after not being able to complete backups for months because our Linux
file server crashed every time, with heavy NFS access.  After years,
they still have not fixed the bugs.  We are using a modern Linux based
TV server for our DVR (because we do not have BSD drivers for the
Hauppauge mpeg encoder cards).  When copying a large mpeg file to/from
a fast machine to a slower machine the Linux box still hangs.  So we
just use it as a dedicated box for streaming the data to our BSD
machines, and use them for the video file servers.

- Vincent