BackupPC

Paul Fox wrote:
>  > Why are full backups needed at all with BackupPC?
>  > 
>  > According to documentation, "BackupPC's CGI interface ``fills-in'' 
>  > incremental backups based on the last full backup, giving every backup a 
>  > ``full'' appearance."
>  > 
>  > So, in theory, it should be enough to make just one full, initial 
>  > backup, and then only incremental backups.
>  > 
>  > Or do I miss something here?
> 
> one reason (there may be others) is that incrementals don't account
> for the removal of files.  if a full contains a file that is later
> removed, it will always appear in that "filled" view, even after
> the file is gone from your system.  so full backups are necessary
> to reestablish a true image of your current contents.
> 
> (this is with tar -- rsync incrementals may actually remove deleted
> files.  i don't use rsync.)

So with rsync it shouldn't be an issue, right? Could anyone comment on that?


Anyway, will several full backups use only one hardlinked file in the 
pool, or do full backups use separate, non-hardlinked files?


-- 
Tomasz Chmielewski
http://wpkg.org

> >  > Why are full backups needed at all with BackupPC?
 > >  > 
 > >  > According to documentation, "BackupPC's CGI interface ``fills-in'' 
 > >  > incremental backups based on the last full backup, giving every backup a 
 > >  > ``full'' appearance."
 > >  > 
 > >  > So, in theory, it should be enough to make just one full, initial 
 > >  > backup, and then only incremental backups.
 > >  > 
 > >  > Or do I miss something here?
 > > 
 > > one reason (there may be others) is that incrementals don't account
 > > for the removal of files.  if a full contains a file that is later
 > > removed, it will always appear in that "filled" view, even after
 > > the file is gone from your system.  so full backups are necessary
 > > to reestablish a true image of your current contents.
 > > 
 > > (this is with tar -- rsync incrementals may actually remove deleted
 > > files.  i don't use rsync.)
 > 
 > So with rsync it shouldn't be an issue, right? Could anyone comment on that?
 > 
 > 
 > Anyway, will several full backups use only one hardlinked file in the 
 > pool, or do full backups use separate, non-hardlinked files?

they use only one hardlinked file -- i.e., no extra space.

paul
=---------------------
 paul fox, pgf@... (arlington, ma, where it's 44.1 degrees)

On Fri, 2006-03-24 at 11:04, Tomasz Chmielewski wrote:

> >  > So, in theory, it should be enough to make just one full, initial 
> >  > backup, and then only incremental backups.
> >  > 
> >  > Or do I miss something here?
> > 
> > one reason (there may be others) is that incrementals don't account
> > for the removal of files.  if a full contains a file that is later
> > removed, it will always appear in that "filled" view, even after
> > the file is gone from your system.  so full backups are necessary
> > to reestablish a true image of your current contents.
> > 
> > (this is with tar -- rsync incrementals may actually remove deleted
> > files.  i don't use rsync.)
> 
> So with rsync it shouldn't be an issue, right? Could anyone comment on that?

They have different issues.  I'm not sure about that particular one.
Tar/smb backups use only the file timestamps to decide what to
take in incremental runs and will thus miss files in a new
location because a directory above was renamed and in at least
the smb case, files that are back-dated by copy mechanisms
that preserve the original timestamp (unzip, etc.).  These
omissions accumulate until you do the next full.

Rsync can always identify new or moved files compared to your
backup, but it will always tranfer the changes made since the
last full, so the size will keep increasing as you repeat
incremental runs.  Rsync also bases comparisons on the file
size/timestamp during incremental runs but does block checksum
comparisons of everything for fulls.

In all cases, identical files are collapsed into hard links
to the pool copy, so the only down side of full runs is the
extra bandwidth for the tar/smb copies and extra time for rsync. 

-- 
  Les Mikesell
   les@...

> > 
 > > You should be able to tell backuppc to make fulls as often as
 > > you want.  The only downside with rsync is the extra time
 > > it takes to do the full block checksum compare on existing
 > > files. 
 > 
 > Is it really the only downside of full backups?
 > 
 > Doesn't a full backup mean that *everything* will be transferred again?

with tar, yes.  with rsync, no -- rsync only recompares checksums.

 > Does the full rsync backup in BackupPC transfer only changes (compared 
 > to the last full backup), or maybe it transfers everything?

it ignores any hints that are used during incrementals (dates,
modes, etc), and transfers everything that isn't already
available in the pool.  the result is the same as if it transferred
everything.

paul
=---------------------
 paul fox, pgf@... (arlington, ma, where it's 34.7 degrees)

Paul Fox wrote:
>  > > 
>  > > You should be able to tell backuppc to make fulls as often as
>  > > you want.  The only downside with rsync is the extra time
>  > > it takes to do the full block checksum compare on existing
>  > > files. 
>  > 
>  > Is it really the only downside of full backups?
>  > 
>  > Doesn't a full backup mean that *everything* will be transferred again?
> 
> with tar, yes.  with rsync, no -- rsync only recompares checksums.

All right, than perhaps it's good to make only "full" backups with 
rsync/rsyncd, and no incremental backups.

What about "smb" method for Windows? Will full backup transfer 
everything (as tar would), or will it do some magic and transfer 
new/changed files only (based on timestamps etc.)?
If it transfers everything, perhaps it's better to install rsynd on a 
Windows machine?


-- 
Tomasz Chmielewski
http://wpkg.org

On Sat, Mar 25, 2006 at 02:48:31PM +0100, Tomasz Chmielewski wrote:
> What about "smb" method for Windows? Will full backup transfer 
> everything (as tar would), or will it do some magic and transfer 
> new/changed files only (based on timestamps etc.)?

it transfers everything.

> If it transfers everything, perhaps it's better to install rsynd on a 
> Windows machine?

perhaps.  depends on whether that additional complexity is worth it to
you.  If you are bandwidth-constrained, then probably better.  If it's
on a LAN, maybe not worth the hassle.

danno
--
dan pritts - systems administrator - internet2
734/352-4953 office        734/834-7224 mobile

On Sun, 26 Mar 2006 09:19:02 -0500
Dan Pritts <danno@...> wrote:
> On Sat, Mar 25, 2006 at 02:48:31PM +0100, Tomasz Chmielewski wrote:
> > What about "smb" method for Windows? Will full backup transfer 
> > everything (as tar would), or will it do some magic and transfer 
> > new/changed files only (based on timestamps etc.)?
> 
> it transfers everything.
> 
> > If it transfers everything, perhaps it's better to install rsynd on a 
> > Windows machine?
> 
> perhaps.  depends on whether that additional complexity is worth it to
> you.  If you are bandwidth-constrained, then probably better.  If it's
> on a LAN, maybe not worth the hassle.

To summarize, for a Windows host:
- rsync over ssh doesn't work.
- rsyncd transfers only diff but you need to connect with a clear password.
- tar over ssh transfers every thing.
- smb transfers every thing and you need to connect with a clear password.

Correct me if I'm wrong.

Laurent Mazet writes:

> To summarize, for a Windows host:
>
> - rsync over ssh doesn't work.

Yes, but I haven't tested it recently.

> - rsyncd transfers only diff but you need to connect with a clear passw=
ord.

Rsyncd doesn't send a clear password over the network.  It uses
a digest-based challenge/response.

> - tar over ssh transfers every thing.

Yes, for a full.

> - smb transfers every thing and you need to connect with a clear passwo=
rd.

I'm not sure whether smb sends clear passwords over the network.

Craig

On Sun, 26 Mar 2006 15:09:06 -0800
Craig Barratt <cbarratt@...> wrote:
> Laurent Mazet writes:
> 
> > To summarize, for a Windows host:
> >
> > - rsync over ssh doesn't work.
> 
> Yes, but I haven't tested it recently.

Unfortunately, I've tried last week with copSSH 1.3.9  and  cwRsync 2.0.6 from http://www.itefix.no without any success

> > - rsyncd transfers only diff but you need to connect with a clear password.
> 
> Rsyncd doesn't send a clear password over the network.  It uses
> a digest-based challenge/response.

At least you need you need to store the password in a plain text
 
> > - tar over ssh transfers every thing.
> 
> Yes, for a full.
> 
> > - smb transfers every thing and you need to connect with a clear password.
>
> I'm not sure whether smb sends clear passwords over the network.

I thing your right. It might depend on a boolean (encrypt passwords) in samba config. But you need to store it in an a plain text and it also gives a full login access. For security reason, I thing that the worst method.

    Laurent

On Mon, Mar 27, 2006 at 12:52:31AM +0200, Laurent Mazet wrote:
> To summarize, for a Windows host:
> - rsync over ssh doesn't work.
> - rsyncd transfers only diff but you need to connect with a clear password.

You might be able to run rsyncd transfers over an ssh tunnel.  I 
don't know if there are issues with that on cygwin or not.

Does anyone know if rsync and ssh will run under uwin?  or if 
uwin still even really works?

danno
--
dan pritts - systems administrator - internet2
734/352-4953 office        734/834-7224 mobile

Rsyncd over ssh work on windows...rsync will still not act like a server 
under cygwin.
Trey

----- Original Message ----- 
From: "Dan Pritts" <danno@...>
To: "Laurent Mazet" <mazet@...>
Cc: <backuppc-users@...>
Sent: Monday, March 27, 2006 10:02 PM
Subject: Re: [BackupPC-users] why are full backups needed with BackupPC?


> On Mon, Mar 27, 2006 at 12:52:31AM +0200, Laurent Mazet wrote:
>> To summarize, for a Windows host:
>> - rsync over ssh doesn't work.
>> - rsyncd transfers only diff but you need to connect with a clear 
>> password.
>
> You might be able to run rsyncd transfers over an ssh tunnel.  I
> don't know if there are issues with that on cygwin or not.
>
> Does anyone know if rsync and ssh will run under uwin?  or if
> uwin still even really works?
>
> danno
> --
> dan pritts - systems administrator - internet2
> 734/352-4953 office        734/834-7224 mobile
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting 
> language
> that extends applications into web and mobile media. Attend the live 
> webcast
> and join the prime developer group breaking into this new coding 
> territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@...
> https://lists.sourceforge.net/lists/listinfo/backuppc-users
> http://backuppc.sourceforge.net/
>

On Mon, 2006-27-03 at 22:11 -0600, Trey Nolen wrote:
> Rsyncd over ssh work on windows...rsync will still not act like a server 
> under cygwin.

I've tried using rsyncd with cygwin on the Windows side - it's never
worked for me.  How did you get it to work? (or am I misunderstanding
something?)

Regards,

Ranbir

-- 
Kanwar Ranbir Sandhu
Linux 2.6.15-1.1833_FC4 i686 GNU/Linux 
11:48:59 up 5:18, 2 users, load average: 0.11, 0.15, 0.16

On Tue, 2006-03-28 at 10:50, Kanwar Ranbir Sandhu wrote:
> On Mon, 2006-27-03 at 22:11 -0600, Trey Nolen wrote:
> > Rsyncd over ssh work on windows...rsync will still not act like a server 
> > under cygwin.
> 
> I've tried using rsyncd with cygwin on the Windows side - it's never
> worked for me.  How did you get it to work? (or am I misunderstanding
> something?)

Rsyncd is rsync listening directly in --daemon  mode instead of
being started by sshd.  You shouldn't have any trouble setting
that up under windows except possibly making it run as a
service.

-- 
  Les Mikesell
   les@...

On Tue, 2006-28-03 at 12:14 -0600, Les Mikesell wrote:
> Rsyncd is rsync listening directly in --daemon  mode instead of
> being started by sshd.  You shouldn't have any trouble setting
> that up under windows except possibly making it run as a
> service.

That bit works, though once in a while the rsyncd service (on Windows)
decides to die once in a while.

What hasn't worked for, ever, is using ssh to connect to the Windows
box, and then pull the backups down by connecting to the rsyncd. That
being said, I seem to recall someone reporting that they got it to work,
so I should probably take a look at the list archives.

Anyway, have you tried this? Has it worked for?

Regards,

Ranbir

-- 
Kanwar Ranbir Sandhu
Linux 2.6.15-1.1833_FC4 i686 GNU/Linux 
14:41:24 up 8:11, 3 users, load average: 0.57, 0.39, 0.34

On Tue, 2006-03-28 at 13:44, Kanwar Ranbir Sandhu wrote:

> What hasn't worked for, ever, is using ssh to connect to the Windows
> box, and then pull the backups down by connecting to the rsyncd. That
> being said, I seem to recall someone reporting that they got it to work,
> so I should probably take a look at the list archives.
> 
> Anyway, have you tried this? Has it worked for?

I haven't done it with backuppc, but it should work using
ssh only as a port-forwarding tunnel.  The only problem is
that you don't know when the backup is complete so you
might have to leave the ssh connection up most of the time.

I have VPN connections (currently CIPE but I'd probably use
openvpn now) to all of the places where I need backups so
I haven't worked very hard at ssh'ing into windows.  But,
I've been hoping someone would track down the deadlock since
about everything else works with cygwin sshd.

-- 
  Les Mikesell
   les@...

On Tue, 2006-28-03 at 15:00 -0600, Les Mikesell wrote:
> On Tue, 2006-03-28 at 13:44, Kanwar Ranbir Sandhu wrote:
> 
> > What hasn't worked for, ever, is using ssh to connect to the Windows
> > box, and then pull the backups down by connecting to the rsyncd. That
> > being said, I seem to recall someone reporting that they got it to work,
> > so I should probably take a look at the list archives.
> > 
> > Anyway, have you tried this? Has it worked for?

Holy shit - what a horrible reply.  I'm half asleep...guess coherence
goes out the window when I type with me eyes half closed.  :)  Sorry
about all the missing words!

> I haven't done it with backuppc, but it should work using
> ssh only as a port-forwarding tunnel.  The only problem is
> that you don't know when the backup is complete so you
> might have to leave the ssh connection up most of the time.

Yeah, that much I know.  Perhaps I didn't do the port forwarding
correctly the last time I tried this.  The other problem of course was
with sites that have multiple servers behind a firewall.  That would
mean poking a few holes through a firewall, and opening SSH up.  I know
access can be restricted, but a lot of sites just don't like it.

> I have VPN connections (currently CIPE but I'd probably use
> openvpn now) to all of the places where I need backups so
> I haven't worked very hard at ssh'ing into windows.  But,
> I've been hoping someone would track down the deadlock since
> about everything else works with cygwin sshd.

That's what I ended up doing.  OpenVPN for all the clients that are
either running Windows or don't want to open up SSH, and SSH for those
Linux sites that don't mind.

SSH is by far the simplest and fastest setup.

Regards,

Ranbir

-- 
Kanwar Ranbir Sandhu
Linux 2.6.15-1.1833_FC4 i686 GNU/Linux 
15:57:03 up 9:26, 3 users, load average: 0.31, 0.50, 0.36