BackupPC

> I ran a several gig backup manually last night (just doing manual starts), 
> and this morning it's still showing:
> - Pool is 0.00GB comprising 25 files and 62 directories (as of 3/1 02:35),
> - Pool hashing gives 0 repeated files with longest chain 0,
> 
> How often do these statistics update? And seeing as though it shows the 62 
> directories, shouldn't it show >0.0GB?

They update each night.  I guess your 25 files are less than 5MB, so that
rounds down to 0.00GB.

I replied to your later rsync email.

Craig

At 10:59 PM 3/1/2003 -0800, you wrote:
> >
> > I then created an rsyncd.conf:
> > log file = test.log
> > [Backup]
> > strict modes = false
> > path = /temp
> > auth users = test
> > secrets file = rsyncd.secrets
> >
> > and a rsyncd.secrets:
> > test:testpassword
> >
> > I think executed rsync like so:
> > rsync --daemon --config=rsyncd.conf
>
>You should also specify:
>
>     use chroot = no
>     strict modes = false
>
>at the top of /etc/rsyncd.conf.  Rsync is refusing to look at the
>secrets file since it is not owned by root (cygwin doesn't emulate
>file ownership by root).


Still no dice - auth failed on module Backup:

rsync rsync://test@...> shows a list of the mount points I have defined 
on the remote daemon, but :

rsync rsync://test@...> *.exe prompts for password, then 
denies access.

When I set BackupPC to use rsyncd and point it to that machine, I get:
2003/3/2 12:12:34 Got fatal error during xfer (Unknown module '/Backup/')
2003/3/2 12:12:44 Dump aborted (Unknown module '/Backup/')

Am I doing something drastically wrong? :)

>I would recommend you also use the newest rsync 2.5.6.  It includes a
>patch that I submitted (see patches/craigb-perf.diff in the distribution)
>that substantially improves the performance with cygwin.

This is available from cygwin.com? I didn't see it in a quick look around, 
and the setup still seems to download 2.5.5.


>You can also use cgyrunsrv to install rsync as a service so that it
>auto-runs every time you boot your WinXX machine.  This worked for
>me on WinXP; not tested on other WinXX:

Cool - I'll dig into that once I get things functioning, thanks.


>I am tempted to bundle up the three files you mentioned plus cygrunsrv.exe
>and put them on SourceForge PerlRsync project.

That'd be slick .. and a quick installer .bat .. send it to the end-users, 
tell em "double click this".


>Rsync is a lot better than smbclient for WinXX: it's faster (3x for
>a full backup for me) and it handles renamed/deleted/any attribute
>changes in incrementals, unlike smbclient.  Plus I bet it is more
>secure.

Well, I don't know about more secure.  The authentication seems to be an 
MD4 128, but the data itself is in clear text afaik.  Ultimately, I'd 
really prefer to run it over ssh, but I doubt there's a real smooth way to 
install that on win9x clients.


>If you want to write your notes and mine up I will add it to the docs
>or FAQ.

Sure, once I get things functioning, I'll document my findings and get it 
to you.


>One last note: apparently cygwin breaks if you have multiple cygwin1.dll
>files on a machine, so people discourage the idea of bundling cygwin1.dll
>with a package like I just proposed.  Perhaps we should recommend the
>two dlls be moved to \WINDOWS\SYSTEM32?

I suppose this could be a job for a little binary installer .. quick scan 
of loaded DLLs or an option to "search local drives for ...".  That sorta 
takes the simplicity out of things, though.  Maybe just tossing it into 
system32 would be the easiest.

Thanks,

denon

> > > I then created an rsyncd.conf:
> > > log file = test.log
> > > [Backup]
> > > strict modes = false
> > > path = /temp
> > > auth users = test
> > > secrets file = rsyncd.secrets
> > >
> > > and a rsyncd.secrets:
> > > test:testpassword
> > >
> > > I think executed rsync like so:
> > > rsync --daemon --config=rsyncd.conf
> >
> >You should also specify:
> >
> >     use chroot = no
> >     strict modes = false
> >
> >at the top of /etc/rsyncd.conf.  Rsync is refusing to look at the
> >secrets file since it is not owned by root (cygwin doesn't emulate
> >file ownership by root).
> 
> Still no dice - auth failed on module Backup:

> secrets file = rsyncd.secrets

How about you try a full path to rsyncd.secrets?  In my rsyncd.conf file
on WinXX I use:

    secrets file = /etc/rsyncd.secrets

> When I set BackupPC to use rsyncd and point it to that machine, I get:
> 2003/3/2 12:12:34 Got fatal error during xfer (Unknown module '/Backup/')
> 2003/3/2 12:12:44 Dump aborted (Unknown module '/Backup/')

The share name should be 'Backup', not '/Backup'.

> This is available from cygwin.com? I didn't see it in a quick look around, 
> and the setup still seems to download 2.5.5.

It's not on cygwin.com yet.  2.5.6 is on the rsync download page and
you can compile it under cygwin pretty easily.

> >Rsync is a lot better than smbclient for WinXX: it's faster (3x for
> >a full backup for me) and it handles renamed/deleted/any attribute
> >changes in incrementals, unlike smbclient.  Plus I bet it is more
> >secure.
> 
> Well, I don't know about more secure.  The authentication seems to be an 
> MD4 128, but the data itself is in clear text afaik.  Ultimately, I'd 
> really prefer to run it over ssh, but I doubt there's a real smooth way to 
> install that on win9x clients.

Yes, the data is in plain text, but so is SMB.

Craig

> I'd like to second this.  I currently specify the folders that I'd like
> backed up.  But, I also would like to specify that any folder named
> cache or temp or files like *tmp or *.gif, no matter what location, be
> ignored.  Is this possible?  For me it's not a huge push, but will be
> nice in the future.

Not possible with smbclient (you can't specify both include and exclude files).
But it is possible with rsync.

Craig

Sorry to follow up on my own question, but I've discovered a little more 
about rsync on win32.  I've copied the following files to an arbitrary 
workstation:
cygpopt-0.dll
cygwin1.dll
rsync.exe

I then created an rsyncd.conf:
log file = test.log
[Backup]
strict modes = false
path = /temp
auth users = test
secrets file = rsyncd.secrets

and a rsyncd.secrets:
test:testpassword

I think executed rsync like so:
rsync --daemon --config=rsyncd.conf

No matter how I tried to access it, though, it would prompt for the 
password. then I'd get an authentication error.  The logs gave no more 
insight. ([2240] auth failed on module backup).

Has anyone had problems with cygwin's rsync?  It seems to do the same thing 
on a fully-installed cygwin, however the same config seems to work ok on a 
freebsd machine, but I may be overlooking something ..


Thank in advance,

denon

>From: "denon" <denon@...>
> 
> Has anyone had problems with cygwin's rsync?  It seems to do the same thing 
> on a fully-installed cygwin, however the same config seems to work ok on a 
> freebsd machine, but I may be overlooking something ..

I've never used password access with it, but you can use IP based
access control or if you have the full cygwin package installed with sshd
as a service you can run it under ssh control just like a unix system
(except for having to use /cygdrive/c, etc. as the path to the drives).

---
  Les Mikesell
      les@...

Thanks.

I'd like ssh, but I'm really trying to get away from installing the cygwin 
package on each of the remote machines.  If there's an alternative, or 
someone's made some nice rollout scripts, I'd love to hear about em. :)



At 12:54 AM 3/2/2003 -0600, you wrote:
> >From: "denon" <denon@...>
> >
> > Has anyone had problems with cygwin's rsync?  It seems to do the same 
> thing
> > on a fully-installed cygwin, however the same config seems to work ok on a
> > freebsd machine, but I may be overlooking something ..
>
>I've never used password access with it, but you can use IP based
>access control or if you have the full cygwin package installed with sshd
>as a service you can run it under ssh control just like a unix system
>(except for having to use /cygdrive/c, etc. as the path to the drives).
>
>---
>   Les Mikesell
>       les@...

Craig,

I'd like to second this.  I currently specify the folders that I'd like
backed up.  But, I also would like to specify that any folder named
cache or temp or files like *tmp or *.gif, no matter what location, be
ignored.  Is this possible?  For me it's not a huge push, but will be
nice in the future.

I guess you could call this my feature request :-)

Doug

Johnny L. Wales wrote:

>Howdy!
>
>I'd like to exclude .exe/.dll/.com/.mp3 files from being backed up with
>BackupPC. I can't for the life of me figure out the correct method for
>doing this. Reading through smbclient's documentation, I should just have
>
>  
>

> > From: Brad Penner
> > 
> > BackupPC is using all 4GB of memory on my server.  Has anyone
> > else seen this type of behavior? 
> 
> How are you checking the usage?  It is normal for Linux systems
> to report all memory as 'used' in buffers and cache.  This is
> really free to be re-used if something else needs it.  However
> if ps or top shows processes related to backuppc with sizes that
> are consuming all your memory you have something wrong.

You are right: this is normal linux behavior.  Brad told me in
a direct email that:

brad> Unfortunately, I can't really tell what process is actually using
brad> the memory.  Top doesn't show any process using that much, it just
brad> says the memory is being used in the summary section.

and

brad> I think it's just a RH thing.  I noticed all of out redhat systems
brad> show the memory as all being used, so it must be a cache thing.

Craig

>   Logging into the Linux box, I see that the BackupPC_Admin cgi was at 
> 99% and the memeory useage (Viewing via top) was
> at 56mb and increasing.  Killing BackupPC's PID didn't effect the cgi.  
> Did a kill on cgi and everything went back to normal.  Started the job
> backup up, did several clicks though the tree again and did several 
> restores with no issues.  Then on my 6th restore (Doing a little at a 
> time), it
> happened again.
> 
>   Ok, it just happened again and I tracked down what steps to take to 
> recreate it:  I don't know if this has been taken care of in the BETA or 
> not:
> 
> 1).      Backuped up a Win2k laptop
> 2).      Went into the BackupPC_Admin cgi.
> 3).      Browse one folder of the tree.
> 4).      Click on the C$ share
> 5).      Click on another folder.
> 6).      Repeat several tims with no problems.
> 7).      Click on a folder, but this time, instead of clicking on the C$ 
> before clicking on another folder, click on a different folder in the 
> list.  BOOOM!
>           CGI hangs, CPU between 70 and 99%, memory usage on the CGI 
> keeps climbing.

This is certainly a bug in BackupPC_Admin.

Could you try the 2.0.0 beta version and see if it still happens?

Craig

> It looks like some of the links in the tree view are missing a slash "/" in 
> front of the "dir" variable being passed in the URL to the CGI script. If you
> copy and paste the link into your address bar and add a forward slash to the 
> "dir" value (e.g. change "&dir=foldername" to "&dir=/foldername"), it will
> open the selected folder just fine.
> 
> I was able to "fix" this problem by changing both instances of "$path" to 
> "/$path" on line 647 of BackupPC_Admin. This is (I assume) a hack that only 
> works because there is some regex earlier in the script that removes the 
> duplicate //'s this fix will create on some of the links.
> 
> Perhaps Craig or someone else can find a more elegant solution, but I'd
> be more than happy to look into it further...

Thanks!  Yes, this bug is still in 2.0.0 beta0.  You are right that it
is caused by the missing "/".  I'll look into it.

Craig

> From: Brad Penner
> 
> BackupPC is using all 4GB of memory on my server.  Has anyone
> else seen this type of behavior? 

How are you checking the usage?  It is normal for Linux systems
to report all memory as 'used' in buffers and cache.  This is
really free to be re-used if something else needs it.  However
if ps or top shows processes related to backuppc with sizes that
are consuming all your memory you have something wrong.

---
  Les Mikesell
    les@...

I just ran into something similar to this, but don't know if it's the 
same. Got home a little early and decided to try a restore
to a laptop with a new hard drive.  After doing a backup of the old 
drive, installed the new drive and went into the
BackupPC_Admin cgi (CVS Pre-2.0 BETA) and started to browse the tree. 
After a couple of folders, it seemed  to hang.

  Logging into the Linux box, I see that the BackupPC_Admin cgi was at 
99% and the memeory useage (Viewing via top) was
at 56mb and increasing.  Killing BackupPC's PID didn't effect the cgi.  
Did a kill on cgi and everything went back to normal.  Started the job
backup up, did several clicks though the tree again and did several 
restores with no issues.  Then on my 6th restore (Doing a little at a 
time), it
happened again.

  Ok, it just happened again and I tracked down what steps to take to 
recreate it:  I don't know if this has been taken care of in the BETA or 
not:

1).      Backuped up a Win2k laptop
2).      Went into the BackupPC_Admin cgi.
3).      Browse one folder of the tree.
4).      Click on the C$ share
5).      Click on another folder.
6).      Repeat several tims with no problems.
7).      Click on a folder, but this time, instead of clicking on the C$ 
before clicking on another folder, click on a different folder in the 
list.  BOOOM!
          CGI hangs, CPU between 70 and 99%, memory usage on the CGI 
keeps climbing.

If you click on the C$ share after every folder, everything is ok, but 
going from one folder to another it hangs.

Doug Lytle




Brad Penner wrote:

> BackupPC is using all 4GB of memory on my server.  Has anyone
> else seen this type of behavior?  When if first starts a backup
> job it seems fine, but within 15 minutes all the memory is
> being used.  It's not using swap, just ram.  Is this normal
> or is there some config to fix it?
>
>
> Details:
> backuppc 1.5.0
> redhat 8.0
> dell 6450 with 4GB of RAM
> all clients are backed up via nfs
>
> Thanks,
> Brad
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@...
> https://lists.sourceforge.net/lists/listinfo/backuppc-users
> http://backuppc.sourceforge.net/
>
>

It looks like some of the links in the tree view are missing a slash "/" in 
front of the "dir" variable being passed in the URL to the CGI script. If you 
copy and paste the link into your address bar and add a forward slash to the 
"dir" value (e.g. change "&dir=foldername" to "&dir=/foldername"), it will open 
the selected folder just fine.




I was able to "fix" this problem by changing both instances of "$path" to 
"/$path" on line 647 of BackupPC_Admin. This is (I assume) a hack that only 
works because there is some regex earlier in the script that removes the 
duplicate //'s this fix will create on some of the links.




Perhaps Craig or someone else can find a more elegant solution, but I'd be more 
than happy to look into it further...




-Ryan Kucera




Quoting Doug Lytle <support@...>:




>   I just ran into something similar to this, but don't know if it's the 


> same. Got home a little early and decided to try a restore


> to a laptop with a new hard drive.  After doing a backup of the old 


> drive, installed the new drive and went into the


> BackupPC_Admin cgi (CVS Pre-2.0 BETA) and started to browse the tree. 


> After a couple of folders, it seemed  to hang.


> 


>   Logging into the Linux box, I see that the BackupPC_Admin cgi was at 


> 99% and the memeory useage (Viewing via top) was


> at 56mb and increasing.  Killing BackupPC's PID didn't effect the cgi.  


> Did a kill on cgi and everything went back to normal.  Started the job


> backup up, did several clicks though the tree again and did several 


> restores with no issues.  Then on my 6th restore (Doing a little at a 


> time), it


> happened again.


> 


>   Ok, it just happened again and I tracked down what steps to take to 


> recreate it:  I don't know if this has been taken care of in the BETA or 


> not:


> 


> 1).      Backuped up a Win2k laptop


> 2).      Went into the BackupPC_Admin cgi.


> 3).      Browse one folder of the tree.


> 4).      Click on the C$ share


> 5).      Click on another folder.


> 6).      Repeat several tims with no problems.


> 7).      Click on a folder, but this time, instead of clicking on the C$ 


> before clicking on another folder, click on a different folder in the 


> list.  BOOOM!


>           CGI hangs, CPU between 70 and 99%, memory usage on the CGI 


> keeps climbing.


> 


> If you click on the C$ share after every folder, everything is ok, but 


> going from one folder to another it hangs.


> 


> Doug Lytle