rsnapshot

Eric S. Johansson wrote:
> I may have spoken about this problem before but it's taken me a while to get the
> problems with the server cleaned up.  *that* experience suggests that one should
> not purchase an HP server without purchasing the full Red Hat plus HP support
> package because, you will need it.
> 
> for some reason, the rsnapshot process stalls  and blocks everything.
> Obviously, this is unacceptable.  I would rather have rsync  retry/timeout
> selected try to diagnose which machines are hanging up.  I think  this is a
> problem  backing up from  Windows boxes for the most part.  is there anyway to
> get this kind of timeout behavior out of rsync and error reporting from rsnapshot?

related q:  can  windows rsync (deltacopy) lockup if it is asked to transfer to
many files?


---eric

Eric S. Johansson wrote:
> I may have spoken about this problem before but it's taken me a while to get the
> problems with the server cleaned up.  *that* experience suggests that one should
> not purchase an HP server without purchasing the full Red Hat plus HP support
> package because, you will need it.
> 
> for some reason, the rsnapshot process stalls  and blocks everything.
> Obviously, this is unacceptable.  I would rather have rsync  retry/timeout
> selected try to diagnose which machines are hanging up.  I think  this is a
> problem  backing up from  Windows boxes for the most part.  is there anyway to
> get this kind of timeout behavior out of rsync and error reporting from rsnapshot?
> 

another detail: Iran a series of test this afternoon and it would hang on
Windows machines only and different ones each time

On Mon, Dec 29, 2008 at 02:50:10PM -0500, Eric S. Johansson wrote:
> 
> for some reason, the rsnapshot process stalls  and blocks everything.
> Obviously, this is unacceptable.  I would rather have rsync  retry/timeout
> selected try to diagnose which machines are hanging up.  I think  this is a
> problem  backing up from  Windows boxes for the most part.  is there anyway to
> get this kind of timeout behavior out of rsync and error reporting from rsnapshot?

I had a problem like that a couple of years ago, when trying to back up
Windows boxes with rsync over ssh, using cygwin.  Most of the time it
would rsync a few files and then hang.  I could reproduce the problem 
with a manual rsync (no need to involve rsnapshot at all).

Adding --timeout=3600 to the rsync options helped a bit (so at least
it would exit after no input/output for 1 hour), but what REALLY helped
was changing it from using rsync over ssh to rsync over rsyncd.

That involved configuring & starting an rsyncd service on the Windows box 
and changing the source syntax for the rsnapshot backup line to something
like :-
	rsync://user@...>
I used cwrsync on the Windows box.

-- 
___________________________________________________________________________
 David Keegel <djk@...>  http://www.cyber.com.au/users/djk/
 Cybersource P/L: Linux/Unix Systems Administration Consulting/Contracting

On Tue, 2008-12-30 at 09:49 +1100, David Keegel wrote:
> On Mon, Dec 29, 2008 at 02:50:10PM -0500, Eric S. Johansson wrote:
> > 
> > for some reason, the rsnapshot process stalls  and blocks everything.
> > Obviously, this is unacceptable.  I would rather have rsync  retry/timeout
> > selected try to diagnose which machines are hanging up.  I think  this is a
> > problem  backing up from  Windows boxes for the most part.  is there anyway to
> > get this kind of timeout behavior out of rsync and error reporting from rsnapshot?
> 
> I had a problem like that a couple of years ago, when trying to back up
> Windows boxes with rsync over ssh, using cygwin.  Most of the time it
> would rsync a few files and then hang.  I could reproduce the problem 
> with a manual rsync (no need to involve rsnapshot at all).

Yes, there's a well-known Cygwin hang bug that rsync seems to hit
especially often.  IIRC, the problem is that the Windows API doesn't
provide the necessary functionality to fully emulate select(2) on pipes,
so Cygwin does the best it can and fails sometimes.  Setting a --bwlimit
may avoid the hang if the speed reduction is acceptable.

> Adding --timeout=3600 to the rsync options helped a bit (so at least
> it would exit after no input/output for 1 hour),

I'll add that one can make rsync restart automatically on timeout using
the recently added "rsync_numtries" configuration parameter.

> but what REALLY helped
> was changing it from using rsync over ssh to rsync over rsyncd.

That often helps, but I do recall one report of a hang when connecting
directly to an rsync daemon.

-- 
Matt

Matt McCutchen wrote:

> Yes, there's a well-known Cygwin hang bug that rsync seems to hit
> especially often.  IIRC, the problem is that the Windows API doesn't
> provide the necessary functionality to fully emulate select(2) on pipes,
> so Cygwin does the best it can and fails sometimes.  Setting a --bwlimit
> may avoid the hang if the speed reduction is acceptable.

interesting.  I will try the bandwidth limit to see if that helps any as well as
the rsync_numtries options.


> 
> That often helps, but I do recall one report of a hang when connecting
> directly to an rsync daemon.

make that two reports.

Hallo, Brian,

Du (brian) meintest am 29.12.08:

> ~sigh~  Now I have to go cleaning up my snapshot volume restoring the
> hard link pool where (incorrectly) new files have been created.
> Silly, silly, silly.

But "freedup" should help a lot ...

        http://freedup.org
        http://arktur.de/Wiki/Zusatzprogramme:rsnapshot#freedup

I've just repaired the backups of one of my machines:

stop the cronjob which might produce a new snapshot

        cd /path/to/hourly.0

        freedup -0Tfalk# -D3700 daily.3 hourly.?

where "daily.3" is a good snapshot.
Then "freedup" uses "daily.3" (or any other good snapshot) as a template  
for the hard links of all the rotten snapshots (here: all "hourly"  
snapshots).

On a system with only 384 MB RAM I had to split the job with

        -o "-size +1k -size -50k"

(etc.) into smaller parts, otherwise I got "segmentation fault".

Viele Gruesse!
Helmut

On Mon, 2008-12-29 at 08:53 -0500, Brian J. Murrell wrote:
> Perhaps if rsync fails for some fatal reason (i.e. not "source file
> bla-bla-bla changed while reading it", etc.) then perhaps an rsync
> --link-dest between n-1 and n should be done to "fill out" whatever
> failed during the original rsync.

This is already implemented; the code calls it "rollback" (though I
argued that "holdover" would be a better term).  When it occurs,
rsnapshot should print a message like "Rolling back BACKUP-POINT-DEST".
Do you not get this message?

Note that rollback in sync_first mode was broken due to a typo until
rsnapshot 1.3.1.

-- 
Matt

On Tue, 2008-12-23 at 15:07 -0600, Dave Dykstra wrote:
> On Tue, Dec 23, 2008 at 01:52:21PM -0500, Matt McCutchen wrote:
> > I too find the current semantics of basis dirs illogical and would
> > support an option for better semantics (though I think we should avoid
> > changing defaults some users may be relying on).  That's why I entered a
> > series of enhancement requests, the first of which is Robert's proposal:
> > 
> > https://bugzilla.samba.org/show_bug.cgi?id=5644
> > https://bugzilla.samba.org/show_bug.cgi?id=5645
> > https://bugzilla.samba.org/show_bug.cgi?id=5646
> > 
> > I would consider it worth forking rsync to have these features, if I
> > ever got around to doing so.
> 
> It (that is, the proposal in your bug first report) sounds like not
> a very difficult or intrusive addition to me, and useful for rotating
> backups.  Wayne said he wouldn't write it, but maybe he'd accept it
> if someone else wrote it.  That would be better than forking unless
> it is more intrusive than it seems.

Of course that would be better.  My point was that I would be willing to
fork if necessary, as an indication of how important I believe the
feature is.

-- 
Matt

On Tue, Dec 23, 2008 at 01:52:21PM -0500, Matt McCutchen wrote:
> On Tue, 2008-12-23 at 23:45 +1100, David Overton wrote:
> > I would also very much like to see this feature. Indeed, this seems
> > far more logical than the current --link-dest behaviour and it's what
> > I assumed --link-dest would do until I read the man page thoroughly
> > (you have to follow the references back from --link-dest to
> > --copy-dest and then --compare-dest and even then the only mention of
> > the actual behaviour is a parenthesised comment "(if the files are
> > missing in the destination directory)"). I be interested to know what
> > use cases the current behaviour was designed for, because I can't
> > see any advantage to not making use of the --link-dest file if it's
> > available. Providing the proposed alternative behaviour as an extra
> > option, if not the default for --link-dest, would be very useful.
> 
> I too find the current semantics of basis dirs illogical and would
> support an option for better semantics (though I think we should avoid
> changing defaults some users may be relying on).  That's why I entered a
> series of enhancement requests, the first of which is Robert's proposal:
> 
> https://bugzilla.samba.org/show_bug.cgi?id=5644
> https://bugzilla.samba.org/show_bug.cgi?id=5645
> https://bugzilla.samba.org/show_bug.cgi?id=5646
> 
> I would consider it worth forking rsync to have these features, if I
> ever got around to doing so.

It (that is, the proposal in your bug first report) sounds like not
a very difficult or intrusive addition to me, and useful for rotating
backups.  Wayne said he wouldn't write it, but maybe he'd accept it
if someone else wrote it.  That would be better than forking unless
it is more intrusive than it seems.

(I'm taking an interest because I contributed the code for the
original --*-dest option --compare-dest and that is primarily what
pulled me in to maintaining rsync for a year or so.  I also integrated
JW Schultz's --link-dest.  --*dest options have evolved quite a bit
since then I see.)

- Dave

On 20081222T182646-0800, Ido Magal <ido.magal@...> wrote:
>I just set up rsnapshot on my ubuntu box.  I cobbled together the following
>script to email the rsync log and the stderr once the backup is complete.
>
>The problem I have is that the bash wait doesn't behave the way I expect it
>given all I've googled.  The script mails immediately and then proceeds to
>rsync.  I would appreciate any help in getting this script to execute as
>designed.
>
>#!/bin/sh
>
>LOGFILE=/mnt/drobo/home/rsync.$1.log
>ERRFILE=/mnt/drobo/home/rsync.$1.err
>CONFFILE=/mnt/drobo/home/rsnapshot.$1.conf
>
>[ -f $LOGFILE ] && rm $LOGFILE
>[ -f $ERRFILE ] && rm $ERRFILE
>
>/usr/bin/rsnapshot -c $CONFFILE $2 2>$ERRFILE
>wait
>[ -f "$LOGFILE" ] %% cat $LOGFILE| mailx -s"[backup log: "$1"]" $1
>[ -f "$ERRFILE" ] %% cat $ERRFILE| mailx -s"[backup errors: "$1"]" $1

Seems like those double percent signs should be && operators.

On Mon, 2008-12-22 at 18:26 -0800, Ido Magal wrote:\
> I just set up rsnapshot on my ubuntu box.  I cobbled together the
> following script to email the rsync log and the stderr once the backup
> is complete.
> 
> The problem I have is that the bash wait doesn't behave the way I
> expect it given all I've googled.  The script mails immediately and
> then proceeds to rsync.  I would appreciate any help in getting this
> script to execute as designed.

> #!/bin/sh 
> 
> LOGFILE=/mnt/drobo/home/rsync.$1.log
> ERRFILE=/mnt/drobo/home/rsync.$1.err
> CONFFILE=/mnt/drobo/home/rsnapshot.$1.conf
> 
> [ -f $LOGFILE ] && rm $LOGFILE
> [ -f $ERRFILE ] && rm $ERRFILE
> 
> /usr/bin/rsnapshot -c $CONFFILE $2 2>$ERRFILE
> wait
> [ -f "$LOGFILE" ] %% cat $LOGFILE| mailx -s"[backup log: "$1"]" $1
> [ -f "$ERRFILE" ] %% cat $ERRFILE| mailx -s"[backup errors: "$1"]" $1

Your rsnapshot command is in the foreground (no &), so bash should be
waiting even without a "wait" command.  The % signs should be & as
Kenyon said, but that doesn't explain the problem.  There must be
something else going on here.

Try changing the shebang line at the top to "#!/bin/bash".  Some people
have gotten into a bad habit of writing "#!/bin/sh" for scripts that use
bash-specific features, and Ubuntu (possibly among other distributions)
is calling them on it.  I know bash but have never taken the time to
learn what features are part of the standard sh, so I just always write
"#!/bin/bash".

If that doesn't fix the problem, try adding "set -x" to see the commands
as the shell is executing them.

-- 
Matt

On Mon, Dec 22, 2008 at 03:28:02PM -0800, Chris Petersen wrote:
> I've been trying to maintain the Fedora/CentOS/RHEL package for
> rsnapshot for awhile, but find that I keep getting "trumped" by other
> maintainers who notice new versions getting released and update the
> packages for me, or complain that I haven't updated them fast enough.
> 
> Do the authors ever plan to have some sort of "announce" list, or at
> least post messages to this list to announce the new versions (hopefully
> along with at least a link to the changelog).  Maybe there already have
> been, but I don't think I've ever seen announcement messages posted here.

I posted an announcement of rsnapshot 1.3.1 to rsnapshot-discuss.
(Forwarded privately to Chris.)

I generally give a summary (highlights) of the ChangeLog in the 
announcement message.  For full details you can check the ChangeLog
file in the tar or rpm or in cvs.

If you're a developer type person and you're looking for clues about
when a release of rsnapshot is going to be coming soon, you might 
want to subscribe to rsnapshot-cvs, which gets notifications of CVS
commits.  (I tend to skip notifications for some trivial changes
I make, for example adding a line to the ChangeLog for a change I 
have just committed to CVS, where the change is obvious.)

-- 
___________________________________________________________________________
 David Keegel <djk@...>  http://www.cyber.com.au/users/djk/
 Cybersource P/L: Linux/Unix Systems Administration Consulting/Contracting

The easiest way to tell is to see the stderr output of rsnapshot.

If you're running rsnapshot from cron, you would normally find 
that wherever root's email goes.  Or it might be easier to just 
run rsnapshot manually, so stderr is displayed on your screen.

It could be something like running out of disk space.

On Sun, Dec 21, 2008 at 08:33:35PM +0100, Hedy Darg??re wrote:
> 
>    Hi guy,
>    First, thanks for this wonderfull tool (rsnapshot)
>    I have a problem on one host, when I'm using rsnapshot.
>    Here the log :
>    ==============
>    WARNING: Rolling back "aedania/"
>    /usr/bin/logger -i -p user.err -t rsnapshot WARNING: Rolling back \
>    Â Â Â  "aedania/"
>    /bin/rm -rf /home/hedy/backups/files/daily.0/aedania/
>    /bin/cp -al /home/hedy/backups/files/daily.1/aedania \
>    Â Â Â  /home/hedy/backups/files/daily.0/aedania
>    ERROR: /bin/cp failed. Perhaps this is not GNU cp?
>    ERROR:    Error!    cp_al("/home/hedy/backups/files/daily.1/aedania/",
>    "/home/hedy/backups/files/daily.0/aedania/")
>    /usr/bin/logger  -i -p user.err -t rsnapshot /usr/bin/rsnapshot daily:
>    \
>    Â Â Â  ERROR: Error!
>    cp_al("/home/hedy/backups/files/daily.1/aedania/", \
>    Â Â Â  "/home/hedy/backups/files/daily.0/aedania/")
>    ==============
>    I don't know why I suddently have this problem (I have the GNU version
>    of  cp  for  long  time  ago  !!!) and rsnapshot work very well with 2
>    others distants hosts.
>    Do you have an idea ?
>    Hedy

> ------------------------------------------------------------------------------

> _______________________________________________
> rsnapshot-discuss mailing list
> rsnapshot-discuss@...
> https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss


-- 
___________________________________________________________________________
 David Keegel <djk@...>  http://www.cyber.com.au/users/djk/
 Cybersource P/L: Linux/Unix Systems Administration Consulting/Contracting

> The easiest way to tell is to see the stderr output of rsnapshot.
>
> If you're running rsnapshot from cron, you would normally find 
> that wherever root's email goes.  Or it might be easier to just 
> run rsnapshot manually, so stderr is displayed on your screen.
>   
In fact, I post the stderr output of rsnapshot
I run rsnapshot manually.

> It could be something like running out of disk space.
>   
I have verified the disk space and it was ok....
I don't know to do else...

Maybe it can help, after another run of rsnapshot, the error change
(why??!!!)

Here the stderr :

============================================================
rsync: connection unexpectedly closed (2825735 bytes received so far)
[receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(453)
[receiver=2.6.9]
rsync: writefd_unbuffered failed to write 1080 bytes [generator]: Broken
pipe (32)
rsync error: error in rsync protocol data stream (code 12) at io.c(1122)
[generator=2.6.9]
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot daily
----------------------------------------------------------------------------
ERROR: /usr/bin/rsync returned 12 while processing
root@.../
/usr/bin/logger -i -p user.err -t rsnapshot /usr/bin/rsnapshot daily: \
    ERROR: /usr/bin/rsync returned 12 while processing \
    root@.../
WARNING: root@.../ skipped due to rollback plan
/usr/bin/logger -i -p user.err -t rsnapshot WARNING: root@.../ \
    skipped due to rollback plan
============================================================

Any idea of the meaning of "skipped due to rollback plan" ?
I had this error several time without understanding what was the problem....

On Mon, Dec 22, 2008 at 12:37:47AM +0100, Hedy Darg?re wrote:
> Maybe it can help, after another run of rsnapshot, the error change
> (why??!!!)
> 
> Here the stderr :
> 
> ============================================================
> rsync: connection unexpectedly closed (2825735 bytes received so far)
> [receiver]
> rsync error: error in rsync protocol data stream (code 12) at io.c(453)
> [receiver=2.6.9]
> rsync: writefd_unbuffered failed to write 1080 bytes [generator]: Broken
> pipe (32)
> rsync error: error in rsync protocol data stream (code 12) at io.c(1122)
> [generator=2.6.9]
> ----------------------------------------------------------------------------

Now we can see the problem.

That indicates a network error or network timeout.
The network connection was broken/lost while rsync was working,
so rsync gave an error (it did not finish its job).

This where you should focus: why was the network connection closed
unexpectedly?

It could be something like the network between server and aedania 
going down, or the rsync process on aedania being killed.

> rsnapshot encountered an error! The program was invoked with these options:
> /usr/bin/rsnapshot daily
> ----------------------------------------------------------------------------
> ERROR: /usr/bin/rsync returned 12 while processing
> root@.../
> /usr/bin/logger -i -p user.err -t rsnapshot /usr/bin/rsnapshot daily: \
>     ERROR: /usr/bin/rsync returned 12 while processing \
>     root@.../
> WARNING: root@.../ skipped due to rollback plan
> /usr/bin/logger -i -p user.err -t rsnapshot WARNING: root@.../ \
>     skipped due to rollback plan
> ============================================================
> 
> Any idea of the meaning of "skipped due to rollback plan" ?
> I had this error several time without understanding what was the problem....

This is because of previous problem (rsync failed with a network error).
So rsnapshot wants to roll the state back to where it was before the
backup was started.

> ------------------------------------------------------------------------------
> _______________________________________________
> rsnapshot-discuss mailing list
> rsnapshot-discuss@...
> https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss

-- 
___________________________________________________________________________
 David Keegel <djk@...>  http://www.cyber.com.au/users/djk/
 Cybersource P/L: Linux/Unix Systems Administration Consulting/Contracting

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
<blockquote cite="mid:20081222011043.GP5216@..."
 type="cite">
  <pre wrap="">
Now we can see the problem.

That indicates a network error or network timeout.
The network connection was broken/lost while rsync was working,
so rsync gave an error (it did not finish its job).

This where you should focus: why was the network connection closed
unexpectedly?

It could be something like the network between server and aedania 
going down, or the rsync process on aedania being killed.
  </pre>
</blockquote>
<br>
<pre wrap="">Hummm I don't understand why the connexion was closed unexpectedly...

In fact, I have several mistake with rsnapshot.
It's look like unreliable because the error message are not very expressives.
Be carrefull : I'm pretty sure the problem comes from my configuration,
not rsnapshot <span class="moz-smiley-s3"><span> ;) </span></span>
I just say that is not easy to understand what's happening, and more
precisely, do know what to do to solve the problem !

And what's means "WARNING: Rolling back" ????


I tried to start a new backup, from fresh directory (my directory
<i class="moz-txt-slash"><span class="moz-txt-tag">/</span>home/hedy/backups/files<span
 class="moz-txt-tag">/</span></i> is empty)
And I had this new error :

====================================================
rsnapshot daily
Setting locale to POSIX "C"
echo 3466 &gt; /var/run/rsnapshot.pid
mkdir -m 0755 -p /home/hedy/backups/files/daily.0/
/usr/bin/rsync -av --delete --numeric-ids --relative --delete-excluded \
    --exclude-from=/etc/rsnapshot-exclude.conf --rsh=/usr/bin/ssh \
    root@... class="moz-txt-slash"><span class="moz-txt-tag">/</span>var/www/virtual /home/hedy/backups/files/daily.0/minggu<span
 class="moz-txt-tag">/</span></i>
receiving file list ... done
created directory /home/hedy/backups/files/daily.0/minggu
rsync: fork failed in do_recv: Cannot allocate memory (12)
rsync error: error in IPC code (code 14) at main.c(709) [receiver=2.6.9]
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot daily
----------------------------------------------------------------------------
ERROR: /usr/bin/rsync returned 14 while processing
root@... class="moz-txt-slash"><span class="moz-txt-tag">/</span>var/www/virtual<span
 class="moz-txt-tag">/</span></i>
/usr/bin/logger -i -p user.err -t rsnapshot /usr/bin/rsnapshot daily: \
    ERROR: /usr/bin/rsync returned 14 while processing \
    root@... class="moz-txt-slash"><span class="moz-txt-tag">/</span>var/www/virtual<span
 class="moz-txt-tag">/</span></i>
WARNING: root@... class="moz-txt-slash"><span class="moz-txt-tag">/</span>etc<span
 class="moz-txt-tag">/</span></i> skipped due to rollback plan
<i class="moz-txt-slash"><span class="moz-txt-tag">/</span>usr/bin/logger -i -p user.err -t rsnapshot WARNING: root@...
 class="moz-txt-tag">/</span></i> \
    skipped due to rollback plan
=====================================================

One again, what's appenning  ???!!!
I have disk free :

=================================================
Sys. de fich.         Tail. Occ. Disp. %Occ. Monté sur
<i class="moz-txt-slash"><span class="moz-txt-tag">/</span>dev<span
 class="moz-txt-tag">/</span></i>.static/dev/sdb1
                      9,7G  557M  8,6G   6% /
tmpfs                  15M     0   15M   0% /lib/init/rw
udev                   10M   48K   10M   1% /dev
tmpfs                  15M     0   15M   0% /dev/shm
<i class="moz-txt-slash"><span class="moz-txt-tag">/</span>dev<span
 class="moz-txt-tag">/</span></i>.static/dev/sda1
                      694G   60G  599G  10% /home/hedy
=================================================

SSH connexions from the server which run rsnapshot and the differents
hosts are working well !!

I don't know what do check <span class="moz-smiley-s2"><span> :( </span></span></pre>
<br>
</body>
</html>