rsnapshot

Hallo, Xavier,

Du meintest am 29.10.10:

> 'sudo rsnapshot hourly' produces the following error:

>> unexpected end of filter rule: -
>> rsync error: syntax or usage error (code 1) at exclude.c(901)


>> /usr/bin/rsync -a --delete --numeric-ids --relative
>>     --delete-excluded \ --exclude-from=/home/xavier/Arc.zip
>>     --exclude-from=/home/xavier/Arc2.zip \ /home
>>     /srv/bak_xrn/shapshots/hourly.0/homes/

Are the 2 exclude files really zipped?

Viele Gruesse!
Helmut

On Fri, Oct 29, 2010 at 09:50:57AM +0200, Xavier wrote:
> 
> 'sudo rsnapshot hourly' produces the following error:
> 
> > unexpected end of filter rule: -
> > rsync error: syntax or usage error (code 1) at exclude.c(901) [client=3.0.7]
...
> 
> If I do 'sudo rsnapshot -t hourly I see that the following line is
> generated:
> 
> > /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
> >     --exclude-from=/home/xavier/Arc.zip --exclude-from=/home/xavier/Arc2.zip \
> >     /home /srv/bak_xrn/shapshots/hourly.0/homes/
... 
> I could track this down to the two following lines in rsnapshot.conf:
> 
> > exclude_file    /home/xavier/Arc.zip
> > exclude_file    /home/xavier/Arc2.zip
> 
> Removing these two lines, rsnapshot runs fine. What's wrong with them?

exclude_file is for situations where you have a file which lists what you
want to exclude.  So for example if you have a file called xyzzy with
contents like this:
	$ cat xyzzy 
	/home/xavier/Arc.zip
	/home/xavier/Arc2.zip
	$
then you could put this in rsnapshot.conf:
	exclude_file	xyzzy


> How am I supposed to exclude a few specific files? I'd like to avoid the
> more complex "exclude" syntax for just to exclude two files if possible.

You could just put this in rsnapshot.conf:
	exclude	/home/xavier/Arc.zip
	exclude	/home/xavier/Arc2.zip
 
-- 
___________________________________________________________________________
 David Keegel <djk@...>  http://www.cyber.com.au/users/djk/
 Cybersource P/L: Linux/Unix Systems Administration Consulting/Contracting

On Fri, Oct 29, 2010 at 09:50:57AM +0200, Xavier wrote:

> If I do 'sudo rsnapshot -t hourly I see that the following line is
> generated:
> 
> > /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
> >     --exclude-from=/home/xavier/Arc.zip --exclude-from=/home/xavier/Arc2.zip \
> >     /home /srv/bak_xrn/shapshots/hourly.0/homes/
> 
> This line will fail as well if it is executed with rsnyc directly with
> the following error:
> 
> > unexpected end of filter rule: -
> > rsync error: syntax or usage error (code 1) at exclude.c(901) [client=3.0.7]
> 
> I could track this down to the two following lines in rsnapshot.conf:
> 
> > exclude_file    /home/xavier/Arc.zip
> > exclude_file    /home/xavier/Arc2.zip
> Removing these two lines, rsnapshot runs fine. What's wrong with them?

--exclude-from's argument should be a text file containing a list of
exclusion rules.

-- 
David Cantrell | A machine for turning tea into grumpiness

We found no search results for "crotchet".  Did you mean "crotch"?

David Keegel wrote:
> exclude_file is for situations where you have a file which lists what you
> want to exclude.  So for example if you have a file called xyzzy with
> contents like this:
> 	$ cat xyzzy 
> 	/home/xavier/Arc.zip
> 	/home/xavier/Arc2.zip
> 	$
> then you could put this in rsnapshot.conf:
> 	exclude_file	xyzzy

Ok I understand my error! I'll use exclude instead (from the examples I
I thought exclude was for patterns, not single files, but I was wrong).

Thanks all for your useful answers!

Xavier

On Fri, Oct 29, 2010 at 01:04:24PM +0200, Xavier wrote:

> Ok I understand my error! I'll use exclude instead (from the examples I
> I thought exclude was for patterns, not single files, but I was wrong).

No, you're right.  You just happen to be able to specify patterns that
match a single file.  For one of my backup targets, I have the following
rsync exclusions:

--exclude=/home/david/*-torrents/

This excludes a bunch of directories in my home dir (all of the contents
of which are legal, I hasten to add! They are mostly videos of technical
talks, and collections of documentation).

--exclude=/home/bob/

This user does his own backups so I don't need to.

--exclude=/web/deps.cpantesters.org/originaldb

There's no need to back up this huge file because it gets downloaded
afresh from another website every night.

Note that the first of those matches several directories, the second
matches one directory (and so excludes everything under that directory),
and the last matches just one file.

-- 
David Cantrell | top google result for "internet beard fetish club"

One person can change the world, but most of the time they shouldn't
    -- Marge Simpson

You definitely want to make sure your exclusions contain full paths 
including the beginning /, for example exclude    /backup. I am not 
familiar with using the wild card parameter (*) within rsnapshot so I 
cannot speak to that.

Linda Hillier
http://computerisms.ca

On 10-10-25 04:04 PM, LuKreme wrote:
> in rsnapshot.conf I have
>
> exclude backup/
>
> on the server I have a drive that mounts at
>
> /backup
>
> I expected that the exclude line would exclude the entire backup drive and all the files under it, but it did not. Since that backup is itself a local rsnapshot, it ended up being massively large since it copied every single instance of the linked files and filled up a 2TB drive over the weekend. I'm about ⅔ of the way through deleting nearly 15,000,000 files.
>
> Yay!
>
> I must be getting confused about something because the exclude lines in the conf file do not seem to be doing what I expect them to do in rsync.
>
> the backup line is:
>
> backup	www::root	www	rsync_long_args=--password-file=/var/rsync.passwd
>
> and there are no include lines in the conf.
>
> The source machine is a freeBSD machine and the machine doing the backup is running OS X Server 10.6.4 with rsnapshot 1.3.1
>
> The full excludes are below:
>
> exclude	SPAM/
> exclude	spool/
> exclude	Maildir/
> exclude	/usr/local/virtual/
> exclude	/usr/ports/
> exclude	*.core
> exclude	tmp/
> exclude	*.mov
> exclude	*.avi
> exclude	*.mp3
> exclude	buops/
> exclude	backup/
> exclude	iPhoto*/
> exclude	tmp/
> exclude	swap*
> exclude	/dev/
> exclude	/tmp/
> exclude	/var/spool/
> exclude	*.bak
>
> (mail is backup-ed separately)

On 25-Oct-2010, at 17:50, Linda Hillier wrote:
> On 10-10-25 04:04 PM, LuKreme wrote:
>> 
>> The full excludes are below:
>> 
>> exclude	SPAM/
>> exclude	spool/
>> exclude	Maildir/
>> exclude	/usr/local/virtual/
>> exclude	/usr/ports/
>> exclude	*.core
>> exclude	tmp/
>> exclude	*.mov
>> exclude	*.avi
>> exclude	*.mp3
>> exclude	buops/
>> exclude	backup/
>> exclude	iPhoto*/
>> exclude	tmp/
>> exclude	swap*
>> exclude	/dev/
>> exclude	/tmp/
>> exclude	/var/spool/
>> exclude	*.bak

> 
> You definitely want to make sure your exclusions contain full paths 
> including the beginning /, for example exclude    /backup. I am not 
> familiar with using the wild card parameter (*) within rsnapshot so I 
> cannot speak to that.
> 
> Linda Hillier
> http://computerisms.ca

On further checking, a LOT of folders that should be excluded are being backed up.

# du -shc hourly.0/mail/usr/home/*/Maildir/ | tail -1
16.4G    total

despite Maildir/ being in the excludes. Similarly, /usr/ports is backed up in the snapshot 

I must be missing something rather basic.

-- 
'Pardon me for living, I'm sure.' NO-ONE GETS PARDONED FOR LIVING.

Hallo, LuKreme,

Du meintest am 25.10.10:

> in rsnapshot.conf I have

> exclude backup/

> on the server I have a drive that mounts at

> /backup

> I expected that the exclude line would exclude the entire backup
> drive and all the files under it, but it did not.


Try

        backup/*

Viele Gruesse!
Helmut

On Tuesday 26 October 2010, LuKreme wrote:
> in rsnapshot.conf I have
> 
> exclude backup/
> 
> on the server I have a drive that mounts at
> 
> /backup
> 
> I expected that the exclude line would exclude the entire backup drive and
> all the files under it, but it did not. Since that backup is itself a
> local rsnapshot, it ended up being massively large since it copied every
> single instance of the linked files and filled up a 2TB drive over the
> weekend. I'm about ⅔ of the way through deleting nearly 15,000,000 files.
> 
> Yay!
> 
> I must be getting confused about something because the exclude lines in the
> conf file do not seem to be doing what I expect them to do in rsync.
> 
> the backup line is:
> 
> backup	www::root	www	rsync_long_args=--password-file=/var/rsync.passwd

I have no experience of using 'bare' rsync, I've only used rsync through ssh, but looking at the manpage I note that the destination always ends in a slash
so
backup<TAB>www::root<TAB>www<TAB>rsync_long_args=--password-file=/var/rsync.passwd
should be
backup<TAB>www::root<TAB>www/<TAB>rsync_long_args=--password-file=/var/rsync.passwd

The source also ends in a backslash in my conf files and there are no examples in the manpage for bare rsync but all the examples have a '/' on the end 
of the source, so I'm not sure how to deal with www::root, it may be that the path in rsyncd.conf should end in '/'

You can also state the rsync_long_args separately, so you could use

rsync_long_args<TAB>--password-file=/var/rsync.passwd

exclude_file<TAB>/usr/local/etc/rsnapshot-excludes
where rsnapshot-excludes contains the exclude lines, makes maintaining things easier.

lockfile<TAB>/var/run/rsnapshot.pid
a pidfile is useful to stop a subsequent instance stumbling over a rsnapshot instance that is taking a long time.

I have excludes like 'backup/' and 'Maildir/' and they work.


> 
> and there are no include lines in the conf.
> 
> The source machine is a freeBSD machine and the machine doing the backup is
> running OS X Server 10.6.4 with rsnapshot 1.3.1
> 
> The full excludes are below:
> 
> exclude	SPAM/
> exclude	spool/
> exclude	Maildir/
> exclude	/usr/local/virtual/
> exclude	/usr/ports/
> exclude	*.core
> exclude	tmp/
> exclude	*.mov
> exclude	*.avi
> exclude	*.mp3
> exclude	buops/
> exclude	backup/
> exclude	iPhoto*/
> exclude	tmp/
> exclude	swap*
> exclude	/dev/
> exclude	/tmp/
> exclude	/var/spool/
> exclude	*.bak
> 
> (mail is backup-ed separately)


-- 
-----------------
Bob Hutchinson
Midwales dot com
-----------------

On Tue, 26 Oct 2010 00:18:22 -0600
LuKreme <kremels@...> wrote:
| 
| On 25-Oct-2010, at 17:50, Linda Hillier wrote:
| > On 10-10-25 04:04 PM, LuKreme wrote:
| > 
| > You definitely want to make sure your exclusions contain full paths 
| > including the beginning /, for example exclude    /backup. I am not 
| > familiar with using the wild card parameter (*) within rsnapshot so I 
| > cannot speak to that.
| > 
| > Linda Hillier
| > http://computerisms.ca
| 
| On further checking, a LOT of folders that should be excluded are being backed up.
| 
| # du -shc hourly.0/mail/usr/home/*/Maildir/ | tail -1
| 16.4G    total
| 
| despite Maildir/ being in the excludes. Similarly, /usr/ports is backed up in the snapshot 
| 
| I must be missing something rather basic.
| 

Could the MailDir already exist in the snapshots.

Exclusions do just that.  Exclude that directory from coping.
that also means it is excluded from deletion if the data already
exists in the backup.

In other words, could those 16.4 Gb just be old data?
Data that is now 'excluded' from both update and deletion.

To test you will need to manually delete those directories from the *.0
directories and then have a look again next time a backup is made.
Or look carfully though backup logs (if you have them).

This problem has caught me out myself, when I was first setting
up a backup system.  Months later that old data was still present
though unchanged, even though it was long deleted in my original
source directories.

In summery:  If you exclude a directory, then you need to remove it.
from at least the last 'snapshot' of each cycle. Or better still, from
all snapshots.



One alternative is to every so often (after exclusion changes)
just do a backup without any hardware linking.  That is just make
a complete and full copy of your home to a empty directory.
This will not contain anything that is now excluded.

When you have done that you can use a 're-link duplicates' type program
to recover the hardlinks between the 'new complete backup' and the
previous backup cycle, so as to recover the disk space.

A program such as 
   http://www.cit.griffith.edu.au/~anthony/software/#linkdups
or one of the many other equivelents will do the task.



Actually I run this program between backup snapshots regardless, as
one cause of hardlink breaking is renaming a directory.  This will
recover the hardlinks between the files.  For a directory of photos
that could recover a LOT of disk space wasted due to duplicated files.




  Anthony Thyssen ( System Programmer )    <A.Thyssen@...>
 --------------------------------------------------------------------------
     I am the terror that flaps in the night...
     Arr Heck...     Lets get dangerous!!!          -- Dark Wing Duck
 --------------------------------------------------------------------------
   Anthony's Castle     http://www.cit.griffith.edu.au/~anthony/

On Oct 26, 2010, at 22:10, Anthony Thyssen <A.Thyssen@...> wrote:

> Could the MailDir already exist in the snapshots.
> 
> Exclusions do just that.  Exclude that directory from coping.
> that also means it is excluded from deletion if the data already
> exists in the backup.
> 
> In other words, could those 16.4 Gb just be old data?
> Data that is now 'excluded' from both update and deletion.

No, this was a new backup location and the first time it was run.

On 23 October 2010 22:26, gilfreund <rsnaphshot-forum@...> wrote:
> Hi,
>
> I am backing up several directories:
> backup&nbsp; user@... remote
> backup&nbsp; user@...; &nbsp; &nbsp; &nbsp;remote
> backup&nbsp; user@...; &nbsp;remote
> backup&nbsp; user@...; &nbsp;remote
>
>
> and I would like to remove the /usr/common from the snapshot structure, so the code, Develop, Develop.old and doc will appear directly under the remote directory. Is that possible?

Not directly tied to rsnapshot, but you could try this hack

        cd /remote/directory
        ln -s usr .
        ln -s common .

This will fail if /usr/common contains directories or files name usr
or common, and you will of course
have these two dummy symlinks present. Probably not the best solution,
just a thought that stroke my mind.

BR Håkon Løvdal

Håkon Løvdal wrote:
> On 23 October 2010 22:26, gilfreund <rsnaphshot-forum < at > backupcentral.com> wrote:
> 
> > Hi,
> > 
> > I am backing up several directories:
> > backup user < at > remote:/usr/common/code remote
> > backup user < at > remote:/usr/common/Develop   remote
> > backup user < at > remote:/usr/common/Develop.old remote
> > backup user < at > remote:/usr/common/doc remote
> > 
> > 
> > and I would like to remove the /usr/common from the snapshot structure, so the code, Develop, Develop.old and doc will appear directly under the remote directory. Is that possible?
> > 
> 
> Not directly tied to rsnapshot, but you could try this hack
> 
> cd /remote/directory
> ln -s usr .
> ln -s common .
> 
> This will fail if /usr/common contains directories or files name usr
> or common, and you will of course
> have these two dummy symlinks present. Probably not the best solution,
> just a thought that stroke my mind.
> 
> BR Håkon Løvdal
> 
> https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss


I found that rsnapshot (might be rsync) does not do too well then the source root is a link.

+----------------------------------------------------------------------
|This was sent by gilf@... via Backup Central.
|Forward SPAM to abuse@....
+----------------------------------------------------------------------

Hallo, bill,

Du meintest am 18.10.10:

> I would like to make one hourly(4:30am)  and one daily backup before
> midnight. What is the best way to set this up in cron and what mods
> would I need to make in rsnapshot.conf.  

a) which Linux distribution
b) how many "daily" backups do you want to hold? One or seven or 365?

Viele Gruesse!
Helmut

On Thu, Oct 7, 2010 at 5:44 AM, Johan Braeken <johan@...> wrote:
> Hi there,
>
> I mount some directories with NFS to be able to back them up. (NAS device).
>
> I noticed rsnapshot refuses to make a backup when the NFS directory is
> not mounted as it thinks there is an error in the configuration file.
> (The directory does not exist.)
>
> Is there a way to force rsnapshot to run, so I have a backup of all
> other servers and directories?

Time to split up your rsnapshot jobs to different files for different
target hosts, perhaps with a shared and "included" master
configuration file. I find this very handy.

> I think rsnapshot should try to run the backup and only give a warning
> when a directory is not available, instead of refusing to run.