Re: [Burp-users] include vs. filesystem boundaries

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hello,

Your original message was enough for me to reproduce the bug yesterday, but
thanks for the additional details, it may come in useful.

On Tue, Aug 10, 2021 at 01:26:31PM +0200, Pascal Suter via Burp-users wrote:
> Hi Graham and Myles
> 
> thanks for looking into this.
> 
> 
> regarding my "typo" in test 4 .. i haven't copy pasted that, i just wrote it
> again for the email, so there was probably no typo when I tried it
> originally..
> 
> i failed to mention that this was all done with the ubuntu 20.04 burp
> package which is version 2.2.18.
> 
> here are some instructions how to reproduce this behavior in a docker
> container:
> 
> docker run --rm -ti -v fs2:/fs2 ubuntu:latest
> 
> apt update
> apt install burp
> burp -c /etc/burp/burp-server.conf
> cat > /etc/burp/burp.conf <<EOF
> mode = client
> port = 4971
> status_port = 4972
> server = localhost
> password = abcdefgh
> cname = testclient
> pidfile = /var/run/burp.client.pid
> syslog = 0
> stdout = 1
> progress_counter = 1
> server_can_restore = 0
> cross_all_filesystems=0
> ca_burp_ca = /usr/sbin/burp_ca
> ca_csr_dir = /etc/burp/CA-client
> ssl_cert_ca = /etc/burp/ssl_cert_ca.pem
> ssl_cert = /etc/burp/ssl_cert-client.pem
> ssl_key = /etc/burp/ssl_cert-client.key
> ssl_peer_cn = burpserver
> include = /
> include = /fs2
> exclude = /fs2
> include = /fs2/test
> exclude_fs = sysfs
> exclude_fs = tmpfs
> nobackup = .nobackup
> exclude_comp=bz2
> exclude_comp=gz
> EOF
> burp -a l
> mkdir /fs2/test
> mkdir /fs2/test2
> touch /fs2/test/file
> touch /fs2/test2/file
> burp -a b -v
> burp -a l -b 1 -r "^/fs2"
> 
> output of last command:
> [...]
> Backup: 0000001 2021-08-10 04:30:08 +0000 (deletable)
> With regex: ^/fs2
> /fs2/test
> /fs2/test
> /fs2/test/file
> /fs2/test/file
> [...]
> 
> interestingly, when listing the contents with the "-d" option it does not
> show the duplicates:
> 
> burp -a l -b 1 -d /fs2
> 
> output:
> Backup: 0000001 2021-08-10 04:30:08 +0000 (deletable)
> Listing directory: /fs2
> test
> 
> 
> to reproduce the issue with the latest 2.5.2 release, simply replace the
> "apt install burp" line in the instructions above with the following:
> 
> apt install wget make pkg-config check g++ librsync-dev libz-dev libssl-dev
> uthash-dev
> wget https://github.com/grke/burp/releases/download/2.5.2/burp-2.5.2.tar.bz2
> tar xvf burp-2.5.2.tar.bz2
> cd burp-2.5.2
> ./configure --prefix=/usr --sysconfdir=/etc/burp --localstatedir=/var
> make
> make install
> make install-configs
> 
> and then continue with the rest of the steps, the result is the same.
> 
> hope that helps to reproduce the issue.
> 
> don't forget to remove the fs2 volume from docker afterwards:
> 
> docker volume rm fs2
> 
> 
> this is not urgent at all by the way, my setup is working flawlessly now
> with the right config options in place :) I've got 9 servers and 8 TB of
> data already migrated to burp :) (i did turn off compression tho as it was
> just too slow ;) )
> 
> 
> cheers
> 
> Pascal
> 
> 
> On 8/10/21 5:47 AM, Myles Loosley-Millman wrote:
> > Just a heads up, there's a typo in test #4's parameters (assuming they
> > were copied/pasted), which could potentially impact results:
> > 
> > -----
> > incldue=/
> > include=/boot
> > exclude=/boot
> > include=/boot/test
> > -----
> > 
> > Myles Loosley-Millman - Priority Colo Inc.
> > my...@pr...  -www.prioritycolo.com
> > 1-888-AS-30176 (1-888-273-0176) x301
> > On 2021-08-09 5:50 p.m., Graham Keeling wrote:
> > > On Sat, Aug 07, 2021 at 09:46:13AM +0200, Pascal Suter via Burp-users wrote:
> > > > Hi
> > > > 
> > > > i am in the process of migrating my backup from a self written rsync wrapper
> > > > to burp. I am having some difficulties wrapping my head around how burp
> > > > works with corssing filesystems and the difference between the "include" and
> > > > "cross_filesystem" config options
> > > > 
> > > > for the following let's assume i have a linux installation with these
> > > > mounts:
> > > > 
> > > > /dev/sda1 -> /
> > > > /dev/sda2 -> /boot
> > > > 
> > > > inside my /boot directory i have two sub directories, /boot/grub and
> > > > /boot/test
> > > > 
> > > > in my config on the burp client i have set cross_all_filesystems=0
> > > > 
> > > > i want to backup only / and /boot/test but nothing else from /boot should be
> > > > backed up
> > > > 
> > > > after some playing around, i found the following config to probably be the
> > > > best solution for what i want:
> > > > 
> > > > include=/
> > > > cross_filesystem=/boot
> > > > exclude=/boot
> > > > include=/boot/test
> > > > 
> > > > here is what else i have tested:
> > > > 
> > > > # test 1
> > > > 
> > > > include=/
> > > > include=/boot/test
> > > > 
> > > > -> this does not backup anything in /boot, it just contains the empty
> > > > mountpoint - this was unexpected, coming from rsync, i would have expected
> > > > this to do what i actually wanted :)
> > > > 
> > > > # test 2
> > > > 
> > > > include=/
> > > > cross_filesystem=/boot/test
> > > > 
> > > > -> same as above, emtpy /boot in my backup - that was expected as the
> > > > manpage says that cross_filesystem should be given mountpoints of fileystems
> > > > to cross, so no surprise there
> > > > 
> > > > # test 3
> > > > 
> > > > include=/
> > > > include=/boot
> > > > 
> > > > -> this backed up the entire boot directory and all its contents, so it
> > > > seems to automatically trigger the cross_filesytsem in the background
> > > > 
> > > > # test 4
> > > > 
> > > > incldue=/
> > > > include=/boot
> > > > exclude=/boot
> > > > include=/boot/test
> > > > 
> > > > -> this resulted in /boot/test and its contents being in the backup twice,
> > > > but other than that nothing else from /boot was in there.. so short of
> > > > creating duplicates this did almost the same as my final solution
> > > > 
> > > > so my question is: is my solution above with setting cross_fileystem to the
> > > > mount point, then excluding the mount point and finally including what i
> > > > really want to backup currently the actual way to go or is there a more
> > > > elegant solution to it?
> > > > 
> > > > a change i would request to make this more intitive: At least for me (as a
> > > > long time rsync user i have to admit, so i might be biased here) it seems a
> > > > lot more intuitive to use the configuration of my "test 1" to achieve what i
> > > > wanted here. especially since "test 3" showed, that when I include a
> > > > mountpoint it automatically crosses over to that filesystem, which I totally
> > > > agree on doing.
> > > > 
> > > > Now if that change was made, and the syntax as used in "test 1" implied
> > > > automatically that crossing into the /boot filesystem was allowed, I  think
> > > > there might be no reason to keep the "cross_filesystem" config option any
> > > > longer as it would just be a sort of a limited synonym for "include" that
> > > > only works on mountpoints :) .. frankly i think this would make things a lot
> > > > easier to comprehend
> > > > 
> > > > the behavior in "test 4" may be expected for someone who knows the internals
> > > > of burp but from a newbie user point of view this is a bug. I see no reason
> > > > to ever have the same file contained twice i the same backup.
> > > > 
> > > > 
> > > > with all that said, thanks for writing, maintaining and most importantly
> > > > sharing burp! it looks like an awesome tool to me :) I have had burp on the
> > > > radar for a while, sort of used it a long time ago to try to backup some
> > > > windows notebooks of friends and family but got lazy on maintaining the
> > > > setup after a while.. now i am "back" moving over from my own rsync based
> > > > backup script due to the ever increasing danger of ransomware worms and what
> > > > they can do to other types of backups. in case you are interested in a
> > > > long(er) read and my thought process that brought me over to burp, have a
> > > > look athttp://wiki.psuter.ch/doku.php?id=rethinking_my_backup_strategy  it
> > > > really shows some areas where burp was ahead of many other solutions back
> > > > when you started developing it.
> > > Hello,
> > > 
> > > Thank you for your kind words.
> > > 
> > > Yes, "test 4" is demonstrating a bug that I didn't know about. I will try to
> > > fix that ASAP.
> > > 
> > > Once I have done that, I will think about making "test 1" do what you expect.
> > > 
> > > 
> > > _______________________________________________
> > > Burp-users mailing list
> > > Bur...@li...
> > > https://lists.sourceforge.net/lists/listinfo/burp-users
> > 
> > 
> > 
> > _______________________________________________
> > Burp-users mailing list
> > Bur...@li...
> > https://lists.sourceforge.net/lists/listinfo/burp-users

> _______________________________________________
> Burp-users mailing list
> Bur...@li...
> https://lists.sourceforge.net/lists/listinfo/burp-users