Re: [Burp-users] include vs. filesystem boundaries

[Pascal S. <d-s...@ps...>]

Hi Graham and Myles

thanks for looking into this.


regarding my "typo" in test 4 .. i haven't copy pasted that, i just 
wrote it again for the email, so there was probably no typo when I tried 
it originally..

i failed to mention that this was all done with the ubuntu 20.04 burp 
package which is version 2.2.18.

here are some instructions how to reproduce this behavior in a docker 
container:

docker run --rm -ti -v fs2:/fs2 ubuntu:latest

apt update
apt install burp
burp -c /etc/burp/burp-server.conf
cat > /etc/burp/burp.conf <<EOF
mode = client
port = 4971
status_port = 4972
server = localhost
password = abcdefgh
cname = testclient
pidfile = /var/run/burp.client.pid
syslog = 0
stdout = 1
progress_counter = 1
server_can_restore = 0
cross_all_filesystems=0
ca_burp_ca = /usr/sbin/burp_ca
ca_csr_dir = /etc/burp/CA-client
ssl_cert_ca = /etc/burp/ssl_cert_ca.pem
ssl_cert = /etc/burp/ssl_cert-client.pem
ssl_key = /etc/burp/ssl_cert-client.key
ssl_peer_cn = burpserver
include = /
include = /fs2
exclude = /fs2
include = /fs2/test
exclude_fs = sysfs
exclude_fs = tmpfs
nobackup = .nobackup
exclude_comp=bz2
exclude_comp=gz
EOF
burp -a l
mkdir /fs2/test
mkdir /fs2/test2
touch /fs2/test/file
touch /fs2/test2/file
burp -a b -v
burp -a l -b 1 -r "^/fs2"

output of last command:
[...]
Backup: 0000001 2021-08-10 04:30:08 +0000 (deletable)
With regex: ^/fs2
/fs2/test
/fs2/test
/fs2/test/file
/fs2/test/file
[...]

interestingly, when listing the contents with the "-d" option it does 
not show the duplicates:

burp -a l -b 1 -d /fs2

output:
Backup: 0000001 2021-08-10 04:30:08 +0000 (deletable)
Listing directory: /fs2
test


to reproduce the issue with the latest 2.5.2 release, simply replace the 
"apt install burp" line in the instructions above with the following:

apt install wget make pkg-config check g++ librsync-dev libz-dev 
libssl-dev uthash-dev
wget https://github.com/grke/burp/releases/download/2.5.2/burp-2.5.2.tar.bz2
tar xvf burp-2.5.2.tar.bz2
cd burp-2.5.2
./configure --prefix=/usr --sysconfdir=/etc/burp --localstatedir=/var
make
make install
make install-configs

and then continue with the rest of the steps, the result is the same.

hope that helps to reproduce the issue.

don't forget to remove the fs2 volume from docker afterwards:

docker volume rm fs2


this is not urgent at all by the way, my setup is working flawlessly now 
with the right config options in place :) I've got 9 servers and 8 TB of 
data already migrated to burp :) (i did turn off compression tho as it 
was just too slow ;) )


cheers

Pascal


On 8/10/21 5:47 AM, Myles Loosley-Millman wrote:
> Just a heads up, there's a typo in test #4's parameters (assuming they 
> were copied/pasted), which could potentially impact results:
>
> -----
> incldue=/
> include=/boot
> exclude=/boot
> include=/boot/test
> -----
>
> Myles Loosley-Millman - Priority Colo Inc.
> my...@pr...  -www.prioritycolo.com
> 1-888-AS-30176 (1-888-273-0176) x301
> On 2021-08-09 5:50 p.m., Graham Keeling wrote:
>> On Sat, Aug 07, 2021 at 09:46:13AM +0200, Pascal Suter via Burp-users wrote:
>>> Hi
>>>
>>> i am in the process of migrating my backup from a self written rsync wrapper
>>> to burp. I am having some difficulties wrapping my head around how burp
>>> works with corssing filesystems and the difference between the "include" and
>>> "cross_filesystem" config options
>>>
>>> for the following let's assume i have a linux installation with these
>>> mounts:
>>>
>>> /dev/sda1 -> /
>>> /dev/sda2 -> /boot
>>>
>>> inside my /boot directory i have two sub directories, /boot/grub and
>>> /boot/test
>>>
>>> in my config on the burp client i have set cross_all_filesystems=0
>>>
>>> i want to backup only / and /boot/test but nothing else from /boot should be
>>> backed up
>>>
>>> after some playing around, i found the following config to probably be the
>>> best solution for what i want:
>>>
>>> include=/
>>> cross_filesystem=/boot
>>> exclude=/boot
>>> include=/boot/test
>>>
>>> here is what else i have tested:
>>>
>>> # test 1
>>>
>>> include=/
>>> include=/boot/test
>>>
>>> -> this does not backup anything in /boot, it just contains the empty
>>> mountpoint - this was unexpected, coming from rsync, i would have expected
>>> this to do what i actually wanted :)
>>>
>>> # test 2
>>>
>>> include=/
>>> cross_filesystem=/boot/test
>>>
>>> -> same as above, emtpy /boot in my backup - that was expected as the
>>> manpage says that cross_filesystem should be given mountpoints of fileystems
>>> to cross, so no surprise there
>>>
>>> # test 3
>>>
>>> include=/
>>> include=/boot
>>>
>>> -> this backed up the entire boot directory and all its contents, so it
>>> seems to automatically trigger the cross_filesytsem in the background
>>>
>>> # test 4
>>>
>>> incldue=/
>>> include=/boot
>>> exclude=/boot
>>> include=/boot/test
>>>
>>> -> this resulted in /boot/test and its contents being in the backup twice,
>>> but other than that nothing else from /boot was in there.. so short of
>>> creating duplicates this did almost the same as my final solution
>>>
>>> so my question is: is my solution above with setting cross_fileystem to the
>>> mount point, then excluding the mount point and finally including what i
>>> really want to backup currently the actual way to go or is there a more
>>> elegant solution to it?
>>>
>>> a change i would request to make this more intitive: At least for me (as a
>>> long time rsync user i have to admit, so i might be biased here) it seems a
>>> lot more intuitive to use the configuration of my "test 1" to achieve what i
>>> wanted here. especially since "test 3" showed, that when I include a
>>> mountpoint it automatically crosses over to that filesystem, which I totally
>>> agree on doing.
>>>
>>> Now if that change was made, and the syntax as used in "test 1" implied
>>> automatically that crossing into the /boot filesystem was allowed, I  think
>>> there might be no reason to keep the "cross_filesystem" config option any
>>> longer as it would just be a sort of a limited synonym for "include" that
>>> only works on mountpoints :) .. frankly i think this would make things a lot
>>> easier to comprehend
>>>
>>> the behavior in "test 4" may be expected for someone who knows the internals
>>> of burp but from a newbie user point of view this is a bug. I see no reason
>>> to ever have the same file contained twice i the same backup.
>>>
>>>
>>> with all that said, thanks for writing, maintaining and most importantly
>>> sharing burp! it looks like an awesome tool to me :) I have had burp on the
>>> radar for a while, sort of used it a long time ago to try to backup some
>>> windows notebooks of friends and family but got lazy on maintaining the
>>> setup after a while.. now i am "back" moving over from my own rsync based
>>> backup script due to the ever increasing danger of ransomware worms and what
>>> they can do to other types of backups. in case you are interested in a
>>> long(er) read and my thought process that brought me over to burp, have a
>>> look athttp://wiki.psuter.ch/doku.php?id=rethinking_my_backup_strategy  it
>>> really shows some areas where burp was ahead of many other solutions back
>>> when you started developing it.
>> Hello,
>>
>> Thank you for your kind words.
>>
>> Yes, "test 4" is demonstrating a bug that I didn't know about. I will try to
>> fix that ASAP.
>>
>> Once I have done that, I will think about making "test 1" do what you expect.
>>
>>
>> _______________________________________________
>> Burp-users mailing list
>> Bur...@li...
>> https://lists.sourceforge.net/lists/listinfo/burp-users
>
>
>
> _______________________________________________
> Burp-users mailing list
> Bur...@li...
> https://lists.sourceforge.net/lists/listinfo/burp-users