Menu
▾
▴
Re: [BackupPC-users] incremental doesn't stay within filter
|
From: Mark S. <ma...@al...> - 2007-05-22 07:54:23
|
Holger Parplies wrote:
>Hi,
>
>I'll have to rearrange some of what I quote for it to make more sense. No
>mis-quoting intended.
>
>Mark Sopuch wrote on 21.05.2007 at 00:09:15 [Re: [BackupPC-users] [BackupPC-devel] incremental doesn't stay within filter]:
>
>
>>Mark Sopuch wrote:
>>
>>
>>>Holger Parplies wrote:
>>>
>>>
>>>>Mark Sopuch wrote on 19.05.2007 at 23:58:39 [[BackupPC-devel] incremental doesn't stay within filter]:
>>>>
>>>>
>
>
>
>>Make of it what you will but the change from c to c$ in the filter
>>combined with the smbclient commands I supplied earlier ran as follows
>>(first the full then the incrmental) :
>>
>>Running: /usr/bin/smbclient \\\\gromit\\C\$ -U AL\\marks! -E -N -d 1 -c tarmode\ full -Tc - /nucoda
>>Running: /usr/bin/smbclient \\\\gromit\\C\$ -U AL\\backuppc -E -N -d 1 -c tarmode\ full -TcN /tomato1/BackupPC/DATA/pc/gromit/timeStamp.level0 -
>>
>>
>
>The differences between the full and the incremental commands are:
>1. different $userName
>2. incremental misses $fileList
>
>This can really only mean that the SmbClientIncrCmd in fact used does *not*
>include a '$fileList' argument (which would explain exactly what you are
>seeing). Maybe one is from the global configuration file and the other from
>the host specific configuration file? That would mean that either only one
>of them is present in the host specific file or that one is misspellt (eg.
>"$Conf {SbmClientIncrCmd} = ...") or something.
>Maybe you could post the output of
>
> egrep '^ *\$Conf *\{Smb' config.pl pc/gromit.pl | grep -v Passwd
>
>
super-user@@tomato - /etc/BackupPC >egrep '^ *\$Conf *\{Smb' config.pl
pc/gromit.pl | grep -v Passwd
config.pl:$Conf{SmbShareName} = [
config.pl:$Conf{SmbShareUserName} = 'AL\\backuppc';
config.pl:$Conf{SmbClientPath} = '/usr/bin/smbclient';
config.pl:$Conf{SmbClientFullCmd} = '$smbClientPath
\\\\$host\\$shareName $I_option -U $userName -E -N -d 1 -c tarmode\\
full -Tc$X_option - $fileList';
config.pl:$Conf{SmbClientIncrCmd} = '$smbClientPath
\\\\$host\\$shareName $I_option -U $userName -E -N -d 1 -c tarmode\\
full -TcN$X_option $timeStampFile - $fileList';
config.pl:$Conf{SmbClientRestoreCmd} = '$smbClientPath
\\\\$host\\$shareName $I_option -U $userName -E -N -d 1 -c tarmode\\
full -Tx -';
pc/gromit.pl:$Conf{SmbShareUserName} = 'AL\\backuppc';
>or
> pcregrep '^\s*\$Conf\s*\{Smb(?!SharePasswd)' config.pl pc/gromit.pl
>
>
super-user@@tomato - /etc/BackupPC >pcregrep '^\s*\$Conf\s*\{Smb(?)'
config.pl pc/gromit.pl
config.pl:$Conf{SmbShareName} = [
config.pl:$Conf{SmbShareUserName} = 'AL\\backuppc';
config.pl:$Conf{SmbSharePasswd} = 'XXXXXXX';
config.pl:$Conf{SmbClientPath} = '/usr/bin/smbclient';
config.pl:$Conf{SmbClientFullCmd} = '$smbClientPath
\\\\$host\\$shareName $I_option -U $userName -E -N -d 1 -c tarmode\\
full -Tc$X_option - $fileList';
config.pl:$Conf{SmbClientIncrCmd} = '$smbClientPath
\\\\$host\\$shareName $I_option -U $userName -E -N -d 1 -c tarmode\\
full -TcN$X_option $timeStampFile - $fileList';
config.pl:$Conf{SmbClientRestoreCmd} = '$smbClientPath
\\\\$host\\$shareName $I_option -U $userName -E -N -d 1 -c tarmode\\
full -Tx -';
pc/gromit.pl:$Conf{SmbSharePasswd} = 'XXXXXXX';
pc/gromit.pl:$Conf{SmbShareUserName} = 'AL\\backuppc';
>(from $ConfDir).
>
>
>
>>As I said, I have not tuned the smbclient commands away from the defaults
>>that were installed.
>>
>>
>
>Well, someone seems to. Maybe you changed them for testing and forgot to
>change something back? Did you use the web interface or did you edit the
>files by hand?
>
>
Global Full: $smbClientPath \\$host\$shareName $I_option -U $userName -E
-N -d 1 -c tarmode\ full -Tc$X_option - $fileList
Global Incr: $smbClientPath \\$host\$shareName $I_option -U $userName -E
-N -d 1 -c tarmode\ full -TcN$X_option $timeStampFile - $fileList
Gromit Full: $smbClientPath \\$host\$shareName $I_option -U $userName -E
-N -d 1 -c tarmode\ full -Tc$X_option - $fileList
Gromit Incr: $smbClientPath \\$host\$shareName $I_option -U $userName -E
-N -d 1 -c tarmode\ full -TcN$X_option $timeStampFile - $fileList
>
>
>>I assumed it is not necessary and they are ready for use with filters out
>>of the box.
>>
>>
>
>They certainly should be.
>
>
>
>>You can see above what the commands interpolate to run as.
>>
>>
>
>
>
>Yes, and that's what makes me wonder. The user name should most obviously
>not be interpolated differently, but also the file list is computed by the
>same piece of code for full and incremental backups. I can't imagine how it
>could be empty in one case and correct in the other. In fact, it's not
>correct in the other case. It appears to be hardcoded into the command
>(without the trailing '/*'), much the same as the user name.
>
>
Username change between backups is my doing. I changed it to instead of
running as my elevated account to use the dedicated backup account. I
also ran the level 1 incremental several times and pruned the backup
data and logs in between each attempt. I did this to easily see if the
incremental polluted the browse tree with stuff from outside of the
filter. I guess I could have just used the Xfer logs alone for that.
It'd be great to see a safer version of a backup expunger in BackupPC
than my hand in a shell. I suspect my actions could have implication
into deduplicated data masters in that backup becoming unavailable would
it? (I haven't studied the dedupe aspect yet)
>Hmm. What is your $Conf {ClientCharset} set to?
>
>
$Conf{ClientCharset} = ''; So I guess that makes it utf8 as is the default.
>
>
>>>It is just that everything works so beautifully
>>>out of the box and when the filter worked for full backup but only
>>>partially for incremental it threw me to confusion. When I say
>>>partially working for incremental I mean that the data from within the
>>>include filter get backed up but I get attrib stubs for everything
>>>else on the c share.
>>>
>>>
>
>The explanation from the commands actually run is that full backups work
>correctly and incrementals are run without the filter setting. As only files
>modified since the time of the last full backup are transfered (but all
>directories), it might appear that the filter is "partially" applied although
>in fact it isn't applied at all.
>
>
That is a possibility and due to some system files having > 0 byte value
outside of the filter Xferlogged I'd say you are right.
>
>
>>>>>$Conf{BackupFilesOnly} = {
>>>>> 'c' => [
>>>>> '/nucoda/*'
>>>>> ]
>>>>>};
>>>>>
>>>>>
>>>[...]
>>>Okay, now that I have pasted this in I have a funny feeling the c
>>>should be a c$ (seeing the SmbShareName = c$).
>>>
>>>
>
>Well, yes, the two need to match for the filter settings to apply. That
>requirement is identical for full and incremental backups, though.
>
>
I was surprised I had to prime the $Conf{BackupFilesOnly}by hand in the
first place so I guess I felt complacently good about myself having
completed that at that point and missed the dollar and case sense!
>
>
>>>Sorry. Here it is for distro, apache+mp2, backuppc, smbclient ...
>>>
>>>Linux tomato 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686
>>>i686 i386 GNU/Linux
>>>
>>>
>
>For people not using the same Linux distribution it's not really obvious
>which one that kernel is from :-).
>
>
Fair enough. RHES4v3.
>
>
>>>>>Assuming filters apply equally to both full and incremental in
>>>>>BackupPC [...]
>>>>>
>>>>>
>>>>Filters don't "apply in BackupPC". Filters are used to construct
>>>>arguments
>>>>to the commands run to gather the data - smbclient in your case.
>>>>
>>>>
>>>Sure, I guess what I meant was that I was assuming that include and
>>>exclude list processing by BackupPC use the same filter definitions ie.
>>>
>>>$Conf{BackupFilesOnly}
>>>
>>>
>
>My point was that BackupPC does not actually apply any filters.
>
I see that now.
>It
>constructs arguments that it passes to external programs such as smbclient,
>tar or rsync. The code fragment that generates these arguments does not
>differentiate between full and incremental backups (for SMB anyway), so the
>arguments generated are identical in both cases.
>Presuming smbclient had a bug causing it to transfer all directories in the
>incremental case, BackupPC would not notice or try to correct that, although
>this would in theory be possible to implement. Apparently, smbclient has no
>such bug but is simply not passed the fileList arguments for the incremental
>backup.
>
>Actually, my guess is that you have
>
> $Conf {SmbShareName} = 'C$';
> $Conf {BackupFilesOnly} = { 'c$' => [ '/nucoda/*' ] };
> $Conf {SmbClientFullCmd} = '... -U AL\\marks! ... -Tc$X_option - /nucoda $fileList';
> $Conf {SmbClientIncrCmd} = '... -U $userName ... -Tc$X_option $timeStampFile - $fileList';
>
>which would explain why $fileList is empty: 'C$' != 'c$'.
>
>
I will probably need to confirm for myself the combinations of c, C, c$,
C$ for support to actually see where BackupPC or indeed smbclient get
fussy - or not. You probably hit the answer here though and by a stroke
of bad luck it appears that the host I could conveniently test this on
(gromit) is down for maintenance all of today. I'll try some other host
I can get my hands on.
>Of course, if you had initially posted the contents of your config files,
>it would have saved me considerable effort, and we could probably have
>resolved this days ago.
>
>
That is probably true. You have gone above the call of duty and your
effort is appreciated.
Spotting the empty $filelist command arg value when filters are expected
to be applied and finding a target share and filter share of different
names creates the situation. Is that the main lesson here?
Thanks again.
>Regards,
>Holger
>
>
|