Re: [BackupPC-users] Use case where get duplicate pool files?

[Jeffrey J. Kosowsky <backuppc@ko...>]

Jeffrey J. Kosowsky wrote at about 10:48:08 -0500 on Sunday, January 23, 2011:
 > Looking at MakeFileLink, I realized that it compares the full file
 > length including the first byte header and the potential  rsync digest
 > trailer.
 > 
 > So, if one starts using rsync and then switches to another transport
 > method, won't you get duplicate pool files since new comparisons will
 > compare a (new) compressed file without rsync digest to an old pool
 > file with rsync digest? (these files will have the same partial file
 > md5sum names in the cpool but will have different suffixes)
 > 
 > This could be a common use case if for example one is backing up mixed
 > Linux and Windows machines with common files using rsync, and smb
 > transport, respectively.
 > 
 > Unless I am missing something, this seem like a potentially major
 > source of pool data duplication.
 > 
 > In my jLib.pm, I have written a function zcompare2 that only compares
 > the compressed zlib data between the first byte header and the
 > potential rsync digest trailer. Since the zlib data envelope never
 > changes this gives pool matches even if one file is straight zlib
 > (first byte =0x78) and the other is rsync digest (first byte = 0xd6 or
 > 0xd7). 
 > 
 > I have also btw, written a slightly streamlined version of compare
 > (which I call jcompare) that strips out some unnecessary code for
 > binary files and also works better with weird filenames.
 > 
 > I have combined these changes plus the ones mentioned in my earlier
 > thread on MakeFileLink to create a new jMakeFileLink function that
 > uses jcompare for non-compressed (pool) files and zcompare2 for
 > compressed (cpool) files.
 > 
 > 
 > 

Here is my version of MakeFileLink that:
1. Includes the code efficiency changes outlined in the other thread
2. Uses jcompare/zcompare2 depending on whether file compressed or not
3. Renames before unlinking so it can be undone:

sub jMakeFileLink2
 {
     my($bpc, $name, $d, $newFile, $compress) = @_;
     my($i, $rawFile);

     return -1 unless -f $name;
	 return -2 unless defined($rawFile = $bpc->MD52Path($d, $compress));

	 my $compare = $compress > 0 ? \&zcompare2 : \&jcompare;
     for ( $i = -1 ; ; $i++ ) {
         $rawFile .= "_$i" if $i >= 0;
         if ( -f $rawFile ) {
             if ( (stat(_))[3] < $bpc->{Conf}{HardLinkMax}
                     && !$compare->($name, $rawFile) ) {
				 my $tempname = mktemp("$name.XXXXXXXXXXXXXXXX");
				 return -5 unless rename($name, $tempname);
				 unless(link($rawFile, $name)) {
					 rename($tempname, $name); #Restore
					 return -3;
				 }
                 unlink($tempname);
                 return 1;
             }
         } elsif ( $newFile && (stat($name))[3] == 1 ) {
			 $rawFile =~ m{(.*)/};
             mkpath($1, 0, 0777) unless -d $1 ;
             return -4 unless link($name, $rawFile);
             return 2;
         } else {
             return 0;
         }
     }
 }

Re: [BackupPC-users] Suggested improvements to MakeFileLink routine in Lib.pm

[Jeffrey J. Kosowsky <backuppc@ko...>]

Jeffrey J. Kosowsky wrote at about 01:09:10 -0500 on Sunday, January 23, 2011:
 > It seems there are some extra tests and computation that are not
 > necessary. I will show my suggested changes with +/- style diff
 > Am I missing something?
 > 
 > 
 >  sub MakeFileLink
 >  {
 >      my($bpc, $name, $d, $newFile, $compress) = @_;
 >      my($i, $rawFile);
 >  
 >      return -1 if ( !-f $name );
 > +	 return -2 if ( !defined($rawFile = $bpc->MD52Path($d, $compress)) );
 >      for ( $i = -1 ; ; $i++ ) {
 > -        return -2 if ( !defined($rawFile = $bpc->MD52Path($d, $compress)) );
 >          $rawFile .= "_$i" if ( $i >= 0 );
 >          if ( -f $rawFile ) {
 >              if ( (stat(_))[3] < $bpc->{Conf}{HardLinkMax}
 >                      && !compare($name, $rawFile) ) {
 >                  unlink($name);
 >                  return -3 if ( !link($rawFile, $name) );
 >                  return 1;
 >              }
 > -         } elsif ( $newFile && -f $name && (stat($name))[3] == 1 ) {
 > +         } elsif ( $newFile && -f $name && (stat($_))[3] == 1 ) {
 > -             my($newDir);
 > -             ($newDir = $rawFile) =~ s{(.*)/.*}{$1};
 > +			 $rawfile =~ m{(.*)/}
 > -             mkpath($newDir, 0, 0777) if ( !-d $newDir );
 > +             mkpath($1, 0, 0777) if ( !-d $1 );
 >              return -4 if ( !link($name, $rawFile) );
 >              return 2;
 >          } else {
 >              return 0;
 >          }
 >      }
 >  }
 > 

One more improvement suggestion...
Since the file is unlinked before making the new link there is a
danger that you delete the file but then fail to make the link
resulting in data loss.

A suggested solution would be to rename first so the file can be
restored in case linking fails:

             if ( (stat(_))[3] < $bpc->{Conf}{HardLinkMax}
                     && !$compare->($name, $rawFile) ) {
				 my $tempname = mktemp("$name.XXXXXXXXXXXXXXX");
				 return -5 unless rename($name, $tempname);
				 unless(link($rawFile, $name)) {
					 rename($tempname, $name); #Restore
					 return -3;
				 }
                 unlink($tempname);
                 return 1;
             }

[BackupPC-users] Use case where get duplicate pool files?

[Jeffrey J. Kosowsky <backuppc@ko...>]

Looking at MakeFileLink, I realized that it compares the full file
length including the first byte header and the potential  rsync digest
trailer.

So, if one starts using rsync and then switches to another transport
method, won't you get duplicate pool files since new comparisons will
compare a (new) compressed file without rsync digest to an old pool
file with rsync digest? (these files will have the same partial file
md5sum names in the cpool but will have different suffixes)

This could be a common use case if for example one is backing up mixed
Linux and Windows machines with common files using rsync, and smb
transport, respectively.

Unless I am missing something, this seem like a potentially major
source of pool data duplication.

In my jLib.pm, I have written a function zcompare2 that only compares
the compressed zlib data between the first byte header and the
potential rsync digest trailer. Since the zlib data envelope never
changes this gives pool matches even if one file is straight zlib
(first byte =0x78) and the other is rsync digest (first byte = 0xd6 or
0xd7). 

I have also btw, written a slightly streamlined version of compare
(which I call jcompare) that strips out some unnecessary code for
binary files and also works better with weird filenames.

I have combined these changes plus the ones mentioned in my earlier
thread on MakeFileLink to create a new jMakeFileLink function that
uses jcompare for non-compressed (pool) files and zcompare2 for
compressed (cpool) files.

Re: [BackupPC-users] Suggested improvements to MakeFileLink routine in Lib.pm

[Jeffrey J. Kosowsky <backuppc@ko...>]

Jeffrey J. Kosowsky wrote at about 01:09:10 -0500 on Sunday, January 23, 2011:
 > It seems there are some extra tests and computation that are not
 > necessary. I will show my suggested changes with +/- style diff
 > Am I missing something?
 > 
 > 
 >  sub MakeFileLink
 >  {
 >      my($bpc, $name, $d, $newFile, $compress) = @_;
 >      my($i, $rawFile);
 >  
 >      return -1 if ( !-f $name );
 > +	 return -2 if ( !defined($rawFile = $bpc->MD52Path($d, $compress)) );
 >      for ( $i = -1 ; ; $i++ ) {
 > -        return -2 if ( !defined($rawFile = $bpc->MD52Path($d, $compress)) );
 >          $rawFile .= "_$i" if ( $i >= 0 );
 >          if ( -f $rawFile ) {
 >              if ( (stat(_))[3] < $bpc->{Conf}{HardLinkMax}
 >                      && !compare($name, $rawFile) ) {
 >                  unlink($name);
 >                  return -3 if ( !link($rawFile, $name) );
 >                  return 1;
 >              }
 > -         } elsif ( $newFile && -f $name && (stat($name))[3] == 1 ) {
 > +         } elsif ( $newFile && -f $name && (stat($_))[3] == 1 ) {
 > -             my($newDir);
 > -             ($newDir = $rawFile) =~ s{(.*)/.*}{$1};
 > +			 $rawfile =~ m{(.*)/}
 > -             mkpath($newDir, 0, 0777) if ( !-d $newDir );
 > +             mkpath($1, 0, 0777) if ( !-d $1 );
 >              return -4 if ( !link($name, $rawFile) );
 >              return 2;
 >          } else {
 >              return 0;
 >          }
 >      }
 >  }

I noticed I left the final semicolon off of the line:
+            $rawfile =~ m{(.*)/}

Also, instead of:
 +         } elsif ( $newFile && -f $name && (stat($_))[3] == 1 ) {

one could perhaps better use:
 +         } elsif ( $newFile && (stat($fname))[3] == 1 ) {
since -f $fname was already tested above.

[BackupPC-users] Suggested improvements to MakeFileLink routine in Lib.pm

[Jeffrey J. Kosowsky <backuppc@ko...>]

It seems there are some extra tests and computation that are not
necessary. I will show my suggested changes with +/- style diff
Am I missing something?


 sub MakeFileLink
 {
     my($bpc, $name, $d, $newFile, $compress) = @_;
     my($i, $rawFile);
 
     return -1 if ( !-f $name );
+	 return -2 if ( !defined($rawFile = $bpc->MD52Path($d, $compress)) );
     for ( $i = -1 ; ; $i++ ) {
-        return -2 if ( !defined($rawFile = $bpc->MD52Path($d, $compress)) );
         $rawFile .= "_$i" if ( $i >= 0 );
         if ( -f $rawFile ) {
             if ( (stat(_))[3] < $bpc->{Conf}{HardLinkMax}
                     && !compare($name, $rawFile) ) {
                 unlink($name);
                 return -3 if ( !link($rawFile, $name) );
                 return 1;
             }
-         } elsif ( $newFile && -f $name && (stat($name))[3] == 1 ) {
+         } elsif ( $newFile && -f $name && (stat($_))[3] == 1 ) {
-             my($newDir);
-             ($newDir = $rawFile) =~ s{(.*)/.*}{$1};
+			 $rawfile =~ m{(.*)/}
-             mkpath($newDir, 0, 0777) if ( !-d $newDir );
+             mkpath($1, 0, 0777) if ( !-d $1 );
             return -4 if ( !link($name, $rawFile) );
             return 2;
         } else {
             return 0;
         }
     }
 }

[BackupPC-users] How are characters that are alphanumeric ASCII represented in f-mangled file names and in the attrib file? and attrib files

[Jeffrey J. Kosowsky <backuppc@ko...>]

I know that in the fmangled, file listing, "/" is converted to %2f and
presumably so are things like spaces, linefeeds and other non alphanumeric ASCII characters.

But what about non-ASCII charsets like the ISO's and UTF8?

What about in the attrib file?

Thanks