[BackupPC-devel] BackupPC 4.0 features - attribute file and backup storage

[Craig B. <cba...@us...>]

The next topic is the attribute file format and how backups
are stored.

In 4.x the only information that appears in a backup tree
(ie: a single backup stored below $TOPDIR/pc/HOST/NNN) are
the attribute files, one per directory.

In contrast, a filled backup in 3.x includes a complete set
of files using hardlinks, in addition to the attribute file.
Also in 3.x a full directory tree had to be created even
for incremental backups.

In 4.x it is no longer necessary for a complete backup tree of
directories to be created.  Only sufficient directories to store
the attribute files necessary for the reverse time deltas are
needed.

In 4.x a new attribute file format is used.  A new magic number
identifies the new format.  For each file in the directory,
the 4.x attributes additionally stores:

 - the file's digest (ie: MD5 full file digest, with possible
   extension for collisions)
   
 - an inode number (inode)

 - an inode link count (nlinks)

 - extended attributes (xattr)

The inode number is generated locally during the backup process
(ie: it is not related to the inode number on the client).

The 3.x BPC_FTYPE_HARDLINK file type has been eliminated (other
than to support legacy 3.x backups).  Client hardlinks are stored
in a more faithful manner that allows the inode reference count
to be accurate, and also allows attribute changes on any of the
file's hardlink instances to correctly be mirrored among all the
instances.

For client hardlinks the inode link count in the attribute file is
greater than 1.  In that case, the inode number (locally generated on
the server, not the real client inode) is used to access a secondary
attribute file that stores the real attributes of hardlinked files.
These attribute files are indexed by inode number.  This provides the
right semantics - multiple hardlinks really do refer to a single file
and corresponding attributes.

Inodes attributes (for client hardlinks) are stored in an additional
tree of directories below each backup directory.  The attribute hash key
is the inode number.  128 directories are used to store up to 128 attrib
files.  The lower 14 bits of the inode number are used to determine the
directory (bits 13 to 7) and the file name within that directory
(bits 0-6).  These two 7 bit values are encoded in hex (XX and YY):

    $TOPDIR/pc/HOST/NNN/inode/XX/attribYY.

where NNN is the backup number.

Craig