Thanks for reading and discussing this, by the way.

Craig Barratt wrote:
Jason writes:

  
What I was getting at in this, mainly, is that the Linux operating 
system is flush with tools to operate on files and filesystems, but by 
essentially creating a separate and incompatible set of files, BackupPC 
cannot benefit from that functionality.
    

Even if the files were named the same, they are compressed and
don't have the right metadata.  So while it might appear standard
tools are more useful, there are pitfalls.
  
Right.  This is why I suggested possibly mounting a compressed volume to handle this case rather than doing so internally.  I haven't done such things before, but I have read about it.  :-)  I defer to experience when discussing pitfalls of such a system, naturally.

[Description of Undo-Stack system for storing changes rather than Redo-Stack] 
    

This has been discussed before.  I agree that storing backups in
the way you describe would be better.  The most recent snapshot is
always filled, no matter whether it was created with an incemental
or full.  Prior snapshots simply store the changes, backwards in
time.  Since the most recent snapshot is typically used for browsing
and restore this is the most efficient approach.  Older snapshots are
recreated by merging backwards (opposite to the forward merging done
currently).

The other big advantage is that the oldest snapshot can be expired
at any time since the snapshot dependencies are in reverse time order.

However, the challenges with implementing this are:

 - doing a new backup is problematic.  There are two approaches:

     - make a complete, new, filled snapshot, and then prune the
       prior filled snapshot to just represent the reverse difference.
       However, this involves making hardlinks for every file in the
       backup, even if just an incremental backup is being done.
       This approach would be the easier one to implement, but would
       have a performance penalty.

     - update the filled snapshot in-place as the backup is done,
       and concurrently create a delta snapshot that represents what
       will become the prior snapshot.  This is more efficient, but
       a lot harder to implement.  The major design issue is how to
       handle a backup failure: the code would have to undo all the
       changes to restore the filled snapshot to its original state.
  
There may be another method.  One could collect the changes in the same way they currently are collected, then as a nightly process, or a post-process after the backup completes, swap the hardlinks between the full and temporary structure.  This allows backups to fail gracefully without affecting the full backup structure, without needing to fully rebuild a snapshot of the machine.

 - this would be an incompatible change, and it would be a lot of work
   to support both storage methods for backward compatibility.  So either
   the implementation complexity is high
  
This is true.  Originally, I was thinking just a 'full' version of the host as a directory full of symlinks that is managed by the backup process, without changing the backup system at all.  That could still be done, but would provide considerably less benefits, and be more error-prone, I think.

Again, thanks for reading and taking time for responding.  I really love BackupPC the way it is.  With the above changes I think it might be a little better.  :-)

JH