Thanks for reading and discussing this, by the way.
Craig Barratt wrote:
Right. This is why I suggested possibly mounting a compressed volume
to handle this case rather than doing so internally. I haven't done
such things before, but I have read about it. :-) I defer to
experience when discussing pitfalls of such a system, naturally.
What I was getting at in this, mainly, is that the Linux operating
system is flush with tools to operate on files and filesystems, but by
essentially creating a separate and incompatible set of files, BackupPC
cannot benefit from that functionality.
Even if the files were named the same, they are compressed and
don't have the right metadata. So while it might appear standard
tools are more useful, there are pitfalls.
There may be another method. One could collect the changes in the same
way they currently are collected, then as a nightly process, or a
post-process after the backup completes, swap the hardlinks between the
full and temporary structure. This allows backups to fail gracefully
without affecting the full backup structure, without needing to fully
rebuild a snapshot of the machine.
[Description of Undo-Stack system for storing changes rather than Redo-Stack]
This has been discussed before. I agree that storing backups in
the way you describe would be better. The most recent snapshot is
always filled, no matter whether it was created with an incemental
or full. Prior snapshots simply store the changes, backwards in
time. Since the most recent snapshot is typically used for browsing
and restore this is the most efficient approach. Older snapshots are
recreated by merging backwards (opposite to the forward merging done
The other big advantage is that the oldest snapshot can be expired
at any time since the snapshot dependencies are in reverse time order.
However, the challenges with implementing this are:
- doing a new backup is problematic. There are two approaches:
- make a complete, new, filled snapshot, and then prune the
prior filled snapshot to just represent the reverse difference.
However, this involves making hardlinks for every file in the
backup, even if just an incremental backup is being done.
This approach would be the easier one to implement, but would
have a performance penalty.
- update the filled snapshot in-place as the backup is done,
and concurrently create a delta snapshot that represents what
will become the prior snapshot. This is more efficient, but
a lot harder to implement. The major design issue is how to
handle a backup failure: the code would have to undo all the
changes to restore the filled snapshot to its original state.
This is true. Originally, I was thinking just a 'full' version of the
host as a directory full of symlinks that is managed by the backup
process, without changing the backup system at all. That could still
be done, but would provide considerably less benefits, and be more
error-prone, I think.
- this would be an incompatible change, and it would be a lot of work
to support both storage methods for backward compatibility. So either
the implementation complexity is high
Again, thanks for reading and taking time for responding. I really
love BackupPC the way it is. With the above changes I think it might
be a little better. :-)