[BackupPC-users] Block-level rsync-like hashing dd?

[Saturn2888 <bac...@ba...>]

Was the conclusion that you could or could not rsync a logical volume? And even if so, what are the speeds to run checksums on a device and compare it with another device? I've got an iSCSI target setup and had a few ideas, all of which failed. Here are my findings on various methodologies or possible solutions to backing up BackupPC's data.

:: lvm2 ::

I was hoping to do something like a pvmove but copy the logical volume; that one failed because there's no way to do it.

:: rsync ::

I thought about rsyncing devices using "--copy-devices", but you'd have to get a patched rsync to do it and word on the street is, it only works well for devices 2GB and under. Copying an lvm snapshot using rsync would still be just as bad so this, also, won't work.

The absolute only way to get rsync to play nice is to break it down into multiple operations and save the checksum table to disk instead of memory and allow it to be usable like that or else you will lose all the hard linking. Either this would be another program completely like Unison, or it'd require some heavy additions to rsync's codebase. It's just not going to happen.

Now, if you had a filesystem that supported block-level dedup on the backup end of all of this, it would be more feasible to depend on that along side rsync provided it worked like I'm assuming.

:: ddsnap ::

Then I thought maybe I could find something else when I came upon an age-old ddsnap project harking back to the now-defunct Zumastor. Since this project's been out of commission since Hardy, I believe it's a no-go.

:: mdadm ::

I could try adding the drive into the Linux Software RAID1 mirror, but that would slow down all transfers because if the iSCSI target I'm writing to are being written to, that will slow down writes of all these really small files with hardlinks on top of that. No matter what, I can't imagine any sort of non-block-level sync associated with RAID1 being anywhere useful over Ethernet when BackupPC's being used. And what if I restart the iSCSI target machine etc? Doesn't seem workable.

:: DRBD ::

Here comes DRBD. I absolutely have no clue how to use this and usage examples online are scarce. The an up-side to DRBD is that it's going to act like a RAID1 mirror, but that it works at the block level which will speed up transfers greatly. The next good thing about it is it's able to work asynchronously. Right there, I have a capable solution with may or may not work for me. From what I've seen, it's really designed for high availability clustering and requires both sides to have DRBD for it to be useful. Since I'm backing this up to an embedded FreeBSD 7.3 box which does not have DRBD, the iSCSI target comes to mind as a capable solution; yet, it almost seems impossible because this machine has to be both the sending and receiving end. I don't quite know yet if this will really be worthwhile or not or if I can even get it working.

:: Final Thoughts ::

I hope that helps anyone trying to figure out how insane this is. My last idea is to either way for BackupPC4 which uses a database or write my own. I'm backing on the latter because that's the quickest way to get something worthwhile accomplished.

+----------------------------------------------------------------------
|This was sent by Sat...@gm... via Backup Central.
|Forward SPAM to ab...@ba....
+----------------------------------------------------------------------