Re: [BackupPC-users] Block-level rsync-like hashing dd?

[Holger P. <wb...@pa...>]

Hi,

Saturn2888 wrote on 2011-04-12 19:40:50 -0700 [[BackupPC-users]  Block-level rsync-like hashing dd?]:
> My goodness that's a lot of replies; although, almost all of them are from
> a post I made a while back which I've elaborated on at least 3 times since.

that is probably due to the fact that you are using inferior forum software to
post to this mailing list, which ends up creating a new unrelated thread each
time *you* get back into the discussion. I've considered doing the equivalent
for you for demonstration purposes - removing reference headers and perhaps
changing the subject for good measure. You would have found this message in
a different "topic" on your board - or, more probably, you wouldn't have found
it at all. I still like the idea, though. Please just *try* to imagine the
concept of following a discussion intermixed throughout three or four
"topics" - even if they happen to all have the same title. That is, basically,
what you are asking us to do.

We have it on good confidence from the board maintainer that he can't fix this
issue, because he is doing everything he does for altruistic reasons (I
believe that was his net reasoning).

You, however, *could* fix the problem, as has been pointed out more than once,
as I believe. If you refuse to do so, you are implicitly asking for exactly
what you get: replies to posts you consider outdated. So don't complain. It
was *your* choice.

And I need to quote Timothy, because I enjoyed his usage of the word
"God-forsaken" so very much :-). Lesson learnt: it pays to read threads
you considered simply skipping, provided the right people are contributing.

> The main point here that I'm trying to get across, at least for my own
> setup, is that I cannot handle dd'ing 1TB/day.

Well, that's simple: then don't. Problem solved.

The main point you *should* probably be trying to get across (and you might
have - I have obviously missed at least one segment of the hyper-thread) is,
what *exactly* you are trying to *achieve*, *under which constraints*. It's
obviously some variation of the old story of pool replication. I'm sure you've
read all the hundreds of previous threads discussing this matter, and your
case is surely significantly enough different to warrant reiterating the
matter yet once more.

If it turns out that what you want *can't be done*, then asking if we can
make an exception for you won't help.

> [...] The reason I am advocating DRBD [...]

It's really your decision. You don't have to persuade us. You *won't* persuade
us to agree with you if you are wrong (I don't know if you are).

If your question is actually, "can DRBD do for me what I want it to", then
*ask* *that* *question* (and that really belongs in capitals!), including what
you want it to do; don't confuse us with a general discussion that most of us
tend to be fed up with, because we've had it literally hundreds of times.

> is because I have it going to an iSCSI target on a machine with ZFS which
> snapshots the pool. The key there is I'm taking snapshots of it so even if
> it corrupts, I'm fine. All I'd have to do is mount the .img as EXT4 in an
> iSCSI target and point a client to that target to retrieve the files.

If you want an honest opinion, that sounds like a few random sentences to me,
fabricated to include certain keywords. To me, it doesn't make any sense
whatsoever. Maybe it's just my lack of understanding, but I don't see how you
can "mount the .img as EXT4 in an iSCSI target". An iSCSI target is a block
device, an opaque array of bytes, exported to a client to use however he
wants. Your iSCSI device might allow you to export image files as iSCSI
devices, but then you wouldn't mount anything server-side (much less "as
EXT4"). Are you sure you *understand* the concepts you are talking about?

> [...] I don't see pool corruption a big deal because I've never experienced
> anything like that.

What exactly are you protecting against if not pool corruption? Theft?
It's perfectly fine to protect (or not protect) against any particular
threat. It's just unclear whether you are achieving what you think you
are, so it might help to explain what you are trying to achieve, presuming
you want any advice.

> [...] just need a method of getting only the changes over there without
> abusing the available bandwidth.

The only thing I can think of to add to what has already been said is that you
have bandwidth at several points here:

- bandwidth from source disk
- bandwidth to destination disk
- bandwidth of data your CPU can handle (i.e. checksum, md4/md5, compression,
  or whatever)
- bandwidth of source NIC
- bandwidth of destination NIC
- bandwidth of network link.

You sound like you are worried about the network link and nothing else. Is
that correct? Is that sensible? All of those resources are limited and may
or may not be precious to you. Depending on which are, the comments differ
greatly (up to the point of being contradictory). It is probably frustrating
for you to read answers worried about a resource you are not worried about,
but the only way to avoid that is to be more clear about what you want. We
can't read your mind.

Hope that helps.

Regards,
Holger