Re: [Dibs-discussion] Congrats & some problems

[Emin M. <em...@al...>]

Christian Stork <dib...@cs...> writes:

> You mean, it keeps the single threaded daemon waiting for 10sec or
> so?

Yes.

> Where exactly lies the problem?  That the daemon does not reply to other
> requests during this time?  

Yes, and this would happen for every peer you couldn't connect to
every time you want to send a message.  For example, imagine you have
6 peers that are behind firewalls, then every time the send_message
command gets issued you would spend about 10 seconds for each peer
waiting for a total of 1 minute.  In the grand scheme of things this
might not seem like a lot, but I think it would be annoying if you
issue a DIBS command and have to wait a minute for something to happen.

Basically, it isn't so much that one couldn't get rid of the
active/passive distinction, but that it would take some thought to
iron out some of these details.  If you feel the active/passive
distinction is annoying enough, please log a feature request on
SourceForge and I'll try to get to it.

> However, I can under stand that DIBS wants to be able to detect and
> exclude the case where both peers are passive, because that's a
> deadlock.  I wouldn't really know how to nicely put this into the UI
> unless we would implement a testing feature or such.

Yes, this is an issue to think about.

> BTW, is my impression correct that you tried to keep all concurrency
> issues a simple as possible by sequentializing everything with a "big"
> database lock? 

Yes.  There is a great saying by Knuth: "Premature optimization is the
death of many a program".  My initial goal is to get DIBS working
while keeping the code reasonably modular.  After the basic features
work, then it will be easier to upgrade various modules (such as the
database) to improve performance.

> Ok, so from DIBS POV this was a misconfiguration, but from a UI POV it
> should at least warn about it and it should simply not store redundant
> information on the same host since the unit of redundancy is "pieces on
> _different_ hosts".

Hmm.  DIBS could warn you that you don't have enough peers for it to
truly take advantage of the number of redundant pieces, but at what
level would you want this warning?  Certainly you don't want to be
warned about this *every* time DIBS tries to store something.  It
could warn you periodically, but keeping track of the last time it's
warned you about something would be a pain.

I suppose my basic philosophy is that every feature or special warning
adds complexity to the code and makes it less robust.  So before
adding something, I'd like to be sure that the complexity is worth
it.  I'm not sure that this issue is important enough to warrant a
complex solution and I don't know of a nice simple solution.  To some
extent this is also how I feel about the active/passive distinction.
Then again, maybe I'm just rationalizing about being too lazy/busy to
have time to fix these things.  :)

> In the above example of 10 peers with redundancy 2 and kbPerFile = 1MB
> the result would be that all files smaller than 1MB are "redundanized"
> (better word?) to three equal-sized copies and these three copies/pieces
> are randomly distributed on different hosts.  Furthermore, each of these
> copies suffices to reconstruct the file.  Is this correct?

Yes.

>> > Even not verifying at all seems to be a better option! 
>> 
>> Yes, that is something worth considering.  The issue with that however
>> is how would you prevent untrusted people from storing stuff with you?
>
> You mean untrusted people sending data from one of your _known_ peers,
> right?

No, I'm untrusted people sending you data period.  Currently, DIBS
authenticates everything through GPG as opposed to from the IP address
they come from.  This is because GPG is more secure than trusting IP
addresses and because many people have dynamic IP addresses.  Thus if
you tried to go with the non-veryifying option, then anyone could
connect to your machine (assuming you are not behind a firewall), and
deliver store requests to DIBS.  Currently, DIBS would just reject
these store messages from non-peers because the GPG signature won't be
valid.  But if you go with the non-veryify option, I'm not sure how to
filter out malicious store requests from non-peers.

Thanks again for your comments,
-Emin