Re: [Seeks-users] DHT: blocking vs non-blocking

[Loic D. <lo...@da...>]

On 01/16/2011 10:34 AM, Emmanuel Benazera wrote:
> Hi Loic,
>
> in my opinion, non-blocking I/O allow for better performance, especially
> on DHT nodes supporting a few hundreds of virtual nodes.
>
>   
I don't see any reason to run a DHT with more than 32 virtual nodes (the
default). If more virtual nodes are needed for some reason, it's
straightforward to run more processes on the same machine. Say you want
320 virtual nodes, you can just run 10 processes and I don't see why it
would be significantly less efficient than running one process with 320
virtual nodes.

Could you be more specific about why you think it would be bad for
performances to have blocking I/O ?
> The reason I see is that in blocking mode the stabilizer iterates among
> elements to stabilize (finger table, successors). Thus every blocking
> operation delays the whole queue of waiting elements.
>   
Yes. But if you want to accelerate the stabilisation for some reason
(although I can't see any at the moment), you can add threads to the
stabilisation (it currently only has one). For instance you could run 4
threads, each responsible for 40 entries in each finger table (the
stabilisation of a finger table entry involves a run of find_successor
which is by far the longest blocking operation of the stabilization
process).
> Also, from what I understand modern DHT such as 'maidsafe' are event-based.
>
>   
It is aso my intuition : event based in more hype than multi-thread
nowadays. I'm sensitive to this trend although it's hardly based on pure
logic ;-)
> Now, how do we get there, to the non-blocking I/O ? What I see are many
> changes to the DHT. Do we have a working (or almost) working blocking 
> design right now ? If yes, I would propose we go with it and review
> the operations before moving ahead with non-blocking. Typically there 
> are potential bugs that I see in the (old) 'dht' branch that need to be 
> discussed.
>   
We're already in the middle of a massive change and going for
non-blocking has a fundamental impact on the code.
> If no, I'd say we go straight for the non-blocking I/O, not waiting 
> 'en route'.
>   
I don't believe a switch from blocking to non blocking can happen
without breaking everything.
> Typically, could you give an estimate of the design & development time 
> needed in both cases ?
>
>   
No I can't. I thought I could a week ago but that was without the time
spent on evaluating the blocking / non blocking impact on the design. In
a nutshell, I will start to be able to estimate how long it will take
after a week of coding without facing significant architectural challenges.
> Thanks for the clear exposure of both architectures. Personnally I don't
> find the non-blocking I/O code to be much more difficult to read than
> the blocking one.
>   
I suspect you will be reminded this statement a couple of times in the
future ;-)

I should add that I did not make up this coding style. It is practiced
by applications using the deferred design pattern. I would be
unconfortable inventing a coding style or a design pattern in the
context of the implementation of the DHT.

Cheers