Re: [Bigdata-developers] Why zookeeper?

[Fred O. <fko...@gm...>]

On Tue, Jul 20, 2010 at 9:24 AM, Bryan Thompson <br...@sy...> wrote:

> In this regard it is more flexible than an Jini system with support for creating global synchronous locks.

I believe that global synchronous locks are more harmful than helpful,
in general. That's why I would like to know how global synchronous
locks help bigdata (not that zookeeper is a bad way to do it if
necessary).

> Services store their configuration state locally for restart in the service directory.  However, we also need to know which persistent services exist, even if they are not running at the moment. That information is captured in zookeeper.  For example, if the target #of logical data services (LDS) is 10, then we do not want to start a new data service if one goes down because the persistent state of the data service can be terabytes of files on local disks and is part of the semantics of its service "identity".

I'm not clear about the problem you are describing. Say we have a
DataService #5 configured one host H with a persistence directory D
containing its UUID, journals, indices, etc. It is either running and
registered with the lookup service (with the UUID) or not. If the
service starter/manager on host H needs this service to be running and
is not registered, start it. (There is a strictly local problem of
starting a duplicate java process because of race conditions, but the
service itself should detect and prevent that.) I don't see how global
synchronization is involved here.

Can you give another example of the need for global synchronization
(excluding HA) or point out what I am missing?

Fred