Re: [Bigdata-developers] Why zookeeper?

[Bryan T. <br...@sy...>]

Fred,

If you are not running the SMS, then you can simply start whatever services you want to start locally.  The SMS is not used for anything besides actually starting the various services.  Alternatively, a simpler SMS implementation could be used which read the list of services to start from a local configuration and ignored zookeeper.

How would you propose to handle HA in that scenario?  There is still a problem with dynamic recruitment from the pool of hot spares.

Thanks,
Bryan 

> -----Original Message-----
> From: Fred Oliver [mailto:fko...@gm...] 
> Sent: Tuesday, July 20, 2010 5:07 PM
> To: Bryan Thompson
> Subject: Re: [Bigdata-developers] Why zookeeper?
> 
> I see. You've created a dynamic assignment of services to 
> hosts. I understand how this might work for transient 
> services (no persistence), but I don't understand why 
> persistent services would be dynamically assigned. I assume 
> you intended to create or allow a non-deterministic behaviour.
> 
> For a production environment, we want to have as 
> deterministic a behaviour as possible. This allows for 
> simpler code, simpler and more complete tests, and more 
> easily diagnosed faults. I would prefer a startup scheme 
> where each host has its own configuration, which describes 
> exactly those services which are to run on that host. The 
> service layout was already determined in order to acquire the 
> hardware.
> 
> Would you consider changing the startup mechanism to 
> something this simple? There would be some convenience in a 
> tool which is given a cluster configuration and spits out 
> configurations specific for each participating host.
> 
> Fred
> 
> On Tue, Jul 20, 2010 at 2:19 PM, Bryan Thompson 
> <br...@sy...> wrote:
> > Fred,
> >
> > Zookeeper is used by the ServiceManagerServices (SMS) to 
> make shared decisions about which nodes will start which 
> services.  The configuration information for the services 
> includes optional constraints on the nodes on which they can 
> run, on the services which must be running before they can 
> start (and I agree with you that services should start 
> without such preconditions and await the appropriate events), etc.
> >
> > Each SMS enters into a queue for each logical service type 
> registered against zookeeper.  When an SMS is at the head of 
> that queue it has the "lock" (this is the zookeeper lock 
> pattern).  It then decides whether or not it can start that 
> service type given the nodes capabilities and the constraints 
> (if any) for that logical service type.  If it can, it starts 
> the service.  This decision making process needs to be 
> globally atomic to prevent nodes from concurrent starts of 
> the same service on different nodes.
> >
> > Bryan
>