Re: [Bigdata-developers] Alternate install/deploy mechanism

[Brian M. <btm...@gm...>]

On Wed, Jun 23, 2010 at 8:23 PM, Bryan Thompson <br...@sy...> wrote:

Right now, bigdata depends on leader election semantics from zookeeper to
> start the appropriate mixture of services.  I did not see zookeeper running
> so I presume that you are handling that differently in this example.
>

No, zookeeper was running. If you run the disco-tool (or a jini browser),
you should see a service of type com.bigdata.service.QuorumPeerService;
which is zookeeper wrapped in a Jini service. Wrapping zookeeper in
a Jini service not only provides a means to more easily start and stop
zookeeper, but also provides a means to dynamically discover zookeeper
in the federation. Furthermore, the QuorumPeerService interface provides
a mechanism to customize how the services interact with zookeeper if
desired.

I would like to understand how we would handle the distributed decision
> making necessary to start an appropriate mixture of services with this
> proposal and also how we would handle the distributed decision making
> required to support the HA quorums.  I've attached an updated version of my
> draft for the HA quorum design and the proposed zookeeper integration.
>

Rather than using zookeeper to decide what gets started, this
mechanism allows one to configure what individual services get
started where, including the appropriate number of zookeeper
instances. Zookeeper would then be viewed as a discoverable
resource that can be used by the other services to determine who
is the leader and whether or not a quorum exists before those
services are used.


> I realize that some jini implementation do provide capabilities similar to
> what zookeeper provides.
>

I'm not sure what jini implementations you're talking about.
Something not in the Jini starter kit?


> Are you suggesting that or did you simply leave zookeeper and its roles in
> configuration management, leader elections, etc. out of the demo?
>

As I said above, zookeeper was not left out. But I also
said in my original posting that this work is not anywhere
near complete, and was posted to give folks an idea of what
could be done with install and deployment if the services
are re-implemented to a smart proxy model and move to
a shared nothing architecture; all of which I believe will be
a significant amount of work.

Perhaps in the future I should hold off on posting until the
work is more complete. Sorry if I caused confusion.

BrianM