Re: [Bigdata-developers] zookeeper discovery

[Bryan T. <br...@sy...>]

Brian,

Ok.  My interest ties it with my other email to the list to break out the bulk loader configuration from the main configuration file [1].  I'd like to be able to configure the bulk loader based on the minimum amount of shared state.  This is not a blocking issue as I could proceed on this issue by replicating the zookeeper client configuration information into a bulk loader configuration file for now and then slim down the configuration further after you finish that wrapper.

Let me know if I can help on the smart proxy patterns for the CS/DS.  The work that I am currently doing on scale-out query evaluation will help to take some of the data out of the RMI messaging but it will not introduce any changes to the CS/DS public APIs.  What I am doing is extending the ResourceService exposed by the DS to move index segment files around to also allow the interchange of ByteBuffers containing intermediate query results.  In combination with a few other things, this is going to tremendously simplify how we handle join evaluation for scale-out.

Thanks,
Bryan

[1] https://sourceforge.net/apps/trac/bigdata/ticket/148

________________________________
From: Brian Murphy [mailto:btm...@gm...]
Sent: Wednesday, August 25, 2010 8:03 AM
To: big...@li...
Subject: Re: [Bigdata-developers] zookeeper discovery

On Wed, Aug 25, 2010 at 5:49 AM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote:

You had been looking at wrapping zookeeper for discovery from jini.  What is the status of that effort?

The wrapper class is complete and checked in to my
development branch, but only manually tested. Currently,
it can be started from pstart or from the boot manager
process added to that branch. It still has to be modified
to be startable from the ServicesManagerService, but
more importantly, a client wrapper needs to be written.
I was planning on doing the remaining work after the
smart proxy work is complete for each of the other services;
in which the data service and the client service remain.

BrianM