From: Mahadev K. <ma...@ya...> - 2008-04-22 22:30:30
|
I would suggest > Use one file per slave that is updated by master and slave and > contains all shards meta data. Regards Mahadev > -----Original Message----- > From: Stefan Groschupf [mailto:sg...@10...] > Sent: Tuesday, April 22, 2008 3:16 PM > To: zoo...@li...; Mahadev Konar > Subject: Re: [Zookeeper-user] zookeeper vs. heartbeat messages > > As mention we plan to build a distributed lucene serving grid system. > One virtual index composed out of many lucene indexes (shards). > Each slave should server a set of shards. The master need to assign > shards to a slave if it becomes available or a new index is deployed. > A slave need to get notifications if a new shard is assigned to it. > The master need to get notifications in case a slave dies or for some > reason can not serve a shard. > The master also need to know some all shards a slave serves. > How you would recommend to structure those information? > Use one file per slave that is updated by master and slave and > contains all shards meta data. > Or one folder per node and each shard is one file? > Or...? > > Thanks a lot! > Stefan > > > > In our case we want to manage shards assign > On Apr 22, 2008, at 2:55 PM, Mahadev Konar wrote: > > Thanks Ted for your reasons listed. I would just like to clarify that > > the 1MB limit is the limit per node. Zookeeper keeps its datatree in > > memory. So theretically you are just limited by the amount of memory > > you > > have on machines you are running zookeeper on. Most of our users use > > Zookeeper for failure detection of slave nodes, master discovery and > > also for assigning workloads among nodes. They haven't had any memory > > issues. > > > > Regards > > Mahadev > > > >> -----Original Message----- > >> From: zoo...@li... > > [mailto:zookeeper-user- > >> bo...@li...] On Behalf Of Ted Dunning > >> Sent: Tuesday, April 22, 2008 2:47 PM > >> To: Stefan Groschupf; zoo...@li... > >> Subject: Re: [Zookeeper-user] zookeeper vs. heartbeat messages > >> > >> > >> I would definitely recommend zookeeper over heartbeat. Here are my > >> reasons: > >> > >> A) it works. Anything you implement using heartbeats will not work > >> initially and you will (should) always worry about that > >> implementation > >> because it won't get as much testing. > >> > >> B) it works. The use of ephemeral files is much easier than doing a > > good > >> heartbeat API. In particular, heartbeat always implies some sort of > >> fail-over which is trivial to implement well in zookeeper and very > > hard to > >> implement (correctly) by hand. > >> > >> C) it works. Heartbeat architectures often lead logically to a > > spiraling > >> number of heartbeats being exchanged. With zookeeper, your server > > will be > >> talking to zookeeper, and everybody else will have low latency > >> updates > > in > >> case of a problem (if they want). > >> > >> > >> Regarding the amount of data, I doubt you will have a problem if you > > keep > >> your zookeeper files reasonably sized. There is an imposed limit of > > 1MB, > >> but would you be going anywhere near that? > >> > >> On 4/22/08 2:31 PM, "Stefan Groschupf" <sg...@10...> wrote: > >> > >>> Hi > >>> I'm new to zookeeper and I work on a system that require classical > >>> master - slave communication similar to hadoop dfs or hbase. > >>> Though in my case the master need to know much faster if a slave > >>> crashes than in other cases. > >>> So I wonder if it would make sense instead of using classical > >>> heartbeat messages in can use zookeeper. > >>> Basically the master need to know if a new slave becomes available > > and > >>> what kind of data it servers. > >>> > >>> Is there any limitations where the amount of data stored within > >>> zookeeper becomes an issue? > >>> Would you recommend to use zookeeper over heartbeat messages? > >>> > >>> Thanks for any hints. > >>> Stefan > >>> > >>> > >>> > > ------------------------------------------------------------------------ > >> - > >>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > >>> Don't miss this year's exciting event. There's still time to save > > $100. > >>> Use priority code J8TL2D2. > >>> > >> > > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j > > av > >> aone > >>> _______________________________________________ > >>> Zookeeper-user mailing list > >>> Zoo...@li... > >>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user > >> > >> > >> > > ------------------------------------------------------------------------ > > - > >> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > >> Don't miss this year's exciting event. There's still time to save > > $100. > >> Use priority code J8TL2D2. > >> > > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j > > av > >> aone > >> _______________________________________________ > >> Zookeeper-user mailing list > >> Zoo...@li... > >> https://lists.sourceforge.net/lists/listinfo/zookeeper-user > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > 101tec Inc. > Menlo Park, California, USA > http://www.101tec.com > |