zookeeper-user Mailing List for ZooKeeper (Page 8)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi Ted,
I like this idea a lot, though I noticed you get a notification on the  
parent folder.
For example adding a node "/slaves/server1" triggers a notification "/ 
slaves".
What basically means my master need to know the children of "/slaves"  
from before to identify the new node in the "/slaves" path. What means  
the master has a kind of state.
Or is there anything I miss?
Thanks
Stefan

On Apr 22, 2008, at 3:28 PM, Ted Dunning wrote:

>
>
> I think I would use a redundant representation where there is
>
> * a directory with ephemeral files, one per slave
>
> * one directory per slave containing references to the shards it  
> should
> serve
>
> * one directory per shard with ephemeral files per slave serving the  
> shard
>
> Suppose the whole system is rooted at /katta, these directories  
> would be
> /katta/slaves, /katta/shard-assignments, /katta/shard-servers.  The  
> purpose
> of /katta/slaves is to allow a quick inventory of all live servers  
> and to
> have a single location where notifications of slave death or birth  
> can be
> done.  The purpose of /katta/shard-assignments is to allow the  
> shards to
> each have a directory on which they can listen for new shards or old  
> shards
> that are being taken away.  The purpose of /katta/shard-servers is  
> so that
> it is easy to find under-served shards whenever a slave goes down or  
> comes
> up.  I think that this structure is such that no locks need ever be  
> used to
> ensure consistency.
>
> One implementation that I have had pretty good luck with is to build a
> stateless rest layer that does complex manipulations on a zookeeper  
> storage
> layer.  This lets me think in terms of large scale operations (get a  
> batch
> of work) and inherently allows easy access from a variety of different
> languages.  Since the rest layer is stateless, it can be scaled
> effortlessly.
>
>
>
> On 4/22/08 3:15 PM, "Stefan Groschupf" <sg...@10...> wrote:
>
>> As mention we plan to build a distributed lucene serving grid system.
>> One virtual index composed out of many lucene indexes (shards).
>> Each slave should server a set of shards. The master need to assign
>> shards to a slave if it becomes available or a new index is deployed.
>> A slave need to get notifications if a new shard is assigned to it.
>> The master need to get notifications in case a slave dies or for some
>> reason can not serve a shard.
>> The master also need to know some all shards a slave serves.
>> How you would recommend to structure those information?
>> Use one file per slave that is updated by master and slave and
>> contains all shards meta data.
>> Or one folder per node and each shard is one file?
>> Or...?
>>
>> Thanks a lot!
>> Stefan
>>
>>
>>
>> In our case we want to manage shards assign
>> On Apr 22, 2008, at 2:55 PM, Mahadev Konar wrote:
>>> Thanks Ted for your reasons listed. I would just like to clarify  
>>> that
>>> the 1MB limit is the limit per node. Zookeeper keeps its datatree in
>>> memory. So theretically you are just limited by the amount of memory
>>> you
>>> have on machines you are running zookeeper on. Most of our users use
>>> Zookeeper for failure detection of slave nodes, master discovery and
>>> also for assigning workloads among nodes. They haven't had any  
>>> memory
>>> issues.
>>>
>>> Regards
>>> Mahadev
>>>
>>>> -----Original Message-----
>>>> From: zoo...@li...
>>> [mailto:zookeeper-user-
>>>> bo...@li...] On Behalf Of Ted Dunning
>>>> Sent: Tuesday, April 22, 2008 2:47 PM
>>>> To: Stefan Groschupf; zoo...@li...
>>>> Subject: Re: [Zookeeper-user] zookeeper vs. heartbeat messages
>>>>
>>>>
>>>> I would definitely recommend zookeeper over heartbeat.  Here are my
>>>> reasons:
>>>>
>>>> A) it works.  Anything you implement using heartbeats will not work
>>>> initially and you will (should) always worry about that
>>>> implementation
>>>> because it won't get as much testing.
>>>>
>>>> B) it works.  The use of ephemeral files is much easier than  
>>>> doing a
>>> good
>>>> heartbeat API.  In particular, heartbeat always implies some sort  
>>>> of
>>>> fail-over which is trivial to implement well in zookeeper and very
>>> hard to
>>>> implement (correctly) by hand.
>>>>
>>>> C) it works.  Heartbeat architectures often lead logically to a
>>> spiraling
>>>> number of heartbeats being exchanged.  With zookeeper, your server
>>> will be
>>>> talking to zookeeper, and everybody else will have low latency
>>>> updates
>>> in
>>>> case of a problem (if they want).
>>>>
>>>>
>>>> Regarding the amount of data, I doubt you will have a problem if  
>>>> you
>>> keep
>>>> your zookeeper files reasonably sized.  There is an imposed limit  
>>>> of
>>> 1MB,
>>>> but would you be going anywhere near that?
>>>>
>>>> On 4/22/08 2:31 PM, "Stefan Groschupf" <sg...@10...> wrote:
>>>>
>>>>> Hi
>>>>> I'm new to zookeeper and I work on a system that require classical
>>>>> master - slave communication similar to hadoop dfs or hbase.
>>>>> Though in my case the master need to know much faster if a slave
>>>>> crashes than in other cases.
>>>>> So I wonder if it would make sense instead of using classical
>>>>> heartbeat messages in can use zookeeper.
>>>>> Basically the master need to know if a new slave becomes available
>>> and
>>>>> what kind of data it servers.
>>>>>
>>>>> Is there any limitations where the amount of data stored within
>>>>> zookeeper becomes an issue?
>>>>> Would you recommend to use zookeeper over heartbeat messages?
>>>>>
>>>>> Thanks for any hints.
>>>>> Stefan
>>>>>
>>>>>
>>>>>
>>> ------------------------------------------------------------------------
>>>> -
>>>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>>>>> Don't miss this year's exciting event. There's still time to save
>>> $100.
>>>>> Use priority code J8TL2D2.
>>>>>
>>>>
>>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
>>> av
>>>> aone
>>>>> _______________________________________________
>>>>> Zookeeper-user mailing list
>>>>> Zoo...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>> -
>>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>>>> Don't miss this year's exciting event. There's still time to save
>>> $100.
>>>> Use priority code J8TL2D2.
>>>>
>>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
>>> av
>>>> aone
>>>> _______________________________________________
>>>> Zookeeper-user mailing list
>>>> Zoo...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user
>>>
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 101tec Inc.
>> Menlo Park, California, USA
>> http://www.101tec.com
>>
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>> Don't miss this year's exciting event. There's still time to save  
>> $100.
>> Use priority code J8TL2D2.
>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
>> _______________________________________________
>> Zookeeper-user mailing list
>> Zoo...@li...
>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com

2007	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov (1)	Dec (26)
2008	Jan (5)	Feb (1)	Mar	Apr (71)	May (22)	Jun (47)	Jul (32)	Aug (18)	Sep (9)	Oct (4)	Nov	Dec
2009	Jan	Feb	Mar	Apr	May (3)	Jun	Jul	Aug	Sep	Oct	Nov	Dec

zookeeper-user Mailing List for ZooKeeper (Page 8)

zookeeper-user — Mailing list for users of ZooKeeper