Re: [Zookeeper-user] zookeeper vs. heartbeat messages

[Stefan G. <sg...@10...>]

As mention we plan to build a distributed lucene serving grid system.  
One virtual index composed out of many lucene indexes (shards).
Each slave should server a set of shards. The master need to assign  
shards to a slave if it becomes available or a new index is deployed.
A slave need to get notifications if a new shard is assigned to it.  
The master need to get notifications in case a slave dies or for some  
reason can not serve a shard.
The master also need to know some all shards a slave serves.
How you would recommend to structure those information?
Use one file per slave that is updated by master and slave and  
contains all shards meta data.
Or one folder per node and each shard is one file?
Or...?

Thanks a lot!
Stefan



In our case we want to manage shards assign
On Apr 22, 2008, at 2:55 PM, Mahadev Konar wrote:
> Thanks Ted for your reasons listed. I would just like to clarify that
> the 1MB limit is the limit per node. Zookeeper keeps its datatree in
> memory. So theretically you are just limited by the amount of memory  
> you
> have on machines you are running zookeeper on. Most of our users use
> Zookeeper for failure detection of slave nodes, master discovery and
> also for assigning workloads among nodes. They haven't had any memory
> issues.
>
> Regards
> Mahadev
>
>> -----Original Message-----
>> From: zoo...@li...
> [mailto:zookeeper-user-
>> bo...@li...] On Behalf Of Ted Dunning
>> Sent: Tuesday, April 22, 2008 2:47 PM
>> To: Stefan Groschupf; zoo...@li...
>> Subject: Re: [Zookeeper-user] zookeeper vs. heartbeat messages
>>
>>
>> I would definitely recommend zookeeper over heartbeat.  Here are my
>> reasons:
>>
>> A) it works.  Anything you implement using heartbeats will not work
>> initially and you will (should) always worry about that  
>> implementation
>> because it won't get as much testing.
>>
>> B) it works.  The use of ephemeral files is much easier than doing a
> good
>> heartbeat API.  In particular, heartbeat always implies some sort of
>> fail-over which is trivial to implement well in zookeeper and very
> hard to
>> implement (correctly) by hand.
>>
>> C) it works.  Heartbeat architectures often lead logically to a
> spiraling
>> number of heartbeats being exchanged.  With zookeeper, your server
> will be
>> talking to zookeeper, and everybody else will have low latency  
>> updates
> in
>> case of a problem (if they want).
>>
>>
>> Regarding the amount of data, I doubt you will have a problem if you
> keep
>> your zookeeper files reasonably sized.  There is an imposed limit of
> 1MB,
>> but would you be going anywhere near that?
>>
>> On 4/22/08 2:31 PM, "Stefan Groschupf" <sg...@10...> wrote:
>>
>>> Hi
>>> I'm new to zookeeper and I work on a system that require classical
>>> master - slave communication similar to hadoop dfs or hbase.
>>> Though in my case the master need to know much faster if a slave
>>> crashes than in other cases.
>>> So I wonder if it would make sense instead of using classical
>>> heartbeat messages in can use zookeeper.
>>> Basically the master need to know if a new slave becomes available
> and
>>> what kind of data it servers.
>>>
>>> Is there any limitations where the amount of data stored within
>>> zookeeper becomes an issue?
>>> Would you recommend to use zookeeper over heartbeat messages?
>>>
>>> Thanks for any hints.
>>> Stefan
>>>
>>>
>>>
> ------------------------------------------------------------------------
>> -
>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>>> Don't miss this year's exciting event. There's still time to save
> $100.
>>> Use priority code J8TL2D2.
>>>
>>
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
> av
>> aone
>>> _______________________________________________
>>> Zookeeper-user mailing list
>>> Zoo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user
>>
>>
>>
> ------------------------------------------------------------------------
> -
>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>> Don't miss this year's exciting event. There's still time to save
> $100.
>> Use priority code J8TL2D2.
>>
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
> av
>> aone
>> _______________________________________________
>> Zookeeper-user mailing list
>> Zoo...@li...
>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com