Re: [Zookeeper-user] zookeeper vs. heartbeat messages

[Patrick H. <ph...@gm...>]

Hi Stefan, have you looked at the "bailey" project on sourceforge? 
Sounds very similar (they are also considering using ZooKeeper).

https://sourceforge.net/projects/bailey

Regards,

Patrick

Stefan Groschupf wrote:
> As mention we plan to build a distributed lucene serving grid system.  
> One virtual index composed out of many lucene indexes (shards).
> Each slave should server a set of shards. The master need to assign  
> shards to a slave if it becomes available or a new index is deployed.
> A slave need to get notifications if a new shard is assigned to it.  
> The master need to get notifications in case a slave dies or for some  
> reason can not serve a shard.
> The master also need to know some all shards a slave serves.
> How you would recommend to structure those information?
> Use one file per slave that is updated by master and slave and  
> contains all shards meta data.
> Or one folder per node and each shard is one file?
> Or...?
> 
> Thanks a lot!
> Stefan
> 
> 
> 
> In our case we want to manage shards assign
> On Apr 22, 2008, at 2:55 PM, Mahadev Konar wrote:
>> Thanks Ted for your reasons listed. I would just like to clarify that
>> the 1MB limit is the limit per node. Zookeeper keeps its datatree in
>> memory. So theretically you are just limited by the amount of memory  
>> you
>> have on machines you are running zookeeper on. Most of our users use
>> Zookeeper for failure detection of slave nodes, master discovery and
>> also for assigning workloads among nodes. They haven't had any memory
>> issues.
>>
>> Regards
>> Mahadev
>>
>>> -----Original Message-----
>>> From: zoo...@li...
>> [mailto:zookeeper-user-
>>> bo...@li...] On Behalf Of Ted Dunning
>>> Sent: Tuesday, April 22, 2008 2:47 PM
>>> To: Stefan Groschupf; zoo...@li...
>>> Subject: Re: [Zookeeper-user] zookeeper vs. heartbeat messages
>>>
>>>
>>> I would definitely recommend zookeeper over heartbeat.  Here are my
>>> reasons:
>>>
>>> A) it works.  Anything you implement using heartbeats will not work
>>> initially and you will (should) always worry about that  
>>> implementation
>>> because it won't get as much testing.
>>>
>>> B) it works.  The use of ephemeral files is much easier than doing a
>> good
>>> heartbeat API.  In particular, heartbeat always implies some sort of
>>> fail-over which is trivial to implement well in zookeeper and very
>> hard to
>>> implement (correctly) by hand.
>>>
>>> C) it works.  Heartbeat architectures often lead logically to a
>> spiraling
>>> number of heartbeats being exchanged.  With zookeeper, your server
>> will be
>>> talking to zookeeper, and everybody else will have low latency  
>>> updates
>> in
>>> case of a problem (if they want).
>>>
>>>
>>> Regarding the amount of data, I doubt you will have a problem if you
>> keep
>>> your zookeeper files reasonably sized.  There is an imposed limit of
>> 1MB,
>>> but would you be going anywhere near that?
>>>
>>> On 4/22/08 2:31 PM, "Stefan Groschupf" <sg...@10...> wrote:
>>>
>>>> Hi
>>>> I'm new to zookeeper and I work on a system that require classical
>>>> master - slave communication similar to hadoop dfs or hbase.
>>>> Though in my case the master need to know much faster if a slave
>>>> crashes than in other cases.
>>>> So I wonder if it would make sense instead of using classical
>>>> heartbeat messages in can use zookeeper.
>>>> Basically the master need to know if a new slave becomes available
>> and
>>>> what kind of data it servers.
>>>>
>>>> Is there any limitations where the amount of data stored within
>>>> zookeeper becomes an issue?
>>>> Would you recommend to use zookeeper over heartbeat messages?
>>>>
>>>> Thanks for any hints.
>>>> Stefan
>>>>
>>>>
>>>>
>> ------------------------------------------------------------------------
>>> -
>>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>>>> Don't miss this year's exciting event. There's still time to save
>> $100.
>>>> Use priority code J8TL2D2.
>>>>
>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
>> av
>>> aone
>>>> _______________________________________________
>>>> Zookeeper-user mailing list
>>>> Zoo...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user
>>>
>>>
>> ------------------------------------------------------------------------
>> -
>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>>> Don't miss this year's exciting event. There's still time to save
>> $100.
>>> Use priority code J8TL2D2.
>>>
>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
>> av
>>> aone
>>> _______________________________________________
>>> Zookeeper-user mailing list
>>> Zoo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> Menlo Park, California, USA
> http://www.101tec.com
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
> Don't miss this year's exciting event. There's still time to save $100. 
> Use priority code J8TL2D2. 
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> Zookeeper-user mailing list
> Zoo...@li...
> https://lists.sourceforge.net/lists/listinfo/zookeeper-user