From: Stefan G. <sg...@10...> - 2008-04-22 23:19:40
|
No - I did not heard about it. Interesting all the right people work on it already. :) On Apr 22, 2008, at 4:04 PM, Patrick Hunt wrote: > Hi Stefan, have you looked at the "bailey" project on sourceforge? > Sounds very similar (they are also considering using ZooKeeper). > > https://sourceforge.net/projects/bailey > > Regards, > > Patrick > > Stefan Groschupf wrote: >> As mention we plan to build a distributed lucene serving grid >> system. One virtual index composed out of many lucene indexes >> (shards). >> Each slave should server a set of shards. The master need to >> assign shards to a slave if it becomes available or a new index is >> deployed. >> A slave need to get notifications if a new shard is assigned to >> it. The master need to get notifications in case a slave dies or >> for some reason can not serve a shard. >> The master also need to know some all shards a slave serves. >> How you would recommend to structure those information? >> Use one file per slave that is updated by master and slave and >> contains all shards meta data. >> Or one folder per node and each shard is one file? >> Or...? >> Thanks a lot! >> Stefan >> In our case we want to manage shards assign >> On Apr 22, 2008, at 2:55 PM, Mahadev Konar wrote: >>> Thanks Ted for your reasons listed. I would just like to clarify >>> that >>> the 1MB limit is the limit per node. Zookeeper keeps its datatree in >>> memory. So theretically you are just limited by the amount of >>> memory you >>> have on machines you are running zookeeper on. Most of our users use >>> Zookeeper for failure detection of slave nodes, master discovery and >>> also for assigning workloads among nodes. They haven't had any >>> memory >>> issues. >>> >>> Regards >>> Mahadev >>> >>>> -----Original Message----- >>>> From: zoo...@li... >>> [mailto:zookeeper-user- >>>> bo...@li...] On Behalf Of Ted Dunning >>>> Sent: Tuesday, April 22, 2008 2:47 PM >>>> To: Stefan Groschupf; zoo...@li... >>>> Subject: Re: [Zookeeper-user] zookeeper vs. heartbeat messages >>>> >>>> >>>> I would definitely recommend zookeeper over heartbeat. Here are my >>>> reasons: >>>> >>>> A) it works. Anything you implement using heartbeats will not work >>>> initially and you will (should) always worry about that >>>> implementation >>>> because it won't get as much testing. >>>> >>>> B) it works. The use of ephemeral files is much easier than >>>> doing a >>> good >>>> heartbeat API. In particular, heartbeat always implies some sort >>>> of >>>> fail-over which is trivial to implement well in zookeeper and very >>> hard to >>>> implement (correctly) by hand. >>>> >>>> C) it works. Heartbeat architectures often lead logically to a >>> spiraling >>>> number of heartbeats being exchanged. With zookeeper, your server >>> will be >>>> talking to zookeeper, and everybody else will have low latency >>>> updates >>> in >>>> case of a problem (if they want). >>>> >>>> >>>> Regarding the amount of data, I doubt you will have a problem if >>>> you >>> keep >>>> your zookeeper files reasonably sized. There is an imposed limit >>>> of >>> 1MB, >>>> but would you be going anywhere near that? >>>> >>>> On 4/22/08 2:31 PM, "Stefan Groschupf" <sg...@10...> wrote: >>>> >>>>> Hi >>>>> I'm new to zookeeper and I work on a system that require classical >>>>> master - slave communication similar to hadoop dfs or hbase. >>>>> Though in my case the master need to know much faster if a slave >>>>> crashes than in other cases. >>>>> So I wonder if it would make sense instead of using classical >>>>> heartbeat messages in can use zookeeper. >>>>> Basically the master need to know if a new slave becomes available >>> and >>>>> what kind of data it servers. >>>>> >>>>> Is there any limitations where the amount of data stored within >>>>> zookeeper becomes an issue? >>>>> Would you recommend to use zookeeper over heartbeat messages? >>>>> >>>>> Thanks for any hints. >>>>> Stefan >>>>> >>>>> >>>>> >>> ------------------------------------------------------------------------ >>>> - >>>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>>>> Don't miss this year's exciting event. There's still time to save >>> $100. >>>>> Use priority code J8TL2D2. >>>>> >>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j >>> av >>>> aone >>>>> _______________________________________________ >>>>> Zookeeper-user mailing list >>>>> Zoo...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user >>>> >>>> >>> ------------------------------------------------------------------------ >>> - >>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>>> Don't miss this year's exciting event. There's still time to save >>> $100. >>>> Use priority code J8TL2D2. >>>> >>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j >>> av >>>> aone >>>> _______________________________________________ >>>> Zookeeper-user mailing list >>>> Zoo...@li... >>>> https://lists.sourceforge.net/lists/listinfo/zookeeper-user >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> 101tec Inc. >> Menlo Park, California, USA >> http://www.101tec.com >> ------------------------------------------------------------------------- >> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >> Don't miss this year's exciting event. There's still time to save >> $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >> _______________________________________________ >> Zookeeper-user mailing list >> Zoo...@li... >> https://lists.sourceforge.net/lists/listinfo/zookeeper-user > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 101tec Inc. Menlo Park, California, USA http://www.101tec.com |