Menu
▾
▴
bigdata-developers — List for bigdata developers
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
(19) |
Mar
(8) |
Apr
(25) |
May
(16) |
Jun
(77) |
Jul
(131) |
Aug
(76) |
Sep
(30) |
Oct
(7) |
Nov
(3) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(2) |
Jun
(2) |
Jul
(16) |
Aug
(3) |
Sep
(1) |
Oct
|
Nov
(7) |
Dec
(7) |
| 2012 |
Jan
(10) |
Feb
(1) |
Mar
(8) |
Apr
(6) |
May
(1) |
Jun
(3) |
Jul
(1) |
Aug
|
Sep
(1) |
Oct
|
Nov
(8) |
Dec
(2) |
| 2013 |
Jan
(5) |
Feb
(12) |
Mar
(2) |
Apr
(1) |
May
(1) |
Jun
(1) |
Jul
(22) |
Aug
(50) |
Sep
(31) |
Oct
(64) |
Nov
(83) |
Dec
(28) |
| 2014 |
Jan
(31) |
Feb
(18) |
Mar
(27) |
Apr
(39) |
May
(45) |
Jun
(15) |
Jul
(6) |
Aug
(27) |
Sep
(6) |
Oct
(67) |
Nov
(70) |
Dec
(1) |
| 2015 |
Jan
(3) |
Feb
(18) |
Mar
(22) |
Apr
(121) |
May
(42) |
Jun
(17) |
Jul
(8) |
Aug
(11) |
Sep
(26) |
Oct
(15) |
Nov
(66) |
Dec
(38) |
| 2016 |
Jan
(14) |
Feb
(59) |
Mar
(28) |
Apr
(44) |
May
(21) |
Jun
(12) |
Jul
(9) |
Aug
(11) |
Sep
(4) |
Oct
(2) |
Nov
(1) |
Dec
|
| 2017 |
Jan
(20) |
Feb
(7) |
Mar
(4) |
Apr
(18) |
May
(7) |
Jun
(3) |
Jul
(13) |
Aug
(2) |
Sep
(4) |
Oct
(9) |
Nov
(2) |
Dec
(5) |
| 2018 |
Jan
|
Feb
|
Mar
|
Apr
(2) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2019 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: husdon <no...@no...> - 2010-07-26 17:38:28
|
See <http://localhost/job/BigData/changes> |
|
From: Fred O. <fko...@gm...> - 2010-07-26 14:40:16
|
I propose to fix ticket 119, removing the signal handlers from the bigdata services, extending the BroadcastSighup tool to handle the functionality and updating the scripts to use the tool. Diffs are atteached to the trac issue: http://sourceforge.net/apps/trac/bigdata/ticket/119 I propose to commit the changes after the lexicon changes are committed. Fred |
|
From: Bryan T. <br...@sy...> - 2010-07-25 22:06:38
|
All, We are preparing to close out the lexicon branch, merging it into the trunk and then tag a release. The lexicon branch incorporates physical schema changes which allow us to inline XSD numerics and some other datatypes into the statement indices. This will be a huge performance win for scale-out analytic queries. This change set BREAKS BINARY COMPATIBILITY with existing triple and quads store instances. Since every tuple in the lexicon and statement indices is different under the new physical schema, the easiest way to migrate is to export/import. In order to support existing installations, I have created a NO_INLINING_BRANCH. This branch starts at r3285. This is a branch from the trunk immediately prior to merging in the physical schema change required to support inlining of XSD numeric and other datatypes into the statement indices. I plan to prepare a release including including the change set from the lexicon branch on Monday / Tuesday. Thanks, Bryan |
|
From: husdon <no...@no...> - 2010-07-23 23:45:39
|
See <http://localhost/job/BigData/changes> |
|
From: husdon <no...@no...> - 2010-07-23 11:48:06
|
See <http://localhost/job/BigData/changes> |
|
From: husdon <no...@no...> - 2010-07-22 21:52:16
|
See <http://localhost/job/BigData/changes> |
|
From: husdon <no...@no...> - 2010-07-22 16:06:04
|
See <http://localhost/job/BigData/changes> |
|
From: husdon <no...@no...> - 2010-07-21 13:44:02
|
See <http://localhost/job/BigData/changes> |
|
From: husdon <no...@no...> - 2010-07-20 23:19:21
|
See <http://localhost/job/BigData/changes> |
|
From: Bryan T. <br...@sy...> - 2010-07-20 23:09:34
|
Fred, As I said, there is no real dependency on the zookeeper / SMS mechanisms for service start outside of the existing bigdata init.d style script and HA. HA does have dependencies on zookeeper, but could run with other mechanisms for service start. It is important for the HA/Quorums that the logical services declare their target replication count. Whether an operator or the SMS handles the service start is less important. However, let me ask why you would introduce a different mechanism when there is an existing implementation which is more flexible solution. Certainly there are a few outstanding issues against the current SMS/Zookeeper integration, but these are readily resolved. Further, the zookeeper/SMS approach can be templated just as you are suggesting for this alternative. In fact, the bigdata configuration file essentially provides that template. So, why should we explore a new mechanism? I suggest that it is easy enough to reduce the existing mechanism to the kind of simple template you are describing, but the reverse is not true. Why would this be a step forward? Also, for people who want the ability to "flex" the cluster, the existing mechanism is more flexible. Would your proposal in fact lead to two different service configuration/start mechanisms and if so, might that make the system less maintainable? I will also note that the existing bigdata init.d style script can be run without shared state. All it requires is a mechanism, such as puppet, to execute 'bigdata XXX', where XXX is the target run state for that node. At present this is done with cron, but that is certainly not a requirement of the script. Likewise, there is no requirement to aggregate logs to a logging service, etc. Bryan ________________________________________ From: Fred Oliver [fko...@gm...] Sent: Tuesday, July 20, 2010 6:26 PM To: Bryan Thompson Cc: Bigdata Developers Subject: Re: [Bigdata-developers] Why zookeeper? Bryan, On Tue, Jul 20, 2010 at 6:04 PM, Bryan Thompson <br...@sy...> wrote: > > The HotSpareXXXService could be used to provide some indirection for that purpose. I assume that this would be a trivial service which discloses the kind of services it is willing to start using Jini. The operator could then use a console or web application to list the known hot spares and the service types each could support and then make the decision to allocate a hot spare. The operator can create and install whatever configuration file is necessary on the cold spare host and start the failed services. The HotSpareDataService (etc.) idea is for the automated approach without SMS synchronization. Would you consider changing the startup mechanism to a simple, per host configuration, non-synchronizing mechanism? Fred |
|
From: Fred O. <fko...@gm...> - 2010-07-20 22:26:42
|
Bryan, On Tue, Jul 20, 2010 at 6:04 PM, Bryan Thompson <br...@sy...> wrote: > > The HotSpareXXXService could be used to provide some indirection for that purpose. I assume that this would be a trivial service which discloses the kind of services it is willing to start using Jini. The operator could then use a console or web application to list the known hot spares and the service types each could support and then make the decision to allocate a hot spare. The operator can create and install whatever configuration file is necessary on the cold spare host and start the failed services. The HotSpareDataService (etc.) idea is for the automated approach without SMS synchronization. Would you consider changing the startup mechanism to a simple, per host configuration, non-synchronizing mechanism? Fred |
|
From: Bryan T. <br...@sy...> - 2010-07-20 22:04:58
|
Fred,
If we assume that the deployment consists of installing a full stack suitable for running each of the different kinds of services and that the machines are capable of running those services, then we can certainly have an operator make a decision to allocate a hot spare to a specific logical service.
The HotSpareXXXService could be used to provide some indirection for that purpose. I assume that this would be a trivial service which discloses the kind of services it is willing to start using Jini. The operator could then use a console or web application to list the known hot spares and the service types each could support and then make the decision to allocate a hot spare.
In terms of the complexity of decision making, hot spare allocation (under the design that we have been targetting) would occur after a suitable unplanned downtime interval - on the order of one or two minutes. At that point a pre-imaged node capable of starting the desired service would start the service and enter into the resynchronization protocol with the quorum. The purpose of the interval before automated allocation is to hide transient failures. If a node was having continuing transient failures then you would probably want to force the allocation of the hot spare. On the other hand, if the node came back online and was able to resynchronize then the hot spare would be retired.
I agree that hot spare (de-)allocation adds complexity to the HA milestone. A related issue which I had planned to defer beyond the initial HA milestone is to dynamically change the size of the quorum. E.g., migrating a cluster from k=1 (no failover) to k=3 or from k=3 to k=5 (increased redundency).
Bryan
> -----Original Message-----
> From: Fred Oliver [mailto:fko...@gm...]
> Sent: Tuesday, July 20, 2010 5:41 PM
> To: Bryan Thompson
> Cc: Bigdata Developers
> Subject: Re: [Bigdata-developers] Why zookeeper?
>
> Bryan,
>
> We have no plan to use hot spares. That is, we expect an HA
> system to stay up and running long enough for an operator to
> diagnose a problem, and either fix it or manually configure
> and start a cold spare if necessary. I feel that automating
> this process when the actual faults are not understood is too
> likely to cause harm than help.
>
> Otherwise, why not have a set of HotSpare{Data,Metadata,...}Service
> instances configured and running on the hot spare machines,
> ready to become full participants when an HA quorum leader
> (or whatever
> mechanism) identifies a need? When a new XXXService is
> needed, a HotSpareXXXService is discovered and activated,
> registering a real XXXService. (Credit to Sean.)
>
> Fred
>
>
> On Tue, Jul 20, 2010 at 5:16 PM, Bryan Thompson
> <br...@sy...> wrote:
> > Fred,
> >
> > If you are not running the SMS, then you can simply start
> whatever services you want to start locally. The SMS is not
> used for anything besides actually starting the various
> services. Alternatively, a simpler SMS implementation could
> be used which read the list of services to start from a local
> configuration and ignored zookeeper.
> >
> > How would you propose to handle HA in that scenario? There
> is still a problem with dynamic recruitment from the pool of
> hot spares.
> >
> > Thanks,
> > Bryan
>
|
|
From: Fred O. <fko...@gm...> - 2010-07-20 21:40:39
|
Bryan,
We have no plan to use hot spares. That is, we expect an HA system to
stay up and running long enough for an operator to diagnose a problem,
and either fix it or manually configure and start a cold spare if
necessary. I feel that automating this process when the actual faults
are not understood is too likely to cause harm than help.
Otherwise, why not have a set of HotSpare{Data,Metadata,...}Service
instances configured and running on the hot spare machines, ready to
become full participants when an HA quorum leader (or whatever
mechanism) identifies a need? When a new XXXService is needed, a
HotSpareXXXService is discovered and activated, registering a real
XXXService. (Credit to Sean.)
Fred
On Tue, Jul 20, 2010 at 5:16 PM, Bryan Thompson <br...@sy...> wrote:
> Fred,
>
> If you are not running the SMS, then you can simply start whatever services you want to start locally. The SMS is not used for anything besides actually starting the various services. Alternatively, a simpler SMS implementation could be used which read the list of services to start from a local configuration and ignored zookeeper.
>
> How would you propose to handle HA in that scenario? There is still a problem with dynamic recruitment from the pool of hot spares.
>
> Thanks,
> Bryan
|
|
From: Bryan T. <br...@sy...> - 2010-07-20 21:17:05
|
Fred, If you are not running the SMS, then you can simply start whatever services you want to start locally. The SMS is not used for anything besides actually starting the various services. Alternatively, a simpler SMS implementation could be used which read the list of services to start from a local configuration and ignored zookeeper. How would you propose to handle HA in that scenario? There is still a problem with dynamic recruitment from the pool of hot spares. Thanks, Bryan > -----Original Message----- > From: Fred Oliver [mailto:fko...@gm...] > Sent: Tuesday, July 20, 2010 5:07 PM > To: Bryan Thompson > Subject: Re: [Bigdata-developers] Why zookeeper? > > I see. You've created a dynamic assignment of services to > hosts. I understand how this might work for transient > services (no persistence), but I don't understand why > persistent services would be dynamically assigned. I assume > you intended to create or allow a non-deterministic behaviour. > > For a production environment, we want to have as > deterministic a behaviour as possible. This allows for > simpler code, simpler and more complete tests, and more > easily diagnosed faults. I would prefer a startup scheme > where each host has its own configuration, which describes > exactly those services which are to run on that host. The > service layout was already determined in order to acquire the > hardware. > > Would you consider changing the startup mechanism to > something this simple? There would be some convenience in a > tool which is given a cluster configuration and spits out > configurations specific for each participating host. > > Fred > > On Tue, Jul 20, 2010 at 2:19 PM, Bryan Thompson > <br...@sy...> wrote: > > Fred, > > > > Zookeeper is used by the ServiceManagerServices (SMS) to > make shared decisions about which nodes will start which > services. The configuration information for the services > includes optional constraints on the nodes on which they can > run, on the services which must be running before they can > start (and I agree with you that services should start > without such preconditions and await the appropriate events), etc. > > > > Each SMS enters into a queue for each logical service type > registered against zookeeper. When an SMS is at the head of > that queue it has the "lock" (this is the zookeeper lock > pattern). It then decides whether or not it can start that > service type given the nodes capabilities and the constraints > (if any) for that logical service type. If it can, it starts > the service. This decision making process needs to be > globally atomic to prevent nodes from concurrent starts of > the same service on different nodes. > > > > Bryan > |
|
From: husdon <no...@no...> - 2010-07-20 20:58:06
|
See <http://localhost/job/BigData/changes> |
|
From: Bryan T. <br...@sy...> - 2010-07-20 18:37:08
|
Fred, I am not sure that this is supported right now. I have an option to enable the IniAdminPlugin, which presumably would then let me make the configuration changes you are suggestion. However, please see [1], which is an active ticket on sourceforge. If you would like to did into this a bit further and monitor this issue, I would be happy to setup the trac mailing list feature. Right now, what I do is maintain three windows for different views [2,3,4]. Thanks, Bryan [1] https://sourceforge.net/apps/trac/sourceforge/ticket/12406 (Issue on sourceforge for trac plugins). [2] https://sourceforge.net/apps/trac/bigdata/report/1?asc=1&sort=ticket (by ticket# so I can see creates). [3] https://sourceforge.net/apps/trac/bigdata/report/3 (by milestone). [4] https://sourceforge.net/apps/trac/bigdata/report/4 (accepted tickets by person). > -----Original Message----- > From: Fred Oliver [mailto:fko...@gm...] > Sent: Tuesday, July 20, 2010 1:38 PM > To: Bryan Thompson > Cc: Bigdata Developers > Subject: Re: [Bigdata-developers] Mailing lists for commit and trac? > > For the trac tickets, documentation here: > > http://trac.edgewall.org/wiki/TracNotification > > suggests that the "smtp_always_cc" or "smtp_always_bcc" > options in the trac configuration can be used to configure > trac to send email notifications. Are those accessible to you > as project owner? > > Fred > > On Tue, Jul 20, 2010 at 11:35 AM, Bryan Thompson > <br...@sy...> wrote: > > Fred, > > > > Yes, I think that is a good idea. I know how this is done > with CVS, but not with SVN. If you can track down how to do > this, I will apply the changes on sourceforge. Sourceforge > itself does post an RSS summary [2] of commits at the bottom > of [1]. However, I also prefer to see this as email traffic. > > > > Thanks, > > Bryan > > > > [1] https://sourceforge.net/projects/bigdata/ > > [2] https://sourceforge.net/export/rss2_keepsake.php?group_id=191861 > |
|
From: Bryan T. <br...@sy...> - 2010-07-20 18:19:57
|
Fred, Zookeeper is used by the ServiceManagerServices (SMS) to make shared decisions about which nodes will start which services. The configuration information for the services includes optional constraints on the nodes on which they can run, on the services which must be running before they can start (and I agree with you that services should start without such preconditions and await the appropriate events), etc. Each SMS enters into a queue for each logical service type registered against zookeeper. When an SMS is at the head of that queue it has the "lock" (this is the zookeeper lock pattern). It then decides whether or not it can start that service type given the nodes capabilities and the constraints (if any) for that logical service type. If it can, it starts the service. This decision making process needs to be globally atomic to prevent nodes from concurrent starts of the same service on different nodes. Bryan > -----Original Message----- > From: Fred Oliver [mailto:fko...@gm...] > Sent: Tuesday, July 20, 2010 1:20 PM > To: Bryan Thompson > Cc: Bigdata Developers > Subject: Re: [Bigdata-developers] Why zookeeper? > > On Tue, Jul 20, 2010 at 12:38 PM, Bryan Thompson > <br...@sy...> wrote: > > Fred, > > > > We should have a separate discussion concerning how bigdata > allocates and start services. I am a bit crushed for time > right now, but maybe we could take that up next week? > > I understand you'll be unavailable. Please pick up the > conversation when you can. > > > We only use global synchronous locks at the moment in > service startup logic, HA, locking out different masters for > the same distributed job. However, I think that specific > applications of bigdata may well want to use global > synchronous locks to make operations such as life cycle > management of a triple or quads store atomic. > > > > Concerning your example, service #5 is either running or it > is not, but we also need to know whether or not it has been > created. Bigdata services have huge amounts of persistent > state. We can not simply substitute another service as a > replacement for #5 if it should fail. Instead we have to > recruit a new service, synchronize it with the state of the > quorum to which #5 belonged, and then bring the new service > on atomically when it is caught up with the quorum. This > "hot spare" allocation process could take hours to bring a > newly recruited node into full synchronization. We speed > that up by working backwards in history so the data service > quickly has a view of the current commit point and then > builds up its history over time. > > I specifically meant to disregard HA for this conversation, > as bigdata has a long history with zookeeper outside of HA. > HA is a much longer conversation, and I don't want to confuse > tomorrow's issues with today's code. > > With that said, I don't understand what "whether or not it > has been created" means. DataService#5 could be called > "created" when the operator wrote (DataService, host H, > persistence directory D) in a config file on host H and > created directory D with #5 in it. After that, the service is > either running, or not running. > > Please explain where global synchronous locks are needed in > service startup logic? > > Fred > > > > > Bryan > > > >> -----Original Message----- > >> From: Fred Oliver [mailto:fko...@gm...] > >> Sent: Tuesday, July 20, 2010 10:59 AM > >> To: Bryan Thompson > >> Cc: Bigdata Developers > >> Subject: Re: [Bigdata-developers] Why zookeeper? > >> > >> On Tue, Jul 20, 2010 at 9:24 AM, Bryan Thompson <br...@sy...> > >> wrote: > >> > >> > In this regard it is more flexible than an Jini system with > >> support for creating global synchronous locks. > >> > >> I believe that global synchronous locks are more harmful than > >> helpful, in general. That's why I would like to know how global > >> synchronous locks help bigdata (not that zookeeper is a > bad way to do > >> it if necessary). > >> > >> > Services store their configuration state locally for > >> restart in the service directory. However, we also need to know > >> which persistent services exist, even if they are not > running at the > >> moment. That information is captured in zookeeper. For > example, if > >> the target #of logical data services (LDS) is 10, then we > do not want > >> to start a new data service if one goes down because the > persistent > >> state of the data service can be terabytes of files on local disks > >> and is part of the semantics of its service "identity". > >> > >> I'm not clear about the problem you are describing. Say we have a > >> DataService #5 configured one host H with a persistence > directory D > >> containing its UUID, journals, indices, etc. It is either > running and > >> registered with the lookup service (with the UUID) or not. If the > >> service starter/manager on host H needs this service to be running > >> and is not registered, start it. (There is a strictly > local problem > >> of starting a duplicate java process because of race > conditions, but > >> the service itself should detect and prevent > >> that.) I don't see how global synchronization is involved here. > >> > >> Can you give another example of the need for global > synchronization > >> (excluding HA) or point out what I am missing? > >> > >> Fred > >> > |
|
From: Fred O. <fko...@gm...> - 2010-07-20 17:38:23
|
For the trac tickets, documentation here: http://trac.edgewall.org/wiki/TracNotification suggests that the "smtp_always_cc" or "smtp_always_bcc" options in the trac configuration can be used to configure trac to send email notifications. Are those accessible to you as project owner? Fred On Tue, Jul 20, 2010 at 11:35 AM, Bryan Thompson <br...@sy...> wrote: > Fred, > > Yes, I think that is a good idea. I know how this is done with CVS, but not with SVN. If you can track down how to do this, I will apply the changes on sourceforge. Sourceforge itself does post an RSS summary [2] of commits at the bottom of [1]. However, I also prefer to see this as email traffic. > > Thanks, > Bryan > > [1] https://sourceforge.net/projects/bigdata/ > [2] https://sourceforge.net/export/rss2_keepsake.php?group_id=191861 |
|
From: Fred O. <fko...@gm...> - 2010-07-20 17:20:11
|
On Tue, Jul 20, 2010 at 12:38 PM, Bryan Thompson <br...@sy...> wrote: > Fred, > > We should have a separate discussion concerning how bigdata allocates and start services. I am a bit crushed for time right now, but maybe we could take that up next week? I understand you'll be unavailable. Please pick up the conversation when you can. > We only use global synchronous locks at the moment in service startup logic, HA, locking out different masters for the same distributed job. However, I think that specific applications of bigdata may well want to use global synchronous locks to make operations such as life cycle management of a triple or quads store atomic. > > Concerning your example, service #5 is either running or it is not, but we also need to know whether or not it has been created. Bigdata services have huge amounts of persistent state. We can not simply substitute another service as a replacement for #5 if it should fail. Instead we have to recruit a new service, synchronize it with the state of the quorum to which #5 belonged, and then bring the new service on atomically when it is caught up with the quorum. This "hot spare" allocation process could take hours to bring a newly recruited node into full synchronization. We speed that up by working backwards in history so the data service quickly has a view of the current commit point and then builds up its history over time. I specifically meant to disregard HA for this conversation, as bigdata has a long history with zookeeper outside of HA. HA is a much longer conversation, and I don't want to confuse tomorrow's issues with today's code. With that said, I don't understand what "whether or not it has been created" means. DataService#5 could be called "created" when the operator wrote (DataService, host H, persistence directory D) in a config file on host H and created directory D with #5 in it. After that, the service is either running, or not running. Please explain where global synchronous locks are needed in service startup logic? Fred > > Bryan > >> -----Original Message----- >> From: Fred Oliver [mailto:fko...@gm...] >> Sent: Tuesday, July 20, 2010 10:59 AM >> To: Bryan Thompson >> Cc: Bigdata Developers >> Subject: Re: [Bigdata-developers] Why zookeeper? >> >> On Tue, Jul 20, 2010 at 9:24 AM, Bryan Thompson >> <br...@sy...> wrote: >> >> > In this regard it is more flexible than an Jini system with >> support for creating global synchronous locks. >> >> I believe that global synchronous locks are more harmful than >> helpful, in general. That's why I would like to know how >> global synchronous locks help bigdata (not that zookeeper is >> a bad way to do it if necessary). >> >> > Services store their configuration state locally for >> restart in the service directory. However, we also need to >> know which persistent services exist, even if they are not >> running at the moment. That information is captured in >> zookeeper. For example, if the target #of logical data >> services (LDS) is 10, then we do not want to start a new data >> service if one goes down because the persistent state of the >> data service can be terabytes of files on local disks and is >> part of the semantics of its service "identity". >> >> I'm not clear about the problem you are describing. Say we >> have a DataService #5 configured one host H with a >> persistence directory D containing its UUID, journals, >> indices, etc. It is either running and registered with the >> lookup service (with the UUID) or not. If the service >> starter/manager on host H needs this service to be running >> and is not registered, start it. (There is a strictly local >> problem of starting a duplicate java process because of race >> conditions, but the service itself should detect and prevent >> that.) I don't see how global synchronization is involved here. >> >> Can you give another example of the need for global >> synchronization (excluding HA) or point out what I am missing? >> >> Fred >> |
|
From: husdon <no...@no...> - 2010-07-20 17:02:32
|
See <http://localhost/job/BigData/changes> |
|
From: Bryan T. <br...@sy...> - 2010-07-20 16:38:58
|
Fred, We should have a separate discussion concerning how bigdata allocates and start services. I am a bit crushed for time right now, but maybe we could take that up next week? We only use global synchronous locks at the moment in service startup logic, HA, locking out different masters for the same distributed job. However, I think that specific applications of bigdata may well want to use global synchronous locks to make operations such as life cycle management of a triple or quads store atomic. Concerning your example, service #5 is either running or it is not, but we also need to know whether or not it has been created. Bigdata services have huge amounts of persistent state. We can not simply substitute another service as a replacement for #5 if it should fail. Instead we have to recruit a new service, synchronize it with the state of the quorum to which #5 belonged, and then bring the new service on atomically when it is caught up with the quorum. This "hot spare" allocation process could take hours to bring a newly recruited node into full synchronization. We speed that up by working backwards in history so the data service quickly has a view of the current commit point and then builds up its history over time. Bryan > -----Original Message----- > From: Fred Oliver [mailto:fko...@gm...] > Sent: Tuesday, July 20, 2010 10:59 AM > To: Bryan Thompson > Cc: Bigdata Developers > Subject: Re: [Bigdata-developers] Why zookeeper? > > On Tue, Jul 20, 2010 at 9:24 AM, Bryan Thompson > <br...@sy...> wrote: > > > In this regard it is more flexible than an Jini system with > support for creating global synchronous locks. > > I believe that global synchronous locks are more harmful than > helpful, in general. That's why I would like to know how > global synchronous locks help bigdata (not that zookeeper is > a bad way to do it if necessary). > > > Services store their configuration state locally for > restart in the service directory. However, we also need to > know which persistent services exist, even if they are not > running at the moment. That information is captured in > zookeeper. For example, if the target #of logical data > services (LDS) is 10, then we do not want to start a new data > service if one goes down because the persistent state of the > data service can be terabytes of files on local disks and is > part of the semantics of its service "identity". > > I'm not clear about the problem you are describing. Say we > have a DataService #5 configured one host H with a > persistence directory D containing its UUID, journals, > indices, etc. It is either running and registered with the > lookup service (with the UUID) or not. If the service > starter/manager on host H needs this service to be running > and is not registered, start it. (There is a strictly local > problem of starting a duplicate java process because of race > conditions, but the service itself should detect and prevent > that.) I don't see how global synchronization is involved here. > > Can you give another example of the need for global > synchronization (excluding HA) or point out what I am missing? > > Fred > |
|
From: Bryan T. <br...@sy...> - 2010-07-20 16:22:32
|
Please note that all committers will have to be authorized for the new list. Mike and I are both able to authorize subscriptions to the net list (a commit will attempt to email the list using your username). (Mike, please look out for those authorization requests). Bryan > -----Original Message----- > From: Bryan Thompson [mailto:br...@sy...] > Sent: Tuesday, July 20, 2010 12:18 PM > To: Bigdata Developers > Subject: Re: [Bigdata-developers] SourceForge.net New Mailing List > > All, you can subscribe to the commit mailing list at [1]. Bryan > > [1] https://lists.sourceforge.net/lists/listinfo/bigdata-commit > > > -----Original Message----- > > From: Bryan Thompson [mailto:br...@sy...] > > Sent: Tuesday, July 20, 2010 11:56 AM > > To: Bigdata Developers > > Subject: [Bigdata-developers] FW: SourceForge.net New Mailing List > > > > Ok. The request to create the list has been queued. Once > the list is > > active I will add the SVN hook and update the wiki > documentation [1] > > so people can subscribe if they are interested in this > traffic. I've > > also added a link to [1] from the developer's section of > the blog [2]. > > > > Thanks, > > Bryan > > > > [1] > > https://sourceforge.net/apps/mediawiki/bigdata/index.php?title > > =Contributors > > [2] http://www.bigdata.com/blog > > > > -----Original Message----- > > From: SourceForge.net [mailto:no...@so...] > > Sent: Tuesday, July 20, 2010 11:50 AM > > To: Bryan Thompson > > Subject: SourceForge.net New Mailing List > > > > > > A mailing list will be created on SourceForge.net in 6-24 hours and > > you are the list administrator. > > > > This list is: big...@li... > > > > Your mailing list info is at: > > http://lists.sourceforge.net/mailman/listinfo/bigdata-commit > > > > List administration can be found at: > > https://lists.sourceforge.net/mailman/admin/bigdata-commit > > > > Thank you for registering your project with SourceForge.net. > > > > -- the SourceForge.net staff > > > > > > -------------------------------------------------------------- > > ---------------- > > This SF.net email is sponsored by Sprint What will you do > first with > > EVO, the first 4G phone? > > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > > _______________________________________________ > > Bigdata-developers mailing list > > Big...@li... > > https://lists.sourceforge.net/lists/listinfo/bigdata-developers > > > -------------------------------------------------------------- > ---------------- > This SF.net email is sponsored by Sprint What will you do > first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Bigdata-developers mailing list > Big...@li... > https://lists.sourceforge.net/lists/listinfo/bigdata-developers > |
|
From: Bryan T. <br...@sy...> - 2010-07-20 16:18:51
|
All, you can subscribe to the commit mailing list at [1]. Bryan [1] https://lists.sourceforge.net/lists/listinfo/bigdata-commit > -----Original Message----- > From: Bryan Thompson [mailto:br...@sy...] > Sent: Tuesday, July 20, 2010 11:56 AM > To: Bigdata Developers > Subject: [Bigdata-developers] FW: SourceForge.net New Mailing List > > Ok. The request to create the list has been queued. Once > the list is active I will add the SVN hook and update the > wiki documentation [1] so people can subscribe if they are > interested in this traffic. I've also added a link to [1] > from the developer's section of the blog [2]. > > Thanks, > Bryan > > [1] > https://sourceforge.net/apps/mediawiki/bigdata/index.php?title > =Contributors > [2] http://www.bigdata.com/blog > > -----Original Message----- > From: SourceForge.net [mailto:no...@so...] > Sent: Tuesday, July 20, 2010 11:50 AM > To: Bryan Thompson > Subject: SourceForge.net New Mailing List > > > A mailing list will be created on SourceForge.net in 6-24 > hours and you are the list administrator. > > This list is: big...@li... > > Your mailing list info is at: > http://lists.sourceforge.net/mailman/listinfo/bigdata-commit > > List administration can be found at: > https://lists.sourceforge.net/mailman/admin/bigdata-commit > > Thank you for registering your project with SourceForge.net. > > -- the SourceForge.net staff > > > -------------------------------------------------------------- > ---------------- > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Bigdata-developers mailing list > Big...@li... > https://lists.sourceforge.net/lists/listinfo/bigdata-developers > |
|
From: Bryan T. <br...@sy...> - 2010-07-20 15:57:04
|
Ok. The request to create the list has been queued. Once the list is active I will add the SVN hook and update the wiki documentation [1] so people can subscribe if they are interested in this traffic. I've also added a link to [1] from the developer's section of the blog [2]. Thanks, Bryan [1] https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=Contributors [2] http://www.bigdata.com/blog -----Original Message----- From: SourceForge.net [mailto:no...@so...] Sent: Tuesday, July 20, 2010 11:50 AM To: Bryan Thompson Subject: SourceForge.net New Mailing List A mailing list will be created on SourceForge.net in 6-24 hours and you are the list administrator. This list is: big...@li... Your mailing list info is at: http://lists.sourceforge.net/mailman/listinfo/bigdata-commit List administration can be found at: https://lists.sourceforge.net/mailman/admin/bigdata-commit Thank you for registering your project with SourceForge.net. -- the SourceForge.net staff |
|
From: <bra...@no...> - 2010-07-20 15:38:51
|
Here you go http://jetcetera.wordpress.com/2007/12/12/configuring-sourceforge-to-automatically-generate-svn-commit-notifications/ brad -----Original Message----- From: ext Bryan Thompson [mailto:br...@sy...] Sent: Tuesday, July 20, 2010 11:35 AM To: Fred Oliver; Bigdata Developers Subject: Re: [Bigdata-developers] Mailing lists for commit and trac? Fred, Yes, I think that is a good idea. I know how this is done with CVS, but not with SVN. If you can track down how to do this, I will apply the changes on sourceforge. Sourceforge itself does post an RSS summary [2] of commits at the bottom of [1]. However, I also prefer to see this as email traffic. Thanks, Bryan [1] https://sourceforge.net/projects/bigdata/ [2] https://sourceforge.net/export/rss2_keepsake.php?group_id=191861 > -----Original Message----- > From: Fred Oliver [mailto:fko...@gm...] > Sent: Tuesday, July 20, 2010 11:31 AM > To: Bigdata Developers > Subject: [Bigdata-developers] Mailing lists for commit and trac? > > Bryan, > > Can you create sourceforge mailing lists for automatic > distribution of svn commit messages and trac issue creation > and updates? > > Fred > > -------------------------------------------------------------- > ---------------- > This SF.net email is sponsored by Sprint What will you do > first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Bigdata-developers mailing list > Big...@li... > https://lists.sourceforge.net/lists/listinfo/bigdata-developers > ------------------------------------------------------------------------------ This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first _______________________________________________ Bigdata-developers mailing list Big...@li... https://lists.sourceforge.net/lists/listinfo/bigdata-developers |
27 messages has been excluded from this view by a project administrator.
