SmartFrog

Zhang Qian wrote:
> Hi All,
> 

Hello!

> I have two questions about SmartFrog:
> 1. Can SmartFrog be used to synchronize configuration on many hosts? I
> have a cluster which contains hundreds of hosts, so it's very
> important to make config changes synchronoursly on these hosts. Is it
> possible to write my own SmartFrog component whose responsibility is
> communicate with my own daemons on all the hosts, and deliver config
> changes to them synchronoursly?


This is one of those really interesting areas where automated deployment 
gets both challenging and fun. I'm actually preparing some slides for a 
talk on that topic for presentation to undergraduates on friday -though 
I wont be going into any details on how to get it to work.

Once you have that many hosts you can't assume that any small set of 
them will remain functional; if you have one or two nodes that are 
declared managers, you can be sure that eventually they will fail and 
your entire farm will go offline. If you have a simple hierarchy of 
deployed components, you can easily create such a failure point

What you have to do instead is make every node standalone, sharing 
awareness of their role amongst their peers

The Anubis component is what we use for this kind of farm management; 
here are the papers:
http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/anubis/doc/

Anubis is a partition-aware version of a tuple space; you insert facts 
into the space, the machines talk by a heartbeat and a tick after you 
insert a fact, it is shared amongst all peers, a tick after that they 
know you know that fact, and so on. if there is a partition event - a 
host goes away, the network gets split- you get notified and the tuple 
space(s) that now exist have to re-evaluate who is in there and who is not.

You bring up every node in the cluster 'unpurposed' and then let them 
decide -based on what else is live- what they are going to be. The first 
one could be the resource manager that allocates work to others; then 
you could bring up some as a database, a filestore, and then finally 
your application itself.

All the anubis components are in the redistributables, and the 
sf-anubis.rpm, where they can be pushed out to servers. We also have an 
anubisdeployer component that exists to deploy anubis itself. There's 
also a nice visualizer application that lets you see what is going on, 
and test failure recovery by triggering partition events.

There's some limitations of the (current) design I should flag early, 
two of which are all based on the use of multicast IP to share information.
  1. it doesnt run on Amazon EC2, as their network doesnt support 
multicasts.
  2. it doesnt like farms where the machines are connected over 
long-haul networks. Single site networks are OK, though the more complex 
the network, the bigger the TTL has to be and the slower you need to 
make the heartbeat.
  3. It is currently dependent on the clocks being synchronised. If NTP 
is not working properly across all nodes, you have problems. this is 
something that could be fixed, as it is less fundamental to the design 
than multicasting.

I'd recommend you have a look at the papers and the examples; talk to us 
if you want more details or help bringing it up. We have used Anubis 
successfully  for 500+ node deployments.


> 
> 2. I am trying to write a sample SmartFrog component for my own, but I
> find it can not take effect until sfDaemon restarts. I have written a
> java class which extends PrimImpl and implements Prim, also written
> related .sf file to describe the config. At first, it works fine, but
> when I change some codes in my java class, and build it by ant, run it
> by sfStart, the code I added will not take effect until I restart
> sfDaemon. Did I missed some thing or made some mistake?

It comes down to if/whether you are using dynamic code downloading, and 
JVM classloading quirks. If Java has loaded a class and there are still 
references to it around somewhere, it tends to keep the old classes loaded.

if you dont use dynamic classloading then yes, you must restart the JVM 
to get the JARs reloaded.

If you do want to dynamically classload, then you must

  -create a list of urls to the JAR files in the sfCodebase. There's 
some documentation on this as you can specify a codebase for parsing the 
deployment files as a JVM property in the sfDeploy operation, and a 
codebase for the actual process classloading. All java URLs are 
supported: http:, https:, ftp:, file:

-for absolute reliability start your deployments in a new JVM. This is 
as simple as using the sfProcessName attribute in a component/compound; 
it tells the runtime to put everything beneath there in a different 
process. A new process is dynamically created, which will pick up all 
the changed files. This is something to try if somehow the JVM is 
hanging on to old class definitions after the components are unloaded.

We have ant tasks to do the start/deploy, tasks that can help set up the 
codebase. If you want some help setting up your build file, you can send 
us those bits of your build.xml and I'll take a look at them.


I just want to close by saying yes, big clusters is what SmartFrog can 
do; its just the changing nature of the hardware changes your deployment 
architecture in interesting ways.

-Steve


-- 
-----------------------
Hewlett-Packard Limited
Registered Office: Cain Road, Bracknell, Berks RG12 1HN
Registered No: 690597 England

Just a couple of corrections:

>  We also have an anubisdeployer component that exists to deploy anubis it=
self.

The AnubisDeployer is a SmartFrog deployer that uses Anubis to determine wh=
ere to deploy things - it doesn't deploy Anubis.

>  3. It is currently dependent on the clocks being synchronised.
>     If NTP is not working properly across all nodes, you have problems. t=
his is something that
>     could be fixed, as it is less fundamental to the design than multicas=
ting.

This statement is not true (but once was). Anubis has two timing protocols:
 - one times clock skew, comms delay and scheduling dely so machines with c=
locks that drift become untimely relative to one another
 - one times comms delay and scheduling delay only, so clocks are allowed t=
o drift.

Paul.

Paul Murray
Hewlett-Packard Laboratories, Bristol

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12=
 1HN
Registered No: 690597 England

The contents of this message and any attachments to it are confidential and=
 may be legally privileged. If you have received this message in error, you=
 should delete it from your system immediately and advise the sender.

To any recipient of this message within HP, unless otherwise stated you sho=
uld consider this message and attachments as "HP CONFIDENTIAL".

-----Original Message-----
From: smartfrog-developer-bounces@... [mailto:smartfrog-d=
eveloper-bounces@...] On Behalf Of Steve Loughran
Sent: 05 December 2007 12:36
To: Zhang Qian; smartfrog-developer
Subject: Re: [Smartfrog-developer] Questions about SmartFrog

Zhang Qian wrote:
> Hi All,
>

Hello!

> I have two questions about SmartFrog:
> 1. Can SmartFrog be used to synchronize configuration on many hosts? I
> have a cluster which contains hundreds of hosts, so it's very
> important to make config changes synchronoursly on these hosts. Is it
> possible to write my own SmartFrog component whose responsibility is
> communicate with my own daemons on all the hosts, and deliver config
> changes to them synchronoursly?


This is one of those really interesting areas where automated deployment ge=
ts both challenging and fun. I'm actually preparing some slides for a talk =
on that topic for presentation to undergraduates on friday -though I wont b=
e going into any details on how to get it to work.

Once you have that many hosts you can't assume that any small set of them w=
ill remain functional; if you have one or two nodes that are declared manag=
ers, you can be sure that eventually they will fail and your entire farm wi=
ll go offline. If you have a simple hierarchy of deployed components, you c=
an easily create such a failure point

What you have to do instead is make every node standalone, sharing awarenes=
s of their role amongst their peers

The Anubis component is what we use for this kind of farm management; here =
are the papers:
http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components=
/anubis/doc/

Anubis is a partition-aware version of a tuple space; you insert facts into=
 the space, the machines talk by a heartbeat and a tick after you insert a =
fact, it is shared amongst all peers, a tick after that they know you know =
that fact, and so on. if there is a partition event - a host goes away, the=
 network gets split- you get notified and the tuple
space(s) that now exist have to re-evaluate who is in there and who is not.

You bring up every node in the cluster 'unpurposed' and then let them decid=
e -based on what else is live- what they are going to be. The first one cou=
ld be the resource manager that allocates work to others; then you could br=
ing up some as a database, a filestore, and then finally your application i=
tself.

All the anubis components are in the redistributables, and the sf-anubis.rp=
m, where they can be pushed out to servers. We also have an anubisdeployer =
component that exists to deploy anubis itself. There's also a nice visualiz=
er application that lets you see what is going on, and test failure recover=
y by triggering partition events.

There's some limitations of the (current) design I should flag early, two o=
f which are all based on the use of multicast IP to share information.
  1. it doesnt run on Amazon EC2, as their network doesnt support multicast=
s.
  2. it doesnt like farms where the machines are connected over long-haul n=
etworks. Single site networks are OK, though the more complex the network, =
the bigger the TTL has to be and the slower you need to make the heartbeat.
  3. It is currently dependent on the clocks being synchronised. If NTP is =
not working properly across all nodes, you have problems. this is something=
 that could be fixed, as it is less fundamental to the design than multicas=
ting.

I'd recommend you have a look at the papers and the examples; talk to us if=
 you want more details or help bringing it up. We have used Anubis successf=
ully  for 500+ node deployments.


>
> 2. I am trying to write a sample SmartFrog component for my own, but I
> find it can not take effect until sfDaemon restarts. I have written a
> java class which extends PrimImpl and implements Prim, also written
> related .sf file to describe the config. At first, it works fine, but
> when I change some codes in my java class, and build it by ant, run it
> by sfStart, the code I added will not take effect until I restart
> sfDaemon. Did I missed some thing or made some mistake?

It comes down to if/whether you are using dynamic code downloading, and JVM=
 classloading quirks. If Java has loaded a class and there are still refere=
nces to it around somewhere, it tends to keep the old classes loaded.

if you dont use dynamic classloading then yes, you must restart the JVM to =
get the JARs reloaded.

If you do want to dynamically classload, then you must

  -create a list of urls to the JAR files in the sfCodebase. There's some d=
ocumentation on this as you can specify a codebase for parsing the deployme=
nt files as a JVM property in the sfDeploy operation, and a codebase for th=
e actual process classloading. All java URLs are
supported: http:, https:, ftp:, file:

-for absolute reliability start your deployments in a new JVM. This is as s=
imple as using the sfProcessName attribute in a component/compound; it tell=
s the runtime to put everything beneath there in a different process. A new=
 process is dynamically created, which will pick up all the changed files. =
This is something to try if somehow the JVM is hanging on to old class defi=
nitions after the components are unloaded.

We have ant tasks to do the start/deploy, tasks that can help set up the co=
debase. If you want some help setting up your build file, you can send us t=
hose bits of your build.xml and I'll take a look at them.


I just want to close by saying yes, big clusters is what SmartFrog can do; =
its just the changing nature of the hardware changes your deployment archit=
ecture in interesting ways.

-Steve


--
-----------------------
Hewlett-Packard Limited
Registered Office: Cain Road, Bracknell, Berks RG12 1HN Registered No: 6905=
97 England

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper from=
 Novell.  From the desktop to the data center, Linux is going mainstream.  =
Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
Smartfrog-developer mailing list
Smartfrog-developer@...
https://lists.sourceforge.net/lists/listinfo/smartfrog-developer

A quick summary of the Anubis deployer - for details look at the documentat=
ion. The Anubis deployer has two aspects.

Firstly allows users to provide state information regarding a particular VM=
 as a set of attributes, these are in three categories: properties (eg x86 =
host, linux os, ...), quantities (e.g. memory available) and lists of enume=
rated values (e.g. free ports [80, 8080, 8088]).

Secondly instead of providing attributes that specify host names for locati=
ons to deploy components, attributes may be given that specify properties (=
e.g. must be linux), required quantities (need 20MB memory) and required nu=
mbers of the values (a port number).

The deployer then matches the properties, ensures that there are sufficient=
 of the quantities, and sufficient of the enumerated values, and deploys on=
 a host that is appropriate. On deployment, advertised quantities are reduc=
ed, the selected values are removed from the sets and put into the descript=
ion that is being deployed - ie the deployed description uses up resources.=
 On termination, those resources (quantities and enumerated values) are put=
 back into the advertised available values and quantities.

Intended for deploying into clusters of worker nodes where location isn't i=
mportant but capacities and certain properties are.

-----Original Message-----
From: smartfrog-developer-bounces@... [mailto:smartfrog-d=
eveloper-bounces@...] On Behalf Of Murray, Paul (HP Labs,=
 Bristol)
Sent: 05 December 2007 13:10
To: Steve Loughran; Zhang Qian; smartfrog-developer
Subject: Re: [Smartfrog-developer] Questions about SmartFrog

Just a couple of corrections:

>  We also have an anubisdeployer component that exists to deploy anubis it=
self.

The AnubisDeployer is a SmartFrog deployer that uses Anubis to determine wh=
ere to deploy things - it doesn't deploy Anubis.

>  3. It is currently dependent on the clocks being synchronised.
>     If NTP is not working properly across all nodes, you have problems. t=
his is something that
>     could be fixed, as it is less fundamental to the design than multicas=
ting.

This statement is not true (but once was). Anubis has two timing protocols:
 - one times clock skew, comms delay and scheduling dely so machines with c=
locks that drift become untimely relative to one another
 - one times comms delay and scheduling delay only, so clocks are allowed t=
o drift.

Paul.

Paul Murray
Hewlett-Packard Laboratories, Bristol

Hewlett-Packard Limited registered Office: Cain Road, Bracknell, Berks RG12=
 1HN
Registered No: 690597 England

The contents of this message and any attachments to it are confidential and=
 may be legally privileged. If you have received this message in error, you=
 should delete it from your system immediately and advise the sender.

To any recipient of this message within HP, unless otherwise stated you sho=
uld consider this message and attachments as "HP CONFIDENTIAL".

-----Original Message-----
From: smartfrog-developer-bounces@... [mailto:smartfrog-d=
eveloper-bounces@...] On Behalf Of Steve Loughran
Sent: 05 December 2007 12:36
To: Zhang Qian; smartfrog-developer
Subject: Re: [Smartfrog-developer] Questions about SmartFrog

Zhang Qian wrote:
> Hi All,
>

Hello!

> I have two questions about SmartFrog:
> 1. Can SmartFrog be used to synchronize configuration on many hosts? I
> have a cluster which contains hundreds of hosts, so it's very
> important to make config changes synchronoursly on these hosts. Is it
> possible to write my own SmartFrog component whose responsibility is
> communicate with my own daemons on all the hosts, and deliver config
> changes to them synchronoursly?


This is one of those really interesting areas where automated deployment ge=
ts both challenging and fun. I'm actually preparing some slides for a talk =
on that topic for presentation to undergraduates on friday -though I wont b=
e going into any details on how to get it to work.

Once you have that many hosts you can't assume that any small set of them w=
ill remain functional; if you have one or two nodes that are declared manag=
ers, you can be sure that eventually they will fail and your entire farm wi=
ll go offline. If you have a simple hierarchy of deployed components, you c=
an easily create such a failure point

What you have to do instead is make every node standalone, sharing awarenes=
s of their role amongst their peers

The Anubis component is what we use for this kind of farm management; here =
are the papers:
http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components=
/anubis/doc/

Anubis is a partition-aware version of a tuple space; you insert facts into=
 the space, the machines talk by a heartbeat and a tick after you insert a =
fact, it is shared amongst all peers, a tick after that they know you know =
that fact, and so on. if there is a partition event - a host goes away, the=
 network gets split- you get notified and the tuple
space(s) that now exist have to re-evaluate who is in there and who is not.

You bring up every node in the cluster 'unpurposed' and then let them decid=
e -based on what else is live- what they are going to be. The first one cou=
ld be the resource manager that allocates work to others; then you could br=
ing up some as a database, a filestore, and then finally your application i=
tself.

All the anubis components are in the redistributables, and the sf-anubis.rp=
m, where they can be pushed out to servers. We also have an anubisdeployer =
component that exists to deploy anubis itself. There's also a nice visualiz=
er application that lets you see what is going on, and test failure recover=
y by triggering partition events.

There's some limitations of the (current) design I should flag early, two o=
f which are all based on the use of multicast IP to share information.
  1. it doesnt run on Amazon EC2, as their network doesnt support multicast=
s.
  2. it doesnt like farms where the machines are connected over long-haul n=
etworks. Single site networks are OK, though the more complex the network, =
the bigger the TTL has to be and the slower you need to make the heartbeat.
  3. It is currently dependent on the clocks being synchronised. If NTP is =
not working properly across all nodes, you have problems. this is something=
 that could be fixed, as it is less fundamental to the design than multicas=
ting.

I'd recommend you have a look at the papers and the examples; talk to us if=
 you want more details or help bringing it up. We have used Anubis successf=
ully  for 500+ node deployments.


>
> 2. I am trying to write a sample SmartFrog component for my own, but I
> find it can not take effect until sfDaemon restarts. I have written a
> java class which extends PrimImpl and implements Prim, also written
> related .sf file to describe the config. At first, it works fine, but
> when I change some codes in my java class, and build it by ant, run it
> by sfStart, the code I added will not take effect until I restart
> sfDaemon. Did I missed some thing or made some mistake?

It comes down to if/whether you are using dynamic code downloading, and JVM=
 classloading quirks. If Java has loaded a class and there are still refere=
nces to it around somewhere, it tends to keep the old classes loaded.

if you dont use dynamic classloading then yes, you must restart the JVM to =
get the JARs reloaded.

If you do want to dynamically classload, then you must

  -create a list of urls to the JAR files in the sfCodebase. There's some d=
ocumentation on this as you can specify a codebase for parsing the deployme=
nt files as a JVM property in the sfDeploy operation, and a codebase for th=
e actual process classloading. All java URLs are
supported: http:, https:, ftp:, file:

-for absolute reliability start your deployments in a new JVM. This is as s=
imple as using the sfProcessName attribute in a component/compound; it tell=
s the runtime to put everything beneath there in a different process. A new=
 process is dynamically created, which will pick up all the changed files. =
This is something to try if somehow the JVM is hanging on to old class defi=
nitions after the components are unloaded.

We have ant tasks to do the start/deploy, tasks that can help set up the co=
debase. If you want some help setting up your build file, you can send us t=
hose bits of your build.xml and I'll take a look at them.


I just want to close by saying yes, big clusters is what SmartFrog can do; =
its just the changing nature of the hardware changes your deployment archit=
ecture in interesting ways.

-Steve


--
-----------------------
Hewlett-Packard Limited
Registered Office: Cain Road, Bracknell, Berks RG12 1HN Registered No: 6905=
97 England

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper from=
 Novell.  From the desktop to the data center, Linux is going mainstream.  =
Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
Smartfrog-developer mailing list
Smartfrog-developer@...
https://lists.sourceforge.net/lists/listinfo/smartfrog-developer

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
Smartfrog-developer mailing list
Smartfrog-developer@...
https://lists.sourceforge.net/lists/listinfo/smartfrog-developer

Murray, Paul (HP Labs, Bristol) wrote:
> Just a couple of corrections:
> 
>>  We also have an anubisdeployer component that exists to deploy anubis itself.
> 
> The AnubisDeployer is a SmartFrog deployer that uses Anubis to determine where to deploy things - it doesn't deploy Anubis.

ah, now I understand. Sorry for causing the confusion

>>  3. It is currently dependent on the clocks being synchronised.
>>     If NTP is not working properly across all nodes, you have problems. this is something that
>>     could be fixed, as it is less fundamental to the design than multicasting.
> 
> This statement is not true (but once was). Anubis has two timing protocols:
>  - one times clock skew, comms delay and scheduling dely so machines with clocks that drift become untimely relative to one another
>  - one times comms delay and scheduling delay only, so clocks are allowed to drift.

the latter works best on virtualized systems, yes?

Hi All,

Thanks for your replies about this topic.

I'd like to share more details about my cluster with you.
As you know, it's a cluster includes hundreds of nodes. We divide
these nodes into
two categories: management nodes and computing nodes.

For computing nodes, they just run the task arranged to them, do not
have management roles, so we don't care it in this case.

For management nodes, we have a dozen of this kind of nodes in the
cluster. Only
one of them is the master node whose responsibility is to manage the
entirely cluster, others are just the master candidates. The reason we
do it in this way is to avoid single point failure, once the master
node fails, a master candidate will take over its job, and become the
new master node. So we have the heartbeat mechanism to detect the node
status to realize fail-over.

Now there is a limitation: our cluster relies on shared-file system(such as NFS)
which can be accessed by all the management nodes.That means all the config
files placed on the shared-file system, all the management nodes need these
config files. It's the master node's responsibilityto update these config file
according to user's request, after a fail-over, the new master node
will read these
config file to know the latest configuration.

Now we want to remove the shared-file system dependency, each management
node has config files in its local file system. So obviously, we need
a mechanism
to synchronize these config files on all the management nodes. That's
why I asked
that questions.

I don't know whether there is a inbuilt component or service can
provide this kind of mechanism in SmartFrog. Certainly I will
investigate Anubis first, thanks for your sharing.


Regards,
Qian

In addition, we have had a management console for user which will
communicate with our daemon in the master node, and deliver config
change to that daemon.
After receive the config change, this daemon will verify and activate
the change first,
then write it into the config file placed on the shared-file system.

This is what we are doing, but we want to remove shared-file system dependency.



Thanks and Regards,
Qian

Zhang Qian wrote:
> Hi All,
> 
> Thanks for your replies about this topic.
> 
> I'd like to share more details about my cluster with you.
> As you know, it's a cluster includes hundreds of nodes. We divide
> these nodes into
> two categories: management nodes and computing nodes.

I see. We've tend to prefer the tactic of letting any node become a 
master (with agreement), because stops you having to decide which 
machines are in charge. Whatever boots up first can take over.

> For computing nodes, they just run the task arranged to them, do not
> have management roles, so we don't care it in this case.

OK -the workers are expected to fail and are told what to do; if they go 
away then something else gets the job.

> 
> For management nodes, we have a dozen of this kind of nodes in the
> cluster. Only
> one of them is the master node whose responsibility is to manage the
> entirely cluster, others are just the master candidates. The reason we
> do it in this way is to avoid single point failure, once the master
> node fails, a master candidate will take over its job, and become the
> new master node. So we have the heartbeat mechanism to detect the node
> status to realize fail-over.

OK. You're giving one machine charge of the resource management problem, 
but by sharing the data amongst candiates, if the master goes away you 
can have an election of some sort to decide who is the new master.



> Now there is a limitation: our cluster relies on shared-file system(such as NFS)
> which can be accessed by all the management nodes.That means all the config
> files placed on the shared-file system, all the management nodes need these
> config files. It's the master node's responsibilityto update these config file
> according to user's request, after a fail-over, the new master node
> will read these
> config file to know the latest configuration.

ah, so
  1. the NFS filestore is a failure point
  2. you need to save the configuration to a filesystem that doesnt go 
out of its way to enable locking


> 
> Now we want to remove the shared-file system dependency, each management
> node has config files in its local file system. So obviously, we need
> a mechanism
> to synchronize these config files on all the management nodes. That's
> why I asked
> that questions.

> I don't know whether there is a inbuilt component or service can
> provide this kind of mechanism in SmartFrog. Certainly I will
> investigate Anubis first, thanks for your sharing.

This is what anubis is designed for, to make a cluster out of a set of 
machines on a LAN. The papers and Paul can provide more details.

 > In addition, we have had a management console for user which will
 > communicate with our daemon in the master node, and deliver config
 > change to that daemon.
 > After receive the config change, this daemon will verify and activate
 > the change first,
 > then write it into the config file placed on the shared-file system.

I see. How does the management console deal with failure of the master? 
Does it discover it using some discovery protocol, or is the active 
master expected to update a dynamic DNS entry?

> I see. How does the management console deal with failure of the master?
> Does it discover it using some discovery protocol, or is the active
> master expected to update a dynamic DNS entry?

Yes, we deal with this issue by DNS way.

Today I took a look at Anubis document. As my understanding, It seems
Anubis is a notification service and provides a detection mechanism
for distributed system.

But in my cluster, we have already had this kind of mechanism for
detecting the status of our key daemons, dealing with master failure,
etc. We don't want to change that, just want to remove the shared-file
system dependency. Anubis looks a little big for this request. As I
know, SmartFrog has shipped some inbuild services for file operation,
downloading in its package. I am wondering it is possible to fulfill
my request by writing a SmartFrog which just extends these inbuilt
service.



Thanks,
Qian

Hi Qian,

What kind of configuration data are you talking about? Is it lots of data o=
r small sets of attribute value pairs?

One way that we have used Anubis is to propagate changes in the configurati=
on data so that all the "master nodes" can see those changes and cache the =
changes locally. This is simple to do with Anubis because of the guaranties=
 and consistency of the Anubis notifications. You could probably do somethi=
ng similar using your notification mechanism.

Then we have, as you mentioned, components to operate with the file system =
and/or with ftp/ssh/... that could be extended to meet your needs. One inte=
resting component could be a wrapper for rsync but this won't help you that=
 much in n+1 configurations.

Other possibilities are: use simple multicast to announce changes in your c=
onfiguration data or use RSS feeds to propagate these changes.

The right solution will depends exactly on your architecture and type/amoun=
t of data to synchronize.

Regards,

Julio Guijarro

-----Original Message-----
From: smartfrog-developer-bounces@... [mailto:smartfrog-d=
eveloper-bounces@...] On Behalf Of Zhang Qian
Sent: 07 December 2007 06:06
To: Steve Loughran
Cc: smartfrog-developer
Subject: Re: [Smartfrog-developer] Questions about SmartFrog

> I see. How does the management console deal with failure of the master?
> Does it discover it using some discovery protocol, or is the active
> master expected to update a dynamic DNS entry?

Yes, we deal with this issue by DNS way.

Today I took a look at Anubis document. As my understanding, It seems
Anubis is a notification service and provides a detection mechanism
for distributed system.

But in my cluster, we have already had this kind of mechanism for
detecting the status of our key daemons, dealing with master failure,
etc. We don't want to change that, just want to remove the shared-file
system dependency. Anubis looks a little big for this request. As I
know, SmartFrog has shipped some inbuild services for file operation,
downloading in its package. I am wondering it is possible to fulfill
my request by writing a SmartFrog which just extends these inbuilt
service.



Thanks,
Qian

-------------------------------------------------------------------------
SF.Net email is sponsored by:
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Smartfrog-developer mailing list
Smartfrog-developer@...
https://lists.sourceforge.net/lists/listinfo/smartfrog-developer



-----Original Message-----
From: smartfrog-developer-bounces@... [mailto:smartfrog-d=
eveloper-bounces@...] On Behalf Of Steve Loughran
Sent: 06 December 2007 13:15
Cc: smartfrog-developer
Subject: Re: [Smartfrog-developer] Questions about SmartFrog

Zhang Qian wrote:
> Hi All,
>
> Thanks for your replies about this topic.
>
> I'd like to share more details about my cluster with you.
> As you know, it's a cluster includes hundreds of nodes. We divide
> these nodes into
> two categories: management nodes and computing nodes.

I see. We've tend to prefer the tactic of letting any node become a
master (with agreement), because stops you having to decide which
machines are in charge. Whatever boots up first can take over.

> For computing nodes, they just run the task arranged to them, do not
> have management roles, so we don't care it in this case.

OK -the workers are expected to fail and are told what to do; if they go
away then something else gets the job.

>
> For management nodes, we have a dozen of this kind of nodes in the
> cluster. Only
> one of them is the master node whose responsibility is to manage the
> entirely cluster, others are just the master candidates. The reason we
> do it in this way is to avoid single point failure, once the master
> node fails, a master candidate will take over its job, and become the
> new master node. So we have the heartbeat mechanism to detect the node
> status to realize fail-over.

OK. You're giving one machine charge of the resource management problem,
but by sharing the data amongst candiates, if the master goes away you
can have an election of some sort to decide who is the new master.



> Now there is a limitation: our cluster relies on shared-file system(such =
as NFS)
> which can be accessed by all the management nodes.That means all the conf=
ig
> files placed on the shared-file system, all the management nodes need the=
se
> config files. It's the master node's responsibilityto update these config=
 file
> according to user's request, after a fail-over, the new master node
> will read these
> config file to know the latest configuration.

ah, so
  1. the NFS filestore is a failure point
  2. you need to save the configuration to a filesystem that doesnt go
out of its way to enable locking


>
> Now we want to remove the shared-file system dependency, each management
> node has config files in its local file system. So obviously, we need
> a mechanism
> to synchronize these config files on all the management nodes. That's
> why I asked
> that questions.

> I don't know whether there is a inbuilt component or service can
> provide this kind of mechanism in SmartFrog. Certainly I will
> investigate Anubis first, thanks for your sharing.

This is what anubis is designed for, to make a cluster out of a set of
machines on a LAN. The papers and Paul can provide more details.

 > In addition, we have had a management console for user which will
 > communicate with our daemon in the master node, and deliver config
 > change to that daemon.
 > After receive the config change, this daemon will verify and activate
 > the change first,
 > then write it into the config file placed on the shared-file system.

I see. How does the management console deal with failure of the master?
Does it discover it using some discovery protocol, or is the active
master expected to update a dynamic DNS entry?




-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
Smartfrog-developer mailing list
Smartfrog-developer@...
https://lists.sourceforge.net/lists/listinfo/smartfrog-developer



-----Original Message-----
From: smartfrog-developer-bounces@... [mailto:smartfrog-d=
eveloper-bounces@...] On Behalf Of Zhang Qian
Sent: 06 December 2007 03:31
To: Steve Loughran
Cc: smartfrog-developer
Subject: Re: [Smartfrog-developer] Questions about SmartFrog

In addition, we have had a management console for user which will
communicate with our daemon in the master node, and deliver config
change to that daemon.
After receive the config change, this daemon will verify and activate
the change first,
then write it into the config file placed on the shared-file system.

This is what we are doing, but we want to remove shared-file system depende=
ncy.



Thanks and Regards,
Qian

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
Smartfrog-developer mailing list
Smartfrog-developer@...
https://lists.sourceforge.net/lists/listinfo/smartfrog-developer

-----Original Message-----
From: smartfrog-developer-bounces@... [mailto:smartfrog-d=
eveloper-bounces@...] On Behalf Of Zhang Qian
Sent: 06 December 2007 02:58
To: Steve Loughran
Cc: smartfrog-developer
Subject: Re: [Smartfrog-developer] Questions about SmartFrog

Hi All,

Thanks for your replies about this topic.

I'd like to share more details about my cluster with you.
As you know, it's a cluster includes hundreds of nodes. We divide
these nodes into
two categories: management nodes and computing nodes.

For computing nodes, they just run the task arranged to them, do not
have management roles, so we don't care it in this case.

For management nodes, we have a dozen of this kind of nodes in the
cluster. Only
one of them is the master node whose responsibility is to manage the
entirely cluster, others are just the master candidates. The reason we
do it in this way is to avoid single point failure, once the master
node fails, a master candidate will take over its job, and become the
new master node. So we have the heartbeat mechanism to detect the node
status to realize fail-over.

Now there is a limitation: our cluster relies on shared-file system(such as=
 NFS)
which can be accessed by all the management nodes.That means all the config
files placed on the shared-file system, all the management nodes need these
config files. It's the master node's responsibilityto update these config f=
ile
according to user's request, after a fail-over, the new master node
will read these
config file to know the latest configuration.

Now we want to remove the shared-file system dependency, each management
node has config files in its local file system. So obviously, we need
a mechanism
to synchronize these config files on all the management nodes. That's
why I asked
that questions.

I don't know whether there is a inbuilt component or service can
provide this kind of mechanism in SmartFrog. Certainly I will
investigate Anubis first, thanks for your sharing.


Regards,
Qian

-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell.  From the desktop to the data center, Linux is going
mainstream.  Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
Smartfrog-developer mailing list
Smartfrog-developer@...
https://lists.sourceforge.net/lists/listinfo/smartfrog-developer

SGkgSnVsaW8sCgpUaGUgY29uZmlndXJhdGlvbiBkYXRhIG9mIG15IGNsdXN0ZXIgYXJlIHNtYWxs
IHNldHMgb2YgYXR0cmlidXRlIHZhbHVlCnBhaXJzLCBub3QgbG90cyBvZiBkYXRhLiBUaGUgZGF0
YSBhbW91bnQgaXMgbm90IGxhcmdlLCBidXQgd2UgcmVhbGx5Cm5lZWQgdGhlIHJlbGlhYmlsaXR5
LgoKVXN1YWxseSwgSSBtYWtlIGNvbmZpZyBjaGFuZ2UgaW4gdGhlIG1hbmFnZW1lbnQgY29uc29s
ZSBvZiBteSBjbHVzdGVyLAp0aGVuIHRoaXMgY29uc29sZSB3aWxsIGNvbW11bmljYXRlIHdpdGgg
dGhlIGRhZW1vbiBpbiB0aGUgbWFzdGVyIG5vZGUsCmFuZCBzZW5kIHRoZSBjb25maWcgY2hhbmdl
IHRvIGl0LiBUaGUgZGFlbW9uIHdpbGwgYWN0aXZhdGUgdGhlIGNoYW5nZQphbmQgd3JpdGUgdGhl
bSBpbiB0aGUgY29uZmlnIGZpbGUgc3RvcmVkIGluIHRoZSBORlMuIFRoZW4gb3RoZXIKbWFuYWdl
bWVudCBub2RlcyB3aWxsIGFsc28gc2VlIHRoZSBjaGFuZ2VzLiBCdXQgb2J2aW91c2x5LCBORlMg
Y291bGRlCmJlIGEgc2luZ2xlLXBvaW50IGZhaWx1cmUgb2YgbXkgY2x1c3Rlci4KCk5vdyBJIGFt
IHRyeWluZyB0byBjaGFuZ2UgdGhpcyBmbG93LiBUaGUgY29uZmlnIGNoYW5nZSBJIG1ha2UgaW4g
dGhlCm1hbmFnZW1lbnQgY29uc29sZSB3aWxsIGJlIHNhdmVkIGFzIGEgLnNmIGZpbGUsIHRoZW4g
SSB3aWxsIHJ1biBteSBvd24KU21hcnRGcm9nIGNvbXBvbmVudCB3aGljaCBleHRlbmRzIHNvbWUg
U21hcnRGcm9nIGluYnVpbHQgc2VydmljZXMuClRoaXMgY29tcG9uZW50IHdpbGwgZ2V0IHRoZSBj
b25maWcgY2hhbmdlIGJ5IHBhcnNpbmcgdGhlIC5zZiBmaWxlIGFuZApzZW5kIHRoZSBjaGFuZ2Ug
dG8gdGhlIGRhZW1vbiBpbiBtYXN0ZXIgbm9kZS4gVGhlIGRhZW1vbiB3aWxsIGFjdGl2YXRlCnRo
aXMgY2hhbmdlLCB0aGVuIHRoZSBjb21wb25lbnQgSSBtZW50aW9uZWQgYmVmb3JlIHdpbGwgd3Jp
dGUgdGhlCmNoYW5nZSB0byB0aGUgbG9jYWwgZmlsZSwgYW5kIHByb3BhZ2F0ZSB0aGlzIGZpbGUg
dG8gYWxsIHRoZQptYW5hbmdlbWVudCBub2Rlcy4KCkFueSBzdWdnZXN0aW9ucyBhYm91dCB0aGlz
IGFwcHJvYWNoPzotKQpUaGFua3OjoQoKCgpSZWdhcmRzLApRaWFuCg==

VGhlIGNvbmZpZ3VyYXRpb24gZGF0YSBvZiBteSBjbHVzdGVyIGFyZSBzbWFsbCBzZXRzIG9mIGF0
dHJpYnV0ZSB2YWx1ZQpwYWlycywgbm90IGxvdHMgb2YgZGF0YS4gVGhlIGRhdGEgYW1vdW50IGlz
IG5vdCBsYXJnZSwgYnV0IHdlIHJlYWxseQpuZWVkIHRoZSByZWxpYWJpbGl0eS4KClVzdWFsbHks
IEkgbWFrZSBjb25maWcgY2hhbmdlIGluIHRoZSBtYW5hZ2VtZW50IGNvbnNvbGUgb2YgbXkgY2x1
c3RlciwKdGhlbiB0aGlzIGNvbnNvbGUgd2lsbCBjb21tdW5pY2F0ZSB3aXRoIHRoZSBkYWVtb24g
aW4gdGhlIG1hc3RlciBub2RlLAphbmQgc2VuZCB0aGUgY29uZmlnIGNoYW5nZSB0byBpdC4gVGhl
IGRhZW1vbiB3aWxsIGFjdGl2YXRlIHRoZSBjaGFuZ2UKYW5kIHdyaXRlIHRoZW0gaW4gdGhlIGNv
bmZpZyBmaWxlIHN0b3JlZCBpbiB0aGUgTkZTLiBUaGVuIG90aGVyCm1hbmFnZW1lbnQgbm9kZXMg
d2lsbCBhbHNvIHNlZSB0aGUgY2hhbmdlcy4gQnV0IG9idmlvdXNseSwgTkZTIGNvdWxkZQpiZSBh
IHNpbmdsZS1wb2ludCBmYWlsdXJlIG9mIG15IGNsdXN0ZXIuCgpOb3cgSSBhbSB0cnlpbmcgdG8g
Y2hhbmdlIHRoaXMgZmxvdy4gVGhlIGNvbmZpZyBjaGFuZ2UgSSBtYWtlIGluIHRoZQptYW5hZ2Vt
ZW50IGNvbnNvbGUgd2lsbCBiZSBzYXZlZCBhcyBhIC5zZiBmaWxlLCB0aGVuIEkgd2lsbCBydW4g
bXkgb3duClNtYXJ0RnJvZyBjb21wb25lbnQgd2hpY2ggZXh0ZW5kcyBzb21lIFNtYXJ0RnJvZyBp
bmJ1aWx0IHNlcnZpY2VzLgpUaGlzIGNvbXBvbmVudCB3aWxsIGdldCB0aGUgY29uZmlnIGNoYW5n
ZSBieSBwYXJzaW5nIHRoZSAuc2YgZmlsZSBhbmQKc2VuZCB0aGUgY2hhbmdlIHRvIHRoZSBkYWVt
b24gaW4gbWFzdGVyIG5vZGUuIFRoZSBkYWVtb24gd2lsbCBhY3RpdmF0ZQp0aGlzIGNoYW5nZSwg
dGhlbiB0aGUgY29tcG9uZW50IEkgbWVudGlvbmVkIGJlZm9yZSB3aWxsIHdyaXRlIHRoZQpjaGFu
Z2UgdG8gdGhlIGxvY2FsIGZpbGUsIGFuZCBwcm9wYWdhdGUgdGhpcyBmaWxlIHRvIGFsbCB0aGUK
bWFuYW5nZW1lbnQgbm9kZXMuCgpBbnkgc3VnZ2VzdGlvbnMgYWJvdXQgdGhpcyBhcHByb2FjaD86
LSkKVGhhbmtzo6EKCgoKUmVnYXJkcywKUWlhbgo=

VGhlIGNvbmZpZ3VyYXRpb24gZGF0YSBvZiBteSBjbHVzdGVyIGFyZSBzbWFsbCBzZXRzIG9mIGF0
dHJpYnV0ZSB2YWx1ZQpwYWlycywgbm90IGxvdHMgb2YgZGF0YS4gVGhlIGRhdGEgYW1vdW50IGlz
IG5vdCBsYXJnZSwgYnV0IHdlIHJlYWxseQpuZWVkIHRoZSByZWxpYWJpbGl0eS4KClVzdWFsbHks
IEkgbWFrZSBjb25maWcgY2hhbmdlIGluIHRoZSBtYW5hZ2VtZW50IGNvbnNvbGUgb2YgbXkgY2x1
c3RlciwKdGhlbiB0aGlzIGNvbnNvbGUgd2lsbCBjb21tdW5pY2F0ZSB3aXRoIHRoZSBkYWVtb24g
aW4gdGhlIG1hc3RlciBub2RlLAphbmQgc2VuZCB0aGUgY29uZmlnIGNoYW5nZSB0byBpdC4gVGhl
IGRhZW1vbiB3aWxsIGFjdGl2YXRlIHRoZSBjaGFuZ2UKYW5kIHdyaXRlIHRoZW0gaW4gdGhlIGNv
bmZpZyBmaWxlIHN0b3JlZCBpbiB0aGUgTkZTLiBUaGVuIG90aGVyCm1hbmFnZW1lbnQgbm9kZXMg
d2lsbCBhbHNvIHNlZSB0aGUgY2hhbmdlcy4gQnV0IG9idmlvdXNseSwgTkZTIGNvdWxkZQpiZSBh
IHNpbmdsZS1wb2ludCBmYWlsdXJlIG9mIG15IGNsdXN0ZXIuCgpOb3cgSSBhbSB0cnlpbmcgdG8g
Y2hhbmdlIHRoaXMgZmxvdy4gVGhlIGNvbmZpZyBjaGFuZ2UgSSBtYWtlIGluIHRoZQptYW5hZ2Vt
ZW50IGNvbnNvbGUgd2lsbCBiZSBzYXZlZCBhcyBhIC5zZiBmaWxlLCB0aGVuIEkgd2lsbCBydW4g
bXkgb3duClNtYXJ0RnJvZyBjb21wb25lbnQgd2hpY2ggZXh0ZW5kcyBzb21lIFNtYXJ0RnJvZyBp
bmJ1aWx0IHNlcnZpY2VzLgpUaGlzIGNvbXBvbmVudCB3aWxsIGdldCB0aGUgY29uZmlnIGNoYW5n
ZSBieSBwYXJzaW5nIHRoZSAuc2YgZmlsZSBhbmQKc2VuZCB0aGUgY2hhbmdlIHRvIHRoZSBkYWVt
b24gaW4gbWFzdGVyIG5vZGUuIFRoZSBkYWVtb24gd2lsbCBhY3RpdmF0ZQp0aGlzIGNoYW5nZSwg
dGhlbiB0aGUgY29tcG9uZW50IEkgbWVudGlvbmVkIGJlZm9yZSB3aWxsIHdyaXRlIHRoZQpjaGFu
Z2UgdG8gdGhlIGxvY2FsIGZpbGUsIGFuZCBwcm9wYWdhdGUgdGhpcyBmaWxlIHRvIGFsbCB0aGUK
bWFuYW5nZW1lbnQgbm9kZXMuCgpBbnkgc3VnZ2VzdGlvbnMgYWJvdXQgdGhpcyBhcHByb2FjaD86
LSkKVGhhbmtzo6EKClJlZ2FyZHMsClFpYW4K

VGhlIG1haWwgSSBzZW50IHRvIHByZXZpb3VzIHRocmVhZCBjYW4gbm90IGJlIGRpc3BsYXllZCBj
b3JyZWN0bHkuCkp1c3QgdHJ5IHRvIHN0YXJ0IGFub3RoZXIgdGhyZWFkLgoKCgpUaGUgY29uZmln
dXJhdGlvbiBkYXRhIG9mIG15IGNsdXN0ZXIgYXJlIHNtYWxsIHNldHMgb2YgYXR0cmlidXRlIHZh
bHVlCnBhaXJzLCBub3QgbG90cyBvZiBkYXRhLiBUaGUgZGF0YSBhbW91bnQgaXMgbm90IGxhcmdl
LCBidXQgd2UgcmVhbGx5Cm5lZWQgdGhlIHJlbGlhYmlsaXR5LgoKVXN1YWxseSwgSSBtYWtlIGNv
bmZpZyBjaGFuZ2UgaW4gdGhlIG1hbmFnZW1lbnQgY29uc29sZSBvZiBteSBjbHVzdGVyLAp0aGVu
IHRoaXMgY29uc29sZSB3aWxsIGNvbW11bmljYXRlIHdpdGggdGhlIGRhZW1vbiBpbiB0aGUgbWFz
dGVyIG5vZGUsCmFuZCBzZW5kIHRoZSBjb25maWcgY2hhbmdlIHRvIGl0LiBUaGUgZGFlbW9uIHdp
bGwgYWN0aXZhdGUgdGhlIGNoYW5nZQphbmQgd3JpdGUgdGhlbSBpbiB0aGUgY29uZmlnIGZpbGUg
c3RvcmVkIGluIHRoZSBORlMuIFRoZW4gb3RoZXIKbWFuYWdlbWVudCBub2RlcyB3aWxsIGFsc28g
c2VlIHRoZSBjaGFuZ2VzLiBCdXQgb2J2aW91c2x5LCBORlMgY291bGRlCmJlIGEgc2luZ2xlLXBv
aW50IGZhaWx1cmUgb2YgbXkgY2x1c3Rlci4KCk5vdyBJIGFtIHRyeWluZyB0byBjaGFuZ2UgdGhp
cyBmbG93LiBUaGUgY29uZmlnIGNoYW5nZSBJIG1ha2UgaW4gdGhlCm1hbmFnZW1lbnQgY29uc29s
ZSB3aWxsIGJlIHNhdmVkIGFzIGEgLnNmIGZpbGUsIHRoZW4gSSB3aWxsIHJ1biBteSBvd24KU21h
cnRGcm9nIGNvbXBvbmVudCB3aGljaCBleHRlbmRzIHNvbWUgU21hcnRGcm9nIGluYnVpbHQgc2Vy
dmljZXMuClRoaXMgY29tcG9uZW50IHdpbGwgZ2V0IHRoZSBjb25maWcgY2hhbmdlIGJ5IHBhcnNp
bmcgdGhlIC5zZiBmaWxlIGFuZApzZW5kIHRoZSBjaGFuZ2UgdG8gdGhlIGRhZW1vbiBpbiBtYXN0
ZXIgbm9kZS4gVGhlIGRhZW1vbiB3aWxsIGFjdGl2YXRlCnRoaXMgY2hhbmdlLCB0aGVuIHRoZSBj
b21wb25lbnQgSSBtZW50aW9uZWQgYmVmb3JlIHdpbGwgd3JpdGUgdGhlCmNoYW5nZSB0byB0aGUg
bG9jYWwgZmlsZSwgYW5kIHByb3BhZ2F0ZSB0aGlzIGZpbGUgdG8gYWxsIHRoZQptYW5hbmdlbWVu
dCBub2Rlcy4KCkFueSBzdWdnZXN0aW9ucyBhYm91dCB0aGlzIGFwcHJvYWNoPzotKQpUaGFua3Oj
oQoKCgpSZWdhcmRzLApRaWFuCg==

HI Qian,

I think your solution should work.

My suggestion or improvement would be to use multicast to propagate the cha=
nges directly to all the management nodes and instead of writing it to a fi=
le sending the file why not to send the .sf file directly to all the manage=
ment nodes.


Another thing that you could do is to use the console to send the new .sf f=
ile to all the nodes (you could do this in text form of in ComponentDescrip=
tion form if the recipients are sf Components). The master node would act o=
n the new configuration and the other nodes would just store the configurat=
ion locally. In this way each management node would have a cached copy of t=
he cluster configuration.

Another thing that you will need to add is how to recover a management node=
 from a failure, for example should a recovered node get a full copy of the=
 entire configuration from the master node or peer or should it try to re-s=
ynch it config data. The first option is probably easier.

This is quite easy to do with Anubis because its protocol guaranties that w=
hat you received is the same that the other nodes in your "partition" see a=
nd it simplifies what you have to do to avoid errors when synchronizing you=
r configuration data in the cluster but as I said, it could work equally we=
ll with your own protocol or with some other form of multicast and extra pr=
ogramming.

Regards,

Julio Guijarro


-----Original Message-----
From: Zhang Qian [mailto:zhq527725@...]
Sent: 09 December 2007 02:15
To: Guijarro, Julio
Cc: Steve Loughran; smartfrog-developer; smartfrog-users@....=
net
Subject: Re: [Smartfrog-developer] Questions about SmartFrog

Hi Julio,

The configuration data of my cluster are small sets of attribute value
pairs, not lots of data. The data amount is not large, but we really
need the reliability.

Usually, I make config change in the management console of my cluster,
then this console will communicate with the daemon in the master node,
and send the config change to it. The daemon will activate the change
and write them in the config file stored in the NFS. Then other
management nodes will also see the changes. But obviously, NFS coulde
be a single-point failure of my cluster.

Now I am trying to change this flow. The config change I make in the
management console will be saved as a .sf file, then I will run my own
SmartFrog component which extends some SmartFrog inbuilt services.
This component will get the config change by parsing the .sf file and
send the change to the daemon in master node. The daemon will activate
this change, then the component I mentioned before will write the
change to the local file, and propagate this file to all the
manangement nodes.

Any suggestions about this approach?:-)
Thanks=1B$B!*=1B(B



Regards,
Qian

Hi Julio,

> My suggestion or improvement would be to use multicast to propagate the
> changes directly to all the management nodes and instead of writing it to a file
> sending the file why not to send the .sf file directly to all the management nodes.
According to our fail-over solution, after fail-over, daemons in the
new master will parse the configuration file to get the latest
configuration. Our daemons can not understand .sf file:-) So I neet to
propagate configuration files to all the management nodes.
Btw, could you tell me more details about the multicast service
provided by SmartFrog? Is it reliable? Or just a wrapper of normal UDP
multicast?

> Another thing that you could do is to use the console to send the new .sf file to
> all the nodes (you could do this in text form of in ComponentDescription form if
> the recipients are sf Components). The master node would act on the new
> configuration and the other nodes would just store the configuration locally. In
> this way each management node would have a cached copy of the cluster
> configuration.
I should propagate configuration files only after our daemon has
verified and activated the changes. Wrong changes should not be
written into configuration file and progagated to other nodes.

> This is quite easy to do with Anubis because its protocol guaranties that what
> you received is the same that the other nodes in your "partition" see and it
> simplifies what you have to do to avoid errors when synchronizing your
> configuration data in the cluster but as I said, it could work equally well with your
> own protocol or with some other form of multicast and extra programming.
Did you mean using ssh/ftp or multicast plus our own monitoring
mechanism can also reach the same level as Anubis. I just think Anubis
is a little big and complex for my requirements.


Regards,
Qian