> > 1) Where is the information of shared hardwareand software resources stored ? ( Also if I have to add a new resource at any node for sharing in the whole cluster how do i do that, and where will be this information be stored ?
Resource sharing is generally automatic. If you mount devfs on /devfs,
you can find devices for any node in /devfs/nodeX/. A filesystem mounted
on one node is automatically seen on all nodes. All processes on all
nodes can be seen in /proc. You can signal any process from any node.
FIFOs and message queues can be used to communicate between processes on
different nodes. So on and so forth...
None of this information is stored in a file. It's all discovered at the
appropriate time (booting a node, forking a process, opening a FIFO,
etc.) and stored in the kernel state of one or more nodes.
> > 2)In the absence of CFS, if my primary node goes down, will a secondary node take over or the whole cluster goes down.
Right now the only two choices for the shared root are CFS and GFS. CFS
does not currently allow failover to a secondary node. That will not be
possible until David Zafman finishes HA-CFS.
The alternative, GFS, allows failover to any other node in the cluster
that is designated a potential master. You can make a node a potential
master with the chnode command or by directly editing /etc/clustertab.
With the current implementation of GFS, however, you'll need a lock
server outside the cluster (unless you have DMEP-equipped hardware). The
lock server is a single point of failure. Once the OpenGFS project
integrates with a DLM, this will no longer be a problem.
> In the absence of CFS you will be using GFS right ? Earlier before we
> had CFS fully supported we were allowing the process migration to open
> reopen the files with name. Now i guess i looks for inode matching
> also. I am not sure about this. Brian Should be able to confirm. But i
> guess now you cannot build a ssi cluster with nodes having local
The inode matching is not an issue for GFS. It just means that a shared
root in now an absolute requirement, unlike the earlier days when each
node could have it's own root. Either CFS or GFS can provide the shared
> > 3)How do I add and remove nodes both in the presence and absence of CFS?
> Same here either CFS or GFS. Any adding node is == modifying
> /etc/clustertab and rebuilding tftp boot image in the case of SSI
Adding nodes is described in INSTALL and INSTALL.gfs. As Aneesh says,
it's the same for both. Removing nodes is not documented, yet. For now
it involves deleting the appropriate line from /etc/clustertab then
running cluster_mkinitrd and cluster_lilo.
> > 4) If I dont want to use CFS what are the features I cannot use?
The only advantage of CFS over GFS is that you don't need expensive
shared disk hardware (must be connected to every node). If you have
access to that hardware, then GFS is generally better. Filesystem
performance is quicker from every node because of the direct connection
to the disk, and the cluster can recover from losing any node in the
cluster (although losing the lock server will cause the cluster to
> > 5)Is it possible to have more than one primary node?
> Yes they are called potential masters. But for the time being is
> meaningful when used along with GFS. CFS doesn't support fail over as
> of now
There's only one active master at a time. Almost all of the information
stored on the master node is duplicated elsewhere in the cluster, so the
cluster can recover quickly from the loss of the master. The one
exception for now, as Aneesh says, is the root filesystem if you use
CFS. That's because the filesystem is only accessible from the master
node. This will change after David finishes HA-CFS.
> > 6) Is it possible to boot from a node other than the primary node?
> You can put a different DHCP server and put the network boot image
> there. But the "/" will be provided by the primary.
Yup. The tools I wrote to support netboot (addnode, chnode,
cluster_lilo, etc.) are currently too limited to support other
configurations, but it is definitely possible.
> Bruce was talking last time about a state in which i can leave a file
> system in a unavailable state and get it served by some other node. ie,
> initially when the system boot /home will be served by node1 now after
> the secondary master ( potential master node ) 2 join you will leave
> /home in unavailable state and later ask node 2 to server /home. This
> way IO can be balanced across multiple nodes in the cluster. That will
> require a sort of layering of CFS over GFS I guess.
This will require a fair amount of development work to make this work.
It's not clear that it's a high enough priority for any of the current
developers to make this happen anytime soon.
Brian Watson | "Now I don't know, but I been told it's
Software Developer | hard to run with the weight of gold,
Open SSI Clustering Project | Other hand I heard it said, it's
Hewlett-Packard Company | just as hard with the weight of lead."
| -Robert Hunter, 1970