#3863 makedns to support SN as dns failover for MN

[Jing Sun]

Required by Arif on the mailinglist. Here is the statement:

---

The purpose of having multiple entries including SN(e.g. MN,SN1,SN2) in networks.nameservers is so that if the MN goes down, then we still have a SN to be able to get DNS and host resolution.

Let me elaborate on another scenario, where having multiple nameservers, and primarily SN being the choice,

I run makehosts and makedns on all service nodes, I was in essence making the SN as a temporary failover for as many services that xCAT MN was providing. The only service not being to failover were the commands that were commands that were tied to the database, which is most of them.

Actually one of our customers due to the nature of the remote access, both the customer and ourselves accidentally rebooted the MN, and jobs happily were able to resolve the names due to the fact the SN was the shadow host for the scheduler and had a named configuration. We tend to find that MN tends to be also the login node, so to have a fail-over like this can be imperative.

---

In xcat dns, the dns server is set up on the management node, while the service node only acts as a dns forwarder, that means if a new host needs to be resolved, then the service node will forward this dns request to the management node, so if the management node goes down, that will not work.

This is a new requirement, we need to set an individual dns server on SNs, so when the MN goes down, SN can be a failover to keep the jobs on compute nodes working.

[Jing Sun]

• Description has changed:

Diff:

--- old
+++ new
@@ -8,6 +8,7 @@
 I run makehosts and makedns on all service nodes, I was in essence making the SN as a temporary failover for as many services that xCAT MN was providing. The only service not being to failover were the commands that were commands that were tied to the database, which is most of them.

 Actually one of our customers due to the nature of the remote access, both the customer and ourselves accidentally rebooted the MN, and jobs happily were able to resolve the names due to the fact the SN was the shadow host for the scheduler and had a named configuration. We tend to find that MN tends to be also the login node, so to have a fail-over like this can be imperative.
+
 -----------------------------

 In xcat dns, the dns server is set up on the management node, while the service node only acts as a dns forwarder, that means if a new host needs to be resolved, then the service node will forward this dns request to the management node, so if the management node goes down, that will not work. 

[Jing Sun]

Here are some discussions, FYI.

From Bruce:
Arif's requirement is valid and i have heard it many times, so we need to come up with a solution. Maybe we need an option on makedns to configure all of the SN DNS instances the same way we configure the MN DNS: have all of the mappings from /etc/hosts (from the MN), and be configured with site.forwarders. In this mode makedns would be similar to makedhcp, in that it would spawn an xcatd request to each of the SNs to configure the DNS on the SNs. Thoughts? Other ideas?

From Sun Jing:
for the implementation to set up an individual dns server on SN, I think this could be something like site.disjointdhcps, it's default to "0", means to set up dns server on MN and dns forwarder cache on SNs, if it's set to "1" means to set up individual dns servers on MN and SNs. while if the SN does not have external network connection, then maybe it does not make sense to set the site.forwarders to the dns server on SN?

Another potential issue might be passing /etc/hosts content from MN to SNs via XML in the scaling cluster, this seems not a xcat strategy - SN should get the resources from database on a file. I'm now investigating if there is any method inside dns to make syncup between dns servers, for example, dig, or rndc, or something else, if so, we can keep the logic to set up dns server on MN, then make syncup on SN without /etc/hosts, will try more.

From Arif:
You probably don't need a full implementation of named on the SN, you can have each of the zones to be slaves to the MN, and therefore once a change is made via rndc on the MN then it will automatically get pushed out. Although an initial configuration will be required on the SN to be slaves, and from which nodes to expect updates from. The MN will require extra options that will say to push to the SN, like any standard DNS master/salve setups.

[Jing Sun]

This bug was fixed in xcat 2.8.4 work item "makedns - support DNS master/slave configuration in hierarchy", now is ready for testing. Here is the materials for your information.

1. mini-design: https://sourceforge.net/apps/mediawiki/xcat/index.php?title=DNS_master/slave_support

2. schema change:
   #tabdump -d servicenode |grep nameserver
   #man servicenode (search "nameserver")

3. documentation: https://sourceforge.net/apps/mediawiki/xcat/index.php?title=Cluster_Name_Resolution#Option_.233:_Run_DNS_on_Management_Node_and_Service_Nodes

WeiHua will test this work item, so please verify this bug together(this bug should be one test case for you).

[Arif Ali]

Hi Chaps,

not sure if this is the best place to inform my issues on this feature

Thanks for this fantastic work, and it works GREAT; one minor issue with hierarchy

How do I get to the xcatd to grab the HA IP rather than the server IP. i.e. I have xCAT configured in HA, the node facing IP to the MN from the SN are in the range 172.27.34.0/26

The primary IPs of the MN are

172.27.34.7 crb111
172.27.34.8 crb113

the HA ip is 172.27.34.11

At this moment, using this feature, it only adds 172.27.34.7 to the resulting named.conf, even though crb113 is the active master ; I would either prefer to add both IPs, or just the HA IP.

Any ideas on this please?

[Guang Cheng Li]

Xiao Peng, could you respond to Arif's question? Thanks.

[XiaoPeng Wang]

Do you mean both of 172.27.34.7 and 172.27.34.8 are active? Could you check the nameserver in /etc/resolv.conf on SN that only 172.27.34.7 was included? Or both?

And stranage that are your HA IP and primary IP in same network?

[Arif Ali]

no, only one of them is active; So I see where the problem is, I had provisioned the service node earlier, where the MN was pointing back to only the primary. I needed to add only the HA IP.

Once I changed the resolv.conf to point to 172.27.34.11, the masters attribute in the zones were correct

on another note, on the MN, are we not supposed to have "also-notify" parameters so that when the change happens on the MN, it gets propagated to the relevant SN?

[XiaoPeng Wang]

Change the nameserver to 172.27.32.11 is correct to do. If you want to have multiple ones from resolve.conf, xcat code must be changed, we only support the first one so far.

[Lissa Valletta]

Could you explain this question "on another note, on the MN, are we not supposed to have "also-notify" parameters so that when the change happens on the MN, it gets propagated to the relevant SN?"
also-notify on what?

[Arif Ali]

let me explain

correct me, if I am wrong

in named.conf on the MN, for each zone, so that the entries can be propagated to the named slaves (servicenodes) we have a field also-notify similar to the example below

    also-notify {
                sn01;
                sn02;
                ...;
        };

[XiaoPeng Wang]

It does support the also-notify, but looks like the whole also-notify section cannot be created if you want to update an old named.conf which does not have this section.

If possible, you could try to run 'makedns -n' to recreate the configuration file.

[Arif Ali]

should have thought about doing that. Thanks ;)