Menu
▾
▴
opensaf-devel
|
From: <mat...@or...> - 2014-03-26 15:39:20
|
Summary: clm: modify log string and respawn clmna for failed node join requests [#816]
Review request for Trac Ticket(s): #816
Peer Reviewer(s): HansF and/or RameshB
Pull request to: <<LIST THE PERSON WITH PUSH ACCESS HERE>>
Affected branch(es): opensaf-4.4.x, default
Development branch: <<IF ANY GIVE THE REPO URL>>
--------------------------------
Impacted area Impact y/n
--------------------------------
Docs n
Build system n
RPM/packaging n
Configuration files n
Startup scripts n
SAF services y
OpenSAF services n
Core libraries n
Samples n
Tests n
Other n
Comments (indicate scope for each "y" above):
---------------------------------------------
This patch series do the following:
patch 1 of 3 : Adds null check to avoid crash during node requests from unconfigured/misconfigured nodes.
patch 2 of 3 : Improves log strings for failed node join requests in scenarios involving
- unconfigured/misconfigured node join request
- duplicate node join requests
- nodes with connectivity problems
patch 3 of 3 : clmna shall pass on the error code(for configuration problems also) to NID
such that NID respawns CLMNA
changeset 5de25467f5d25ccdff0bbbc7d9f0fbe95cf43c2e
Author: Mathivanan N.P.<mat...@or...>
Date: Wed, 26 Mar 2014 20:14:34 -0400
clm: add null check for unconfigured node join request [#816] When a node
tries to join with a node name that is not configured or just gets simply
mis-configured, then CLM crashes causing due to lack of null check during
the processing of node join request. When this issue happens, the following
logging is done by CLM:
Mar 26 18:44:52 SC-1 local0.err osafclmd[418]: ER CLM NodeName: 'SC-3' doesn't match entry in imm.xml. Specify a correct node
name in/etc/opensaf/node_name.
This patch 1 of 1 adds a null check such that CLM server doesn't crashes.
changeset bc69e7c6d6f9f5bf8eaaf81d92a130ac65928f4e
Author: Mathivanan N.P.<mat...@or...>
Date: Wed, 26 Mar 2014 20:30:32 -0400
clm: change log message for unconfigured nodes, connectivity problems [#816]
When a node join request from an unconfigured/misconfigured node is
attempted, then CLM prints the following log message:
Mar 26 18:44:52 SC-1 local0.err osafclmd[418]: ER CLM NodeName: 'SC-3' doesn't match entry in
imm.xml. Specify a correct node name in/etc/opensaf/node_name.
This log message is currently categorized as ERROR. However, there are suggestions to
improve the current log message such that the message more closely depicts
the case of unconfigured or misconfigured nodes. The log string is now
changed as below:
Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO CLM NodeName: 'PL-8' is not a configured cluster node.
Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO /etc/opensaf/node_name should contain the
rdn value of a configured CLM node object name
In the case of problems with connectivity problems, the following modified
log strings are added:
"Send timed out. Check status(network, process) of ACTIVE controller."
and
"Send failed: %u. Check network connectivity"
changeset 11f7ed6e040e6e42f6acad397e14a14a8576aadb
Author: Mathivanan N.P.<mat...@or...>
Date: Wed, 26 Mar 2014 20:45:34 -0400
clm: clmna to be respawned by nid when node join request fails [#816]
When a node join request for an unconfigured/misconfigured node or when a node join
request with a duplicate node_name is attempted, then clmna should report
those errors to NID such that NID attempts to respawan clmna. With the
introduction of this change, the following happens(can be seen in the
syslog) in the case of a unconfigured/misconfigured node join request:
At the ACTIVE controller syslog:
Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO CLM NodeName: 'PL-8' is not a configured cluster node.
Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO /etc/opensaf/node_name
should contain the rdn value of a configured CLM node object name
At the unconfigured/misconfigured node, the syslog will be like as below:
Mar 26 19:03:54 PL-3 local0.notice osafclmna[871]: Started
Mar 26 19:03:54 PL-3 local0.err osafclmna[871]: ER PL-8 is not a configured node
Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Failed DESC:CLMNA
Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Going for recovery
Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Trying To RESPAWN
/usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #1
Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Sending SIGKILL to CLMNA, pid=868
Mar 26 19:03:54 PL-3 local0.err osafclmna[871]: ER Exiting
Mar 26 19:03:54 PL-3 local0.notice osafclmna[871]: exiting for shutdown
Mar 26 19:04:09 PL-3 local0.notice osafclmna[895]: Started
Mar 26 19:04:09 PL-3 local0.err osafclmna[895]: ER PL-8 is not a configured node
Mar 26 19:04:09 PL-3 local0.err osafclmna[895]: ER Exiting
Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Could Not RESPAWN CLMNA
Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Failed DESC:CLMNA
Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Trying To RESPAWN
/usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #2
Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Sending SIGKILL to CLMNA, pid=892
Mar 26 19:04:09 PL-3 local0.notice osafclmna[895]: exiting for shutdown
Mar 26 19:04:24 PL-3 local0.notice osafclmna[919]: Started
Mar 26 19:04:25 PL-3 local0.err osafclmna[919]: ER PL-8 is not a configured node
Mar 26 19:04:25 PL-3 local0.err osafclmna[919]: ER Exiting
Mar 26 19:04:25 PL-3 local0.err opensafd[837]: ER Could Not RESPAWN CLMNA
Mar 26 19:04:25 PL-3 local0.err opensafd[837]: ER Failed DESC:CLMNA Mar 26
19:04:25 PL-3 local0.err opensafd[837]: ER FAILED TO RESPAWN Mar 26 19:04:27
PL-3 local0.notice osafclmna[919]: exiting for shutdown Mar 26 19:04:28 PL-3
local0.notice osafimmnd[864]: exiting for shutdown
For cases when a duplicate node join request comes, the following syslog
message will be seen at the ACTIVE controller:
Mar 26 19:07:43 SC-1 local0.err osafclmd[418]: ER Duplicate node join request for CLM node: 'SC-2'. Specify a unique node name in/etc/opensaf/node_name
Mar 26 19:07:59 SC-1 local0.err osafclmd[418]: ER Duplicate node join request for CLM node: 'SC-2'. Specify a unique node name in/etc/opensaf/node_name
And the following will be seen at the node on which the duplicate request is
attempted:
Mar 26 19:07:43 PL-3 local0.err osafclmna[1459]: ER SC-2 is already up. Specify a unique name in/etc/opensaf/node_name
Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Failed DESC:CLMNA
Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Going for recovery
Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #1
Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Sending SIGKILL to CLMNA, pid=1456
Mar 26 19:07:43 PL-3 local0.err osafclmna[1459]: ER Exiting
Mar 26 19:07:44 PL-3 local0.notice osafclmna[1459]: exiting for shutdown
Mar 26 19:07:59 PL-3 local0.notice osafclmna[1483]: Started
Mar 26 19:07:59 PL-3 local0.err osafclmna[1483]: ER SC-2 is already up. Specify a unique name in/etc/opensaf/node_name
Mar 26 19:07:59 PL-3 local0.err osafclmna[1483]: ER Exiting
Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Could Not RESPAWN CLMNA
Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Failed DESC:CLMNA
Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #2
Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Sending SIGKILL to CLMNA, pid=1480
Mar 26 19:07:59 PL-3 local0.notice osafclmna[1483]: exiting for shutdown
Complete diffstat:
------------------
osaf/services/saf/clmsv/clms/clms_evt.c | 60 +++++++++++++++++++++++++++++++++++++-----------------------
osaf/services/saf/clmsv/nodeagent/main.c | 15 +++++++++++----
2 files changed, 48 insertions(+), 27 deletions(-)
Testing Commands:
-----------------
- Start a cluster with 3 or more nodes.
- one of the nodes should contain /etc/opensaf/node_name
set to a node name that does not contain the rdn value
of the clm node as specified in nodes.cfg or during
dynamic addition of a clm node through IMM commands.
- The active should not crash.
- The log string should give an indiciation of the problem.
- CLMNA is re-instantiated on the node on which this misconfiguration
occurs.
Testing, Expected Results:
--------------------------
Same as above.
Conditions of Submission:
-------------------------
Ack from Hans and/or Ramesh.
Arch Built Started Linux distro
-------------------------------------------
mips n n
mips64 n n
x86 n n
x86_64 y y
powerpc n n
powerpc64 n n
Reviewer Checklist:
-------------------
[Submitters: make sure that your review doesn't trigger any checkmarks!]
Your checkin has not passed review because (see checked entries):
___ Your RR template is generally incomplete; it has too many blank entries
that need proper data filled in.
___ You have failed to nominate the proper persons for review and push.
___ Your patches do not have proper short+long header
___ You have grammar/spelling in your header that is unacceptable.
___ You have exceeded a sensible line length in your headers/comments/text.
___ You have failed to put in a proper Trac Ticket # into your commits.
___ You have incorrectly put/left internal data in your comments/files
(i.e. internal bug tracking tool IDs, product names etc)
___ You have not given any evidence of testing beyond basic build tests.
Demonstrate some level of runtime or other sanity testing.
___ You have ^M present in some of your files. These have to be removed.
___ You have needlessly changed whitespace or added whitespace crimes
like trailing spaces, or spaces before tabs.
___ You have mixed real technical changes with whitespace and other
cosmetic code cleanup changes. These have to be separate commits.
___ You need to refactor your submission into logical chunks; there is
too much content into a single commit.
___ You have extraneous garbage in your review (merge commits etc)
___ You have giant attachments which should never have been sent;
Instead you should place your content in a public tree to be pulled.
___ You have too many commits attached to an e-mail; resend as threaded
commits, or place in a public tree for a pull.
___ You have resent this content multiple times without a clear indication
of what has changed between each re-send.
___ You have failed to adequately and individually address all of the
comments and change requests that were proposed in the initial review.
___ You have a misconfigured ~/.hgrc file (i.e. username, email etc)
___ Your computer have a badly configured date and time; confusing the
the threaded patch review.
___ Your changes affect IPC mechanism, and you don't present any results
for in-service upgradability test.
___ Your changes affect user manual and documentation, your patch series
do not contain the patch that updates the Doxygen manual.
|
|
From: <mat...@or...> - 2014-03-26 15:39:21
|
osaf/services/saf/clmsv/clms/clms_evt.c | 48 ++++++++++++++++++--------------
1 files changed, 27 insertions(+), 21 deletions(-)
When a node tries to join with a node name that is not configured
or just gets simply mis-configured, then CLM crashes causing due
to lack of null check during the processing of node join request.
When this issue happens, the following logging is done by CLM:
Mar 26 18:44:52 SC-1 local0.err osafclmd[418]: ER CLM NodeName: 'SC-3' doesn't match entry in imm.xml.
Specify a correct node name in/etc/opensaf/node_name.
This patch 1 of 1 adds a null check such that CLM server doesn't crashes.
diff --git a/osaf/services/saf/clmsv/clms/clms_evt.c b/osaf/services/saf/clmsv/clms/clms_evt.c
--- a/osaf/services/saf/clmsv/clms/clms_evt.c
+++ b/osaf/services/saf/clmsv/clms/clms_evt.c
@@ -286,34 +286,40 @@ uint32_t proc_node_up_msg(CLMS_CB * cb,
nodeup_info->node_name.value);
}
- /* Retrieve IP information */
- if ((ip = (IPLIST *)ncs_patricia_tree_get(&clms_cb->iplist, (uint8_t *)&nodeid)) == NULL) {
- clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
- LOG_ER("IP information not found for: %s with node_id: %u", nodeup_info->node_name.value, nodeid);
- } else {
- if (ip->addr.length) { /* If length = 0, it is AF_TIPC. So process only IPv4 or 6 addresses */
- /* We might want to validate IP */
- if (ip_matched(ip->addr.family, ip->addr.value, node->node_addr.family,node->node_addr.value)) {
- clm_msg.info.api_resp_info.rc = SA_AIS_OK;
- } else {
- clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
- LOG_ER("IP address on %s is not matching the ipaddress in" PKGSYSCONFDIR "/imm.xml",
+ if (node != NULL) {
+ /* Retrieve IP information */
+ if ((ip = (IPLIST *)ncs_patricia_tree_get(&clms_cb->iplist, (uint8_t *)&nodeid)) == NULL) {
+ clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
+ LOG_ER("IP information not found for: %s with node_id: %u",
+ nodeup_info->node_name.value, nodeid);
+ } else {
+ if (ip->addr.length) { /* If length = 0, it is AF_TIPC. So process only IPv4 or 6 addresses */
+ /* We might want to validate IP */
+ if (ip_matched(ip->addr.family, ip->addr.value,
+ node->node_addr.family,node->node_addr.value)) {
+ clm_msg.info.api_resp_info.rc = SA_AIS_OK;
+ } else {
+ clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
+ LOG_ER("IP address on %s is not matching the ipaddress in" PKGSYSCONFDIR "/imm.xml",
nodeup_info->node_name.value);
+ }
}
}
- }
- if ((cb->node_id != nodeup_info->node_id) && (node->nodeup == SA_TRUE)){
- clm_msg.info.api_resp_info.rc = SA_AIS_ERR_EXIST;
- LOG_ER("Duplicate node join request for CLM node: '%s'. Specify a unique node name in" PKGSYSCONFDIR "/node_name",
- nodeup_info->node_name.value);
- } else
- clm_msg.info.api_resp_info.rc = SA_AIS_OK;
-
+ if ((cb->node_id != nodeup_info->node_id) && (node->nodeup == SA_TRUE)){
+ clm_msg.info.api_resp_info.rc = SA_AIS_ERR_EXIST;
+ LOG_ER("Duplicate node join request for CLM node: '%s'. "
+ "Specify a unique node name in" PKGSYSCONFDIR "/node_name",
+ nodeup_info->node_name.value);
+ } else
+ clm_msg.info.api_resp_info.rc = SA_AIS_OK;
+
+ } /* Node exists in DB */
+
if (clm_msg.info.api_resp_info.rc != SA_AIS_OK) {
clm_msg.evt_type = CLMSV_CLMS_TO_CLMA_API_RESP_MSG;
clm_msg.info.api_resp_info.type = CLMSV_CLUSTER_JOIN_RESP;
- clm_msg.info.api_resp_info.param.node_name = node_name;
+ clm_msg.info.api_resp_info.param.node_name = nodeup_info->node_name;
rc = clms_mds_msg_send(cb, &clm_msg, &evt->fr_dest, &evt->mds_ctxt, MDS_SEND_PRIORITY_HIGH,
NCSMDS_SVC_ID_CLMNA);
if (rc != NCSCC_RC_SUCCESS)
|
|
From: Hans F. <han...@er...> - 2014-04-03 10:50:48
|
Ack
/Hans
On 03/27/2014 02:13 AM, mat...@or... wrote:
> osaf/services/saf/clmsv/clms/clms_evt.c | 48 ++++++++++++++++++--------------
> 1 files changed, 27 insertions(+), 21 deletions(-)
>
>
> When a node tries to join with a node name that is not configured
> or just gets simply mis-configured, then CLM crashes causing due
> to lack of null check during the processing of node join request.
> When this issue happens, the following logging is done by CLM:
> Mar 26 18:44:52 SC-1 local0.err osafclmd[418]: ER CLM NodeName: 'SC-3' doesn't match entry in imm.xml.
> Specify a correct node name in/etc/opensaf/node_name.
> This patch 1 of 1 adds a null check such that CLM server doesn't crashes.
>
> diff --git a/osaf/services/saf/clmsv/clms/clms_evt.c b/osaf/services/saf/clmsv/clms/clms_evt.c
> --- a/osaf/services/saf/clmsv/clms/clms_evt.c
> +++ b/osaf/services/saf/clmsv/clms/clms_evt.c
> @@ -286,34 +286,40 @@ uint32_t proc_node_up_msg(CLMS_CB * cb,
> nodeup_info->node_name.value);
> }
>
> - /* Retrieve IP information */
> - if ((ip = (IPLIST *)ncs_patricia_tree_get(&clms_cb->iplist, (uint8_t *)&nodeid)) == NULL) {
> - clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
> - LOG_ER("IP information not found for: %s with node_id: %u", nodeup_info->node_name.value, nodeid);
> - } else {
> - if (ip->addr.length) { /* If length = 0, it is AF_TIPC. So process only IPv4 or 6 addresses */
> - /* We might want to validate IP */
> - if (ip_matched(ip->addr.family, ip->addr.value, node->node_addr.family,node->node_addr.value)) {
> - clm_msg.info.api_resp_info.rc = SA_AIS_OK;
> - } else {
> - clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
> - LOG_ER("IP address on %s is not matching the ipaddress in" PKGSYSCONFDIR "/imm.xml",
> + if (node != NULL) {
> + /* Retrieve IP information */
> + if ((ip = (IPLIST *)ncs_patricia_tree_get(&clms_cb->iplist, (uint8_t *)&nodeid)) == NULL) {
> + clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
> + LOG_ER("IP information not found for: %s with node_id: %u",
> + nodeup_info->node_name.value, nodeid);
> + } else {
> + if (ip->addr.length) { /* If length = 0, it is AF_TIPC. So process only IPv4 or 6 addresses */
> + /* We might want to validate IP */
> + if (ip_matched(ip->addr.family, ip->addr.value,
> + node->node_addr.family,node->node_addr.value)) {
> + clm_msg.info.api_resp_info.rc = SA_AIS_OK;
> + } else {
> + clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
> + LOG_ER("IP address on %s is not matching the ipaddress in" PKGSYSCONFDIR "/imm.xml",
> nodeup_info->node_name.value);
> + }
> }
> }
> - }
>
> - if ((cb->node_id != nodeup_info->node_id) && (node->nodeup == SA_TRUE)){
> - clm_msg.info.api_resp_info.rc = SA_AIS_ERR_EXIST;
> - LOG_ER("Duplicate node join request for CLM node: '%s'. Specify a unique node name in" PKGSYSCONFDIR "/node_name",
> - nodeup_info->node_name.value);
> - } else
> - clm_msg.info.api_resp_info.rc = SA_AIS_OK;
> -
> + if ((cb->node_id != nodeup_info->node_id) && (node->nodeup == SA_TRUE)){
> + clm_msg.info.api_resp_info.rc = SA_AIS_ERR_EXIST;
> + LOG_ER("Duplicate node join request for CLM node: '%s'. "
> + "Specify a unique node name in" PKGSYSCONFDIR "/node_name",
> + nodeup_info->node_name.value);
> + } else
> + clm_msg.info.api_resp_info.rc = SA_AIS_OK;
> +
> + } /* Node exists in DB */
> +
> if (clm_msg.info.api_resp_info.rc != SA_AIS_OK) {
> clm_msg.evt_type = CLMSV_CLMS_TO_CLMA_API_RESP_MSG;
> clm_msg.info.api_resp_info.type = CLMSV_CLUSTER_JOIN_RESP;
> - clm_msg.info.api_resp_info.param.node_name = node_name;
> + clm_msg.info.api_resp_info.param.node_name = nodeup_info->node_name;
> rc = clms_mds_msg_send(cb, &clm_msg, &evt->fr_dest, &evt->mds_ctxt, MDS_SEND_PRIORITY_HIGH,
> NCSMDS_SVC_ID_CLMNA);
> if (rc != NCSCC_RC_SUCCESS)
>
>
|
|
From: <mat...@or...> - 2014-03-26 15:39:23
|
osaf/services/saf/clmsv/clms/clms_evt.c | 12 ++++++++++--
osaf/services/saf/clmsv/nodeagent/main.c | 7 +++----
2 files changed, 13 insertions(+), 6 deletions(-)
When a node join request from an unconfigured/misconfigured node is attempted, then
CLM prints the following log message:
Mar 26 18:44:52 SC-1 local0.err osafclmd[418]: ER CLM NodeName: 'SC-3' doesn't match entry in imm.xml.
Specify a correct node name in/etc/opensaf/node_name.
This log message is currently categorized as ERROR. However, there are suggestions to improve
the current log message such that the message more closely depicts
the case of unconfigured or misconfigured nodes.
The log string is now changed as below:
Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO CLM NodeName: 'PL-8' is not a configured cluster node.
Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO /etc/opensaf/node_name should contain the rdn value of a configured CLM node object name
In the case of problems with connectivity problems, the following modified log strings are added:
Send timed out. Check status(network, process) of ACTIVE controller
and
Send failed: %u. Check network connectivity
diff --git a/osaf/services/saf/clmsv/clms/clms_evt.c b/osaf/services/saf/clmsv/clms/clms_evt.c
--- a/osaf/services/saf/clmsv/clms/clms_evt.c
+++ b/osaf/services/saf/clmsv/clms/clms_evt.c
@@ -282,8 +282,16 @@ uint32_t proc_node_up_msg(CLMS_CB * cb,
node = clms_node_get_by_name(&node_name);
if (node == NULL) {
clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
- LOG_ER("CLM NodeName: '%s' doesn't match entry in imm.xml. Specify a correct node name in" PKGSYSCONFDIR "/node_name",
- nodeup_info->node_name.value);
+ /* The /etc/opensaf/node_name is an user exposed configuration file.
+ * The node_name file contains the RDN value of the CLM node name.
+ * (a) When opensaf cluster configuration is pre-provisioned using the OpenSAF IMM tools:
+ * the /etc/opensaf/node_name should contain one of the values specified
+ * in nodes.cfg while generating the imm.xml.
+ * (b) When opensaf cluster nodes are dynamically added at runtime:
+ * the /etc/opensaf/node_name should contain the rdn value.
+ */
+ LOG_NO("CLM NodeName: '%s' is not a configured cluster node.", nodeup_info->node_name.value);
+ LOG_NO(PKGSYSCONFDIR "/node_name should contain the rdn value of configured CLM node object name");
}
if (node != NULL) {
diff --git a/osaf/services/saf/clmsv/nodeagent/main.c b/osaf/services/saf/clmsv/nodeagent/main.c
--- a/osaf/services/saf/clmsv/nodeagent/main.c
+++ b/osaf/services/saf/clmsv/nodeagent/main.c
@@ -442,11 +442,10 @@ SaAisErrorT clmna_process_dummyup_msg(vo
break;
case NCSCC_RC_REQ_TIMOUT:
error = SA_AIS_ERR_TIMEOUT;
- break;
- LOG_ER("clmna_mds_msg_sync_send Timed Out");
+ LOG_ER("Send timed out. Check status(network, process) of ACTIVE controller");
goto done;
default:
- LOG_ER("clmna_mds_msg_sync_send FAILED: %u", rc);
+ LOG_ER("Send failed: %u. Check network connectivity", rc);
goto done;
}
@@ -456,7 +455,7 @@ SaAisErrorT clmna_process_dummyup_msg(vo
error = SA_AIS_ERR_NO_RESOURCES;
if (error == SA_AIS_ERR_NOT_EXIST) {
- LOG_ER("%s is Not a member of cluster",
+ LOG_ER("%s is not a configured node",
o_msg->info.api_resp_info.param.node_name.value);
free(o_msg);
goto done;
|
|
From: Hans F. <han...@er...> - 2014-04-03 11:38:00
|
See inline
Thanks,
Hans
On 03/27/2014 02:13 AM, mat...@or... wrote:
> osaf/services/saf/clmsv/clms/clms_evt.c | 12 ++++++++++--
> osaf/services/saf/clmsv/nodeagent/main.c | 7 +++----
> 2 files changed, 13 insertions(+), 6 deletions(-)
>
>
> When a node join request from an unconfigured/misconfigured node is attempted, then
> CLM prints the following log message:
> Mar 26 18:44:52 SC-1 local0.err osafclmd[418]: ER CLM NodeName: 'SC-3' doesn't match entry in imm.xml.
> Specify a correct node name in/etc/opensaf/node_name.
> This log message is currently categorized as ERROR. However, there are suggestions to improve
> the current log message such that the message more closely depicts
> the case of unconfigured or misconfigured nodes.
> The log string is now changed as below:
> Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO CLM NodeName: 'PL-8' is not a configured cluster node.
> Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO /etc/opensaf/node_name should contain the rdn value of a configured CLM node object name
>
> In the case of problems with connectivity problems, the following modified log strings are added:
> Send timed out. Check status(network, process) of ACTIVE controller
> and
> Send failed: %u. Check network connectivity
>
> diff --git a/osaf/services/saf/clmsv/clms/clms_evt.c b/osaf/services/saf/clmsv/clms/clms_evt.c
> --- a/osaf/services/saf/clmsv/clms/clms_evt.c
> +++ b/osaf/services/saf/clmsv/clms/clms_evt.c
> @@ -282,8 +282,16 @@ uint32_t proc_node_up_msg(CLMS_CB * cb,
> node = clms_node_get_by_name(&node_name);
> if (node == NULL) {
> clm_msg.info.api_resp_info.rc = SA_AIS_ERR_NOT_EXIST;
> - LOG_ER("CLM NodeName: '%s' doesn't match entry in imm.xml. Specify a correct node name in" PKGSYSCONFDIR "/node_name",
> - nodeup_info->node_name.value);
> + /* The /etc/opensaf/node_name is an user exposed configuration file.
> + * The node_name file contains the RDN value of the CLM node name.
> + * (a) When opensaf cluster configuration is pre-provisioned using the OpenSAF IMM tools:
> + * the /etc/opensaf/node_name should contain one of the values specified
> + * in nodes.cfg while generating the imm.xml.
> + * (b) When opensaf cluster nodes are dynamically added at runtime:
> + * the /etc/opensaf/node_name should contain the rdn value.
> + */
> + LOG_NO("CLM NodeName: '%s' is not a configured cluster node.", nodeup_info->node_name.value);
In this log I would like to see the "join the cluster" and "unconfigured" that I tried to read out from the CLM spec
3.1.2. For example:
"Node '%s' [attempts|requests] to join the cluster but is unconfigured"
> + LOG_NO(PKGSYSCONFDIR "/node_name should contain the rdn value of configured CLM node object name");
This log is not relevant/correct and should be removed. CLM cannot know if the name presented is wrong or just about to
be configured.
> }
>
> if (node != NULL) {
> diff --git a/osaf/services/saf/clmsv/nodeagent/main.c b/osaf/services/saf/clmsv/nodeagent/main.c
> --- a/osaf/services/saf/clmsv/nodeagent/main.c
> +++ b/osaf/services/saf/clmsv/nodeagent/main.c
> @@ -442,11 +442,10 @@ SaAisErrorT clmna_process_dummyup_msg(vo
> break;
> case NCSCC_RC_REQ_TIMOUT:
> error = SA_AIS_ERR_TIMEOUT;
> - break;
> - LOG_ER("clmna_mds_msg_sync_send Timed Out");
> + LOG_ER("Send timed out. Check status(network, process) of ACTIVE controller");
> goto done;
> default:
> - LOG_ER("clmna_mds_msg_sync_send FAILED: %u", rc);
> + LOG_ER("Send failed: %u. Check network connectivity", rc);
> goto done;
> }
>
> @@ -456,7 +455,7 @@ SaAisErrorT clmna_process_dummyup_msg(vo
> error = SA_AIS_ERR_NO_RESOURCES;
>
> if (error == SA_AIS_ERR_NOT_EXIST) {
> - LOG_ER("%s is Not a member of cluster",
> + LOG_ER("%s is not a configured node",
> o_msg->info.api_resp_info.param.node_name.value);
> free(o_msg);
> goto done;
>
>
|
|
From: <mat...@or...> - 2014-03-26 15:39:25
|
osaf/services/saf/clmsv/nodeagent/main.c | 8 ++++++++
1 files changed, 8 insertions(+), 0 deletions(-)
When a node join request for an unconfigured/misconfigured node or
when a node join request with a duplicate node_name is attempted, then
clmna should report those errors to NID such that NID attempts to
respawan clmna.
With the introduction of this change, the following happens(can be seen in the syslog) in
the case of a unconfigured/misconfigured node join request:
At the ACTIVE controller syslog:
Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO CLM NodeName: 'PL-8' is not a configured cluster node.
Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO /etc/opensaf/node_name should contain the rdn value of a configured CLM node object name
At the unconfigured/misconfigured node, the syslog will be like as below:
Mar 26 19:03:54 PL-3 local0.notice osafclmna[871]: Started
Mar 26 19:03:54 PL-3 local0.err osafclmna[871]: ER PL-8 is not a configured node
Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Failed DESC:CLMNA
Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Going for recovery
Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #1
Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Sending SIGKILL to CLMNA, pid=868
Mar 26 19:03:54 PL-3 local0.err osafclmna[871]: ER Exiting
Mar 26 19:03:54 PL-3 local0.notice osafclmna[871]: exiting for shutdown
Mar 26 19:04:09 PL-3 local0.notice osafclmna[895]: Started
Mar 26 19:04:09 PL-3 local0.err osafclmna[895]: ER PL-8 is not a configured node
Mar 26 19:04:09 PL-3 local0.err osafclmna[895]: ER Exiting
Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Could Not RESPAWN CLMNA
Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Failed DESC:CLMNA
Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #2
Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Sending SIGKILL to CLMNA, pid=892
Mar 26 19:04:09 PL-3 local0.notice osafclmna[895]: exiting for shutdown
Mar 26 19:04:24 PL-3 local0.notice osafclmna[919]: Started
Mar 26 19:04:25 PL-3 local0.err osafclmna[919]: ER PL-8 is not a configured node
Mar 26 19:04:25 PL-3 local0.err osafclmna[919]: ER Exiting
Mar 26 19:04:25 PL-3 local0.err opensafd[837]: ER Could Not RESPAWN CLMNA
Mar 26 19:04:25 PL-3 local0.err opensafd[837]: ER Failed DESC:CLMNA
Mar 26 19:04:25 PL-3 local0.err opensafd[837]: ER FAILED TO RESPAWN
Mar 26 19:04:27 PL-3 local0.notice osafclmna[919]: exiting for shutdown
Mar 26 19:04:28 PL-3 local0.notice osafimmnd[864]: exiting for shutdown
For cases when a duplicate node join request comes, the following syslog
message will be seen at the ACTIVE controller:
Mar 26 19:07:43 SC-1 local0.err osafclmd[418]: ER Duplicate node join request for CLM node: 'SC-2'. Specify a unique node name in/etc/opensaf/node_name
Mar 26 19:07:59 SC-1 local0.err osafclmd[418]: ER Duplicate node join request for CLM node: 'SC-2'. Specify a unique node name in/etc/opensaf/node_name
And the following will be seen at the node on which the duplicate request is attempted:
Mar 26 19:07:43 PL-3 local0.err osafclmna[1459]: ER SC-2 is already up. Specify a unique name in/etc/opensaf/node_name
Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Failed DESC:CLMNA
Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Going for recovery
Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #1
Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Sending SIGKILL to CLMNA, pid=1456
Mar 26 19:07:43 PL-3 local0.err osafclmna[1459]: ER Exiting
Mar 26 19:07:44 PL-3 local0.notice osafclmna[1459]: exiting for shutdown
Mar 26 19:07:59 PL-3 local0.notice osafclmna[1483]: Started
Mar 26 19:07:59 PL-3 local0.err osafclmna[1483]: ER SC-2 is already up. Specify a unique name in/etc/opensaf/node_name
Mar 26 19:07:59 PL-3 local0.err osafclmna[1483]: ER Exiting
Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Could Not RESPAWN CLMNA
Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Failed DESC:CLMNA
Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #2
Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Sending SIGKILL to CLMNA, pid=1480
Mar 26 19:07:59 PL-3 local0.notice osafclmna[1483]: exiting for shutdown
diff --git a/osaf/services/saf/clmsv/nodeagent/main.c b/osaf/services/saf/clmsv/nodeagent/main.c
--- a/osaf/services/saf/clmsv/nodeagent/main.c
+++ b/osaf/services/saf/clmsv/nodeagent/main.c
@@ -458,11 +458,19 @@ SaAisErrorT clmna_process_dummyup_msg(vo
LOG_ER("%s is not a configured node",
o_msg->info.api_resp_info.param.node_name.value);
free(o_msg);
+ rc = error; /* For now, just pass on the error to nid.
+ * This is not needed in future when node local
+ * cluster management policy based decisions can be made.
+ */
goto done;
} else if (error == SA_AIS_ERR_EXIST) {
LOG_ER("%s is already up. Specify a unique name in" PKGSYSCONFDIR "/node_name",
o_msg->info.api_resp_info.param.node_name.value);
free(o_msg);
+ rc = error; /* This is not needed in future when node local
+ * cluster management policy based decisions can be made.
+ * For now, just pass on the error to nid.
+ */
goto done;
}
|
|
From: Hans F. <han...@er...> - 2014-04-03 11:48:32
|
On the controller I get an extra "send failed" log message that is ugly:
Apr 3 13:40:01 SC-1 local0.notice osafclmd[417]: NO CLM NodeName: 'PL-6' is not a configured cluster node.
Apr 3 13:40:01 SC-1 local0.notice osafclmd[417]: NO /etc/opensaf/node_name should contain the rdn value of configured
CLM node object name
Apr 3 13:40:02 SC-1 local0.notice osafclmd[417]: NO proc_initialize_msg: send failed. dest:2060f19cd6007
Apr 3 13:40:17 SC-1 local0.notice osafclmd[417]: NO CLM NodeName: 'PL-6' is not a configured cluster node.
Apr 3 13:40:17 SC-1 local0.notice osafclmd[417]: NO /etc/opensaf/node_name should contain the rdn value of configured
CLM node object name
Apr 3 13:40:17 SC-1 local0.notice osafclmd[417]: NO proc_initialize_msg: send failed. dest:2060f19cda006
Apr 3 13:40:32 SC-1 local0.notice osafclmd[417]: NO CLM NodeName: 'PL-6' is not a configured cluster node.
Apr 3 13:40:32 SC-1 local0.notice osafclmd[417]: NO /etc/opensaf/node_name should contain the rdn value of configured
CLM node object name
Apr 3 13:40:33 SC-1 local0.notice osafclmd[417]: NO proc_initialize_msg: send failed. dest:2060f19cda009
Apr 3 13:40:34 SC-1 local0.notice osafimmnd[388]: NO Global discard node received for nodeId:2060f pid:359
Apr 3 13:40:39 SC-1 user.warn kernel: tipc: Resetting link <1.1.1:eth0-1.1.6:eth0>, peer not responding
I would like to have a try again loop inside clmna instead. Then we could have the nid to supervise and every now and
then do a node reboot. There is not much point in trying three times and give up. I think clmna should retry forever and
let nodeinit handle the bigger loop. The try interval needs to be configured but should have good default such as every
30 sec.
Thanks,
Hans
On 03/27/2014 02:13 AM, mat...@or... wrote:
> osaf/services/saf/clmsv/nodeagent/main.c | 8 ++++++++
> 1 files changed, 8 insertions(+), 0 deletions(-)
>
>
> When a node join request for an unconfigured/misconfigured node or
> when a node join request with a duplicate node_name is attempted, then
> clmna should report those errors to NID such that NID attempts to
> respawan clmna.
> With the introduction of this change, the following happens(can be seen in the syslog) in
> the case of a unconfigured/misconfigured node join request:
> At the ACTIVE controller syslog:
> Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO CLM NodeName: 'PL-8' is not a configured cluster node.
> Mar 26 20:23:32 SC-1 local0.notice osafclmd[420]: NO /etc/opensaf/node_name should contain the rdn value of a configured CLM node object name
>
> At the unconfigured/misconfigured node, the syslog will be like as below:
> Mar 26 19:03:54 PL-3 local0.notice osafclmna[871]: Started
> Mar 26 19:03:54 PL-3 local0.err osafclmna[871]: ER PL-8 is not a configured node
> Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Failed DESC:CLMNA
> Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Going for recovery
> Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #1
> Mar 26 19:03:54 PL-3 local0.err opensafd[837]: ER Sending SIGKILL to CLMNA, pid=868
> Mar 26 19:03:54 PL-3 local0.err osafclmna[871]: ER Exiting
> Mar 26 19:03:54 PL-3 local0.notice osafclmna[871]: exiting for shutdown
> Mar 26 19:04:09 PL-3 local0.notice osafclmna[895]: Started
> Mar 26 19:04:09 PL-3 local0.err osafclmna[895]: ER PL-8 is not a configured node
> Mar 26 19:04:09 PL-3 local0.err osafclmna[895]: ER Exiting
> Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Could Not RESPAWN CLMNA
> Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Failed DESC:CLMNA
> Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #2
> Mar 26 19:04:09 PL-3 local0.err opensafd[837]: ER Sending SIGKILL to CLMNA, pid=892
> Mar 26 19:04:09 PL-3 local0.notice osafclmna[895]: exiting for shutdown
> Mar 26 19:04:24 PL-3 local0.notice osafclmna[919]: Started
> Mar 26 19:04:25 PL-3 local0.err osafclmna[919]: ER PL-8 is not a configured node
> Mar 26 19:04:25 PL-3 local0.err osafclmna[919]: ER Exiting
> Mar 26 19:04:25 PL-3 local0.err opensafd[837]: ER Could Not RESPAWN CLMNA
> Mar 26 19:04:25 PL-3 local0.err opensafd[837]: ER Failed DESC:CLMNA
> Mar 26 19:04:25 PL-3 local0.err opensafd[837]: ER FAILED TO RESPAWN
> Mar 26 19:04:27 PL-3 local0.notice osafclmna[919]: exiting for shutdown
> Mar 26 19:04:28 PL-3 local0.notice osafimmnd[864]: exiting for shutdown
>
>
> For cases when a duplicate node join request comes, the following syslog
> message will be seen at the ACTIVE controller:
> Mar 26 19:07:43 SC-1 local0.err osafclmd[418]: ER Duplicate node join request for CLM node: 'SC-2'. Specify a unique node name in/etc/opensaf/node_name
> Mar 26 19:07:59 SC-1 local0.err osafclmd[418]: ER Duplicate node join request for CLM node: 'SC-2'. Specify a unique node name in/etc/opensaf/node_name
>
> And the following will be seen at the node on which the duplicate request is attempted:
> Mar 26 19:07:43 PL-3 local0.err osafclmna[1459]: ER SC-2 is already up. Specify a unique name in/etc/opensaf/node_name
> Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Failed DESC:CLMNA
> Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Going for recovery
> Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #1
> Mar 26 19:07:43 PL-3 local0.err opensafd[1425]: ER Sending SIGKILL to CLMNA, pid=1456
> Mar 26 19:07:43 PL-3 local0.err osafclmna[1459]: ER Exiting
> Mar 26 19:07:44 PL-3 local0.notice osafclmna[1459]: exiting for shutdown
> Mar 26 19:07:59 PL-3 local0.notice osafclmna[1483]: Started
> Mar 26 19:07:59 PL-3 local0.err osafclmna[1483]: ER SC-2 is already up. Specify a unique name in/etc/opensaf/node_name
> Mar 26 19:07:59 PL-3 local0.err osafclmna[1483]: ER Exiting
> Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Could Not RESPAWN CLMNA
> Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Failed DESC:CLMNA
> Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Trying To RESPAWN /usr/local/lib/opensaf/clc-cli/osaf-clmna attempt #2
> Mar 26 19:07:59 PL-3 local0.err opensafd[1425]: ER Sending SIGKILL to CLMNA, pid=1480
> Mar 26 19:07:59 PL-3 local0.notice osafclmna[1483]: exiting for shutdown
>
> diff --git a/osaf/services/saf/clmsv/nodeagent/main.c b/osaf/services/saf/clmsv/nodeagent/main.c
> --- a/osaf/services/saf/clmsv/nodeagent/main.c
> +++ b/osaf/services/saf/clmsv/nodeagent/main.c
> @@ -458,11 +458,19 @@ SaAisErrorT clmna_process_dummyup_msg(vo
> LOG_ER("%s is not a configured node",
> o_msg->info.api_resp_info.param.node_name.value);
> free(o_msg);
> + rc = error; /* For now, just pass on the error to nid.
> + * This is not needed in future when node local
> + * cluster management policy based decisions can be made.
> + */
> goto done;
> } else if (error == SA_AIS_ERR_EXIST) {
> LOG_ER("%s is already up. Specify a unique name in" PKGSYSCONFDIR "/node_name",
> o_msg->info.api_resp_info.param.node_name.value);
> free(o_msg);
> + rc = error; /* This is not needed in future when node local
> + * cluster management policy based decisions can be made.
> + * For now, just pass on the error to nid.
> + */
> goto done;
> }
>
>
>
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X