#1683 Amf: Standby Amfd crash during cold sync

[Nagendra Kumar]

Test Steps:
1. Comment out the following line from safAmfNodeGroup=AllNodes,safAmfCluster=myAmfCluster and safAmfNodeGroup=PLs,safAmfCluster=myAmfCluster:
<value>safAmfNode=PL-3,safAmfCluster=myAmfCluster</value>
Start Act controller and upload demo appl configuration(SU2 hosted on PL-3).
2. Lock and lock-in PL-3.
3. In Standby Amfd keep a sleep of 7 sec after confg read and before applier set as below:

+++ b/osaf/services/saf/amf/amfd/role.cc
@@ -219,6 +219,9 @@ uint32_t avd_standby_role_initialization
}

    cb->init_state = AVD_CFG_DONE;

• LOG_ER("1. Waiting ..........................");
• sleep(7);

• LOG_ER("2. Waiting ..........................");

  if (avd_imm_applier_set() != SA_AIS_OK) {
          LOG_ER("avd_imm_applier_set FAILED");
  

  4.Start Standby controller. When Amfd comes up and waiting for 7 sec, delete SU3 as below:
  immcfg -d safSu=SU2,safSg=AmfDemo_2N,safApp=AmfDemo1
  5.When Amfd wakes up and become applier, delete the PL-3 node:
  immcfg -d safAmfNode=PL-3,safAmfCluster=myAmfCluster
  6. Amfd crashes and Standby Amfd traces attached and syslogs the below:

  Feb 26 10:23:15 PM_SC-2 osafamfd[28437]: ER 1. Waiting ..........................
  

  Feb 26 10:23:17 PM_SC-2 osafimmnd[28387]: NO Ccb 3 COMMITTED (immcfg_PM_SC-1_8718)
  Feb 26 10:23:17 PM_SC-2 osafimmnd[28387]: NO Ccb 4 COMMITTED (immcfg_PM_SC-1_8721)
  Feb 26 10:23:17 PM_SC-2 osafimmnd[28387]: NO Ccb 5 COMMITTED (immcfg_PM_SC-1_8724)
  Feb 26 10:23:22 PM_SC-2 osafamfd[28437]: ER 2. Waiting ..........................
  Feb 26 10:23:22 PM_SC-2 osafimmnd[28387]: NO Implementer (applier) connected: 10 (@safAmfService2020f) <9, 2020f>
  Feb 26 10:23:25 PM_SC-2 osafamfd[28437]: NO Cold sync complete!
  Feb 26 10:23:25 PM_SC-2 osafamfnd[28447]: Started
  Feb 26 10:23:25 PM_SC-2 osafamfnd[28447]: NO 'safSu=SC-2,safSg=2N,safApp=OpenSAF' Presence State UNINSTANTIATED => INSTANTIATING
  Feb 26 10:23:25 PM_SC-2 osafamfnd[28447]: NO 'safSu=SC-2,safSg=NoRed,safApp=OpenSAF' Presence State UNINSTANTIATED => INSTANTIATING
  Feb 26 10:23:25 PM_SC-2 osafamfwd[28464]: Started
  Feb 26 10:23:25 PM_SC-2 osafckptd[28462]: Started
  Feb 26 10:23:25 PM_SC-2 osafevtd[28481]: Started
  Feb 26 10:23:25 PM_SC-2 osafckptnd[28483]: Started
  Feb 26 10:23:25 PM_SC-2 osaflcknd[28501]: Started
  Feb 26 10:23:25 PM_SC-2 osaflckd[28514]: Started
  Feb 26 10:23:25 PM_SC-2 osafmsgnd[28535]: Started
  Feb 26 10:23:25 PM_SC-2 osafimmnd[28387]: NO Implementer connected: 11 (MsgQueueService131599) <326, 2020f>
  Feb 26 10:23:25 PM_SC-2 osafsmfnd[28553]: Started
  Feb 26 10:23:25 PM_SC-2 osafmsgd[28564]: Started
  Feb 26 10:23:25 PM_SC-2 osafamfnd[28447]: NO 'safSu=SC-2,safSg=NoRed,safApp=OpenSAF' Presence State INSTANTIATING => INSTANTIATED
  Feb 26 10:23:25 PM_SC-2 osafamfnd[28447]: NO Assigning 'safSi=NoRed2,safApp=OpenSAF' ACTIVE to 'safSu=SC-2,safSg=NoRed,safApp=OpenSAF'
  Feb 26 10:23:25 PM_SC-2 osafamfnd[28447]: NO Assigned 'safSi=NoRed2,safApp=OpenSAF' ACTIVE to 'safSu=SC-2,safSg=NoRed,safApp=OpenSAF'
  Feb 26 10:23:25 PM_SC-2 osafsmfd[28583]: Started
  Feb 26 10:23:26 PM_SC-2 osafamfnd[28447]: NO 'safSu=SC-2,safSg=2N,safApp=OpenSAF' Presence State INSTANTIATING => INSTANTIATED
  Feb 26 10:23:26 PM_SC-2 osafamfnd[28447]: NO Assigning 'safSi=SC-2N,safApp=OpenSAF' STANDBY to 'safSu=SC-2,safSg=2N,safApp=OpenSAF'
  Feb 26 10:23:26 PM_SC-2 osafamfnd[28447]: NO Assigned 'safSi=SC-2N,safApp=OpenSAF' STANDBY to 'safSu=SC-2,safSg=2N,safApp=OpenSAF'
  Feb 26 10:23:26 PM_SC-2 opensafd: OpenSAF(4.7.0 - 16:031538a65597:) services successfully started
  Feb 26 10:23:27 PM_SC-2 osafimmnd[28387]: NO Ccb 6 COMMITTED (immcfg_PM_SC-1_8731)
  Feb 26 10:23:27 PM_SC-2 osafamfnd[28447]: ER AMF director unexpectedly crashed
  Feb 26 10:23:27 PM_SC-2 osafamfnd[28447]: Rebooting OpenSAF NodeId = 131599 EE Name = , Reason: local AVD down(Adest) or both AVD down(Vdest) received, OwnNodeId = 131599, SupervisionTime = 60
  Feb 26 10:23:27 PM_SC-2 osafimmnd[28387]: NO Implementer locally disconnected. Marking it as doomed 10 <9, 2020f> (@safAmfService2020f)
  Feb 26 10:23:27 PM_SC-2 osafimmnd[28387]: NO Implementer disconnected 10 <9, 2020f> (@safAmfService2020f)
  Feb 26 10:23:27 PM_SC-2 opensaf_reboot: Rebooting local node; timeout=60
  Feb 26 10:23:35 PM_SC-2 opensafd: Stopping OpenSAF Services
  Feb 26 10:23:45 PM_SC-2 osafsmfd[28583]: AL AMF Node Director is down, terminate this process
  Feb 26 10:23:45 PM_SC-2 osafmsgd[28564]: AL AMF Node Director is down, terminate this process
  Feb 26 10:23:45 PM_SC-2 osafsmfnd[28553]: AL AMF Node Director is down, terminate this process
  Feb 26 10:23:45 PM_SC-2 osafmsgnd[28535]: AL AMF Node Director is down, terminate this process
  Feb 26 10:23:45 PM_SC-2 osaflckd[28514]: AL AMF Node Director is down, terminate this process

[Nagendra Kumar]

changeset: 7295:c470eb0dc821
branch: opensaf-4.6.x
parent: 7285:9cbee254345e
user: Nagendra Kumarnagendra.k@oracle.com
date: Fri Mar 04 14:18:40 2016 +0530
summary: amfd: delete su and its child objects at stdby amfd [#1683]

changeset: 7296:d3a8f6ad8a1a
branch: opensaf-4.7.x
parent: 7286:eaad57f8cc9c
user: Nagendra Kumarnagendra.k@oracle.com
date: Fri Mar 04 14:19:20 2016 +0530
summary: amfd: delete su and its child objects at stdby amfd [#1683]

changeset: 7297:449f3e716b5f
tag: tip
parent: 7294:c05ef9b8a3b9
user: Nagendra Kumarnagendra.k@oracle.com
date: Fri Mar 04 14:19:34 2016 +0530
summary: amfd: delete su and its child objects at stdby amfd [#1683]

[staging:c470eb]
[staging:d3a8f6]
[staging:449f3e]

changeset: 7320:496873f05a94
branch: opensaf-4.6.x
tag: tip
parent: 7310:a2210ab06c47
user: Nagendra Kumarnagendra.k@oracle.com
date: Fri Mar 11 14:15:00 2016 +0530
summary: amfd: fix compilation error [#1683]
[staging:496873]