#1359 AMF: opensafd fails to come up on payload during MW switchover

[Srikanth R]

Changeset:6490

Issue : opensafd fails to come up on payload during MW switchover

Steps ran :

-> Initially opensafd is up on 5 node cluster with 3 payloads configured.

-> For reproduction of #371, performed couple of switchovers and stopped opensafd on PL-3.

-> Also tried the scenario to start opensafd on payload, during switchover.

-> Out of five times, opensafd on payload failed to come up two times.

-> Middleware SU for the PL-3 is struck in INSTANTIATING state and after a timeout of 15 minutes or so, opensafd totally exited and the SU moved to UNINSTANTIATED state.

safSu=PL-3,safSg=NoRed,safApp=OpenSAF
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=DISABLED(2)
saAmfSUPresenceState=INSTANTIATING(2)
saAmfSUReadinessState=OUT-OF-SERVICE(1)

Following are the logs observed on the syslog of the new active controller ( after switchover) :

Apr 30 11:31:21 CONTROLLER-2 osafrded[2592]: NO RDE role set to ACTIVE
Apr 30 11:31:21 CONTROLLER-2 osafclmd[2658]: NO ACTIVE request
Apr 30 11:31:21 CONTROLLER-2 osafamfd[2677]: NO Controller switch over done
Apr 30 11:31:21 CONTROLLER-2 osafamfd[2677]: ER avd_data_update_req_evh: invalid node state 3
Apr 30 11:31:21 CONTROLLER-2 osafamfd[2677]: WA avd_msg_sanity_chk: invalid msg id 25, from 2030f should be 24

Syslog on the payload :

Apr 30 11:31:18 SYSTEST-PLD-1 opensafd: Starting OpenSAF Services (Using TIPC)
Apr 30 11:31:18 SYSTEST-PLD-1 kernel: [ 1737.865592] TIPC: Activated (version 2.0.0)
....
Apr 30 11:47:51 SYSTEST-PLD-1 opensafd[6355]: ER Timed-out for response from AMFND
Apr 30 11:47:51 SYSTEST-PLD-1 opensafd[6355]: ER
Apr 30 11:47:51 SYSTEST-PLD-1 opensafd[6355]: ER Going for recovery
Apr 30 11:47:51 SYSTEST-PLD-1 osafamfnd[6406]: NO Shutdown initiated
Apr 30 11:47:51 SYSTEST-PLD-1 osafamfnd[6406]: NO Terminating all AMF components

[Nagendra Kumar]

changeset: 6525:32079e2039d2
branch: opensaf-4.5.x
parent: 6519:5ea169175607
user: Nagendra Kumarnagendra.k@oracle.com
date: Fri May 08 12:10:55 2015 +0530
summary: amfd: send data verify message to node in NO_CONFIG and NCS_INIT also [#1359]

changeset: 6526:c4e69c3a8485
branch: opensaf-4.6.x
parent: 6522:454a164b3fa9
user: Nagendra Kumarnagendra.k@oracle.com
date: Fri May 08 12:12:12 2015 +0530
summary: amfd: send data verify message to node in NO_CONFIG and NCS_INIT also [#1359]

changeset: 6527:3f6b601f8b3e
tag: tip
parent: 6523:5ee4540aab11
user: Nagendra Kumarnagendra.k@oracle.com
date: Fri May 08 12:12:21 2015 +0530
summary: amfd: send data verify message to node in NO_CONFIG and NCS_INIT also [#1359]

[staging:32079e]
[staging:c4e69c]
[staging:3f6b60]