#3312 fmd: sc failed to failover in roamming mode

[Huu The Truong]

Shutdown SC-6 (role is standby):
2022-03-07 12:14:52.551 INFO: * Stop standby SC (SC-6)

SC-10 changed role to standby:
2022-03-07 12:14:54.919 SC-10 osafrded[384]: NO RDE role set to STANDBY
However, a service of the old standby is still alive, lead to SC-10 received peer info from the old standby (SC-6). It mistakes this service as active SC is downing.

SC-10 changed role to active then rebooted.
2022-03-07 12:14:55.522 SC-10 osaffmd[394]: NO Controller Failover: Setting role to ACTIVE
2022-03-07 12:14:55.522 SC-10 osafrded[384]: NO RDE role set to ACTIVE
2022-03-07 12:14:55.522 SC-10 osafrded[384]: NO Running '/usr/local/lib/opensaf/opensaf_sc_active' with 0 argument(s)
2022-03-07 12:14:55.654 SC-10 opensaf_sc_active: 49cbd770-9e07-11ec-b3b4-525400fd3480 expected on SC-1
2022-03-07 12:14:55.656 SC-10 osafntfd[439]: NO ACTIVE request
2022-03-07 12:14:55.656 SC-10 osaffmd[394]: NO Controller promoted. Stop supervision timer
2022-03-07 12:14:55.657 SC-10 osafclmd[450]: NO ACTIVE request
2022-03-07 12:14:55.657 SC-10 osafamfd[461]: NO FAILOVER StandBy --> Active
2022-03-07 12:14:55.657 SC-10 osafamfd[461]: ER FAILOVER StandBy --> Active FAILED, Standby OUT OF SYNC
2022-03-07 12:14:55.657 SC-10 osafamfd[461]: Rebooting OpenSAF NodeId = 0 EE Name = No EE Mapped, Reason: FAILOVER failed, OwnNodeId = 20a0f, SupervisionTime = 60

[Mohan Kanakam]

Hi Huu The Truong,
I tried to reproduce the issue but i did not succeed.
I configured cluster with 4 nodes(SC-1(ACTIVE),SC-2(STANDBY),SC-3(QUIESCED),SC-4(QUIESCED)) and manually shutdown the SC-2(STANDBY) node to reproduce the issue.
Thanks
Mohan