#2105 AMF : SG is unstable, if app responds during node link loss detection time period

[Srikanth R]

Setup :
Changeset : 8190
5 node SLES setup with 2 controllers and 3 payloads ( TIPC -- headless enabled)
2n application deployed on 2 payloads.

Issue :

-> Perform admin operation on an AMF enity.
-> Do not respond to the callback and invoke headless scenario.
-> On a VM with TIPC setup, 3 seconds is taken to detect the node down.
-> If the application responds to a callback in admin operation during this time period when the last controller is down, the message shall not reach any controller. Amfnd on payload shall send the "Assigned" message but not store this message.

For this scenario, SG shall move to unstable state. Below is the snippet from syslog, where application responded at 15:48:28 and at 15:48:31 payloads detected that last controller is down.

Oct 7 15:48:28 SYSTEST-PLD-1 osafamfnd[9976]: NO Assigned 'safSi=TestApp_SI1,safApp=TestApp_TwoN' ACTIVE to 'safSu=TestApp_SU1,safSg=TestApp_SG1,safApp=TestApp_TwoN'
Oct 7 15:48:31 SYSTEST-PLD-1 osafamfnd[9976]: WA AMF director unexpectedly crashed
Oct 7 15:48:31 SYSTEST-PLD-1 osafamfnd[9976]: NO Checking 'safSu=PL-3,safSg=NoRed,safApp=OpenSAF' for pending messages
Oct 7 15:48:31 SYSTEST-PLD-1 osafamfnd[9976]: NO Checking 'safSu=TestApp_SU1,safSg=TestApp_SG1,safApp=TestApp_TwoN' for pending messages
Oct 7 15:48:31 SYSTEST-PLD-1 osafimmnd[9957]: WA SC Absence IS allowed:900 IMMD service is DOWN
Oct 7 15:48:31 SYSTEST-PLD-1 osafimmnd[9957]: NO IMMD SERVICE IS DOWN, HYDRA IS CONFIGURED => UNREGISTERING IMMND form MDS

-> Below is the scenario, when payload detected that there is no controller at 18:31:34 and amfnd shall call avnd_di_susi_resp_send after the controllers join back the cluster. Application responded at 18:31:41.

Oct 7 18:31:34 SYSTEST-PLD-1 osafimmnd[12448]: WA SC Absence IS allowed:900 IMMD service is DOWN
Oct 7 18:31:34 SYSTEST-PLD-1 osafimmnd[12448]: NO IMMD SERVICE IS DOWN, HYDRA IS CONFIGURED => UNREGISTERING IMMND form MDS
Oct 7 18:31:41 SYSTEST-PLD-1 osafamfnd[12467]: NO Assigned 'safSi=TestApp_SI4,safApp=TestApp_TwoN' ACTIVE to 'safSu=TestApp_SU1,safSg=TestApp_SG1,safApp=TestApp_TwoN'
Oct 7 18:31:41 SYSTEST-PLD-1 osafamfnd[12467]: NO avnd_di_susi_resp_send() deferred as AMF director is offline

[Minh Hon Chau]

this type of problem will be documented in ticket #2179, one possible solution is that change the assignment messages to synchronized ones.

[Praveen]

Attached traces when AMFD drops the message from Amfnd.

[Minh Hon Chau]

attached traces where problem of #2105 happens

[Minh Hon Chau]

commit b24a27c0a4c9ed6ee4f8a9dbce499466cc5e2ae1
Author: Minh Chau minh.chau@dektech.com.au
Date: Fri May 19 15:51:28 2017 +1000

commit 3327fbe68a0dfb978ba6ee316927d96381f42037
Author: Minh Chau minh.chau@dektech.com.au
Date: Fri May 19 15:51:28 2017 +1000

changeset: 8802:6737117c6904
tag: tip
user: Minh Hon Chau minh.chau@dektech.com.au
date: Fri May 19 20:44:32 2017 +1000