#1302 amf: get_comp_capability: Assertion 'ctcs_type' failed

[Neelakanta Reddy]

Nodes : 4 nodes SC-1, SC-2, PL-3, PL-4

stop the PL-4 and amfd got asserted

Apr 3 15:59:59 SLES1 osafamfd[3935]: NO Node 'PL-4' left the cluster
Apr 3 15:59:59 SLES1 osafimmnd[3880]: NO Implementer connected: 18 (MsgQueueService132111) <964, 2010f>
Apr 3 15:59:59 SLES1 osafimmnd[3880]: NO Implementer locally disconnected. Marking it as doomed 18 <964, 2010f> (MsgQueueService132111)
Apr 3 15:59:59 SLES1 osafimmnd[3880]: NO Implementer disconnected 18 <964, 2010f> (MsgQueueService132111)
Apr 3 15:59:59 SLES1 osafamfd[3935]: util.cc:683: get_comp_capability: Assertion 'ctcs_type' failed.
Apr 3 15:59:59 SLES1 osafamfnd[3945]: ER AMF director unexpectedly crashed

[Neelakanta Reddy]

The problem is observed when campaign is in committed state and PL-4 is rebooted

[Praveen]

There is a simple way to reproduce the problem without running any SMF campaign.
Steps to reproduce:
1)Bring up AMF_DEMO sample up.
2)Now create a new compType with following command:
immcfg -c SaAmfCompType safVersion=5.0.0,safCompType=AmfDemo1 -a saAmfCtCompCategory=1 -a saAmfCtDefClcCliTimeout=10000000000 -a saAmfCtDefCallbackTimeout=10000000000 -a saAmfCtRelPathInstantiateCmd=amf_demo_script -a saAmfCtDefInstantiateCmdArgv=instantiate -a saAmfCtRelPathCleanupCmd=amf_demo_script -a saAmfCtDefCleanupCmdArgv=cleanup -a saAmfCtDefRecoveryOnError=2 -a saAmfCtDefDisableRestart=0 -a saAmfCtSwBundle=safSmfBundle=AmfDemo
3)Change the compType in each SU.
For SU1 :
immcfg -a saAmfCompType=safVersion=5.0.0,safCompType=AmfDemo1 safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1

For SU2:
immcfg -a saAmfCompType=safVersion=5.0.0,safCompType=AmfDemo1 safComp=AmfDemo,safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1

4)Lock the active SU, amfd will crash.

[Praveen]

Approach for fixing this problem:
During the modification of compType in assigned comp, AMF should check existence of association object of cstype of assigned csi and new comptype. Modification should be allowed if this association object exists in the system.

At the same time assert needs to be avoided while upgrading any deployed application running with an invalid configuration. Since AMF reads only comp capability from ctcstype, some default value can be set if ctcstype is not found. Remember default comp capability should be valid for all red model. As per AMF spec page 184 section 3.7, only x_active_and_y_standby is valid for all red models for PI component. For NPI component, default value will be SA_AMF_COMP_NON_PRE_INSTANTIABLE.
In this way, user will not observe any service outage for any csi and assert will also be avoided.

Please comment on this approach.

[Hans Nordebäck]

There seem to be at least two problem, either the SaAmfCtCsType exists (with a wrong name) or it does not exists and is referenced from a comptype. The following steps can be run on the amf demo app:

i)

immcfg -c SaAmfCompType -a saAmfCtSwBundle="safSmfBundle=AmfDemo" -a saAmfCtRelPathInstantiateCmd=amf_demo_script -a saAmfCtRelPathCleanupCmd=amf_demo_script -a saAmfCtDefRecoveryOnError=2 -a saAmfCtDefQuiescingCompleteTimeout=10000000000 -a saAmfCtDefInstantiateCmdArgv=instantiate -asaAmfCtDefDisableRestart=0 -a saAmfCtDefCmdEnv="AMF_DEMO_VAR2=CT_VALUE2 AMF_DEMO_VAR1=CT_VALUE1" -a saAmfCtDefCleanupCmdArgv=cleanup -a saAmfCtDefClcCliTimeout=10000000000 -a saAmfCtDefCallbackTimeout=10000000000 -a saAmfCtCompCategory=1 safVersion=2,safCompType=AmfDemo1

ii)

DN not correct, or don't create this association class leads to util.cc:683: get_comp_capability: Assertion 'ctcs_type' failed. If this association class is not created the system will not boot!

osafamfd[459]: WA configuration validation error: 'safSupportedCsType=safVersion=1\,safCSType=AmfDemo1,safVersion=2,safCompType=AmfDemo1' does not exist in model
May 20 15:33:26 SC-1 local0.err osafamfd[459]: ER Failed to read configuration, AMF will not start
May 20 15:33:26 SC-1 local0.err osafamfd[459]: ER avd_imm_config_get FAILED
May 20 15:33:26 SC-1 local0.err osafamfd[459]: ER avd_standby_role_initialization FAILED
May 20 15:33:26 SC-1 local0.err opensafd[344]: ER Failed DESC:AMFD
May 20 15:33:26 SC-1 local0.err opensafd[344]: ER Going for recovery
May 20 15:33:26 SC-1 local0.err osafamfd[459]: ER initialize failed, exiting

immcfg -c SaAmfCtCsType -a saAmfCtCompCapability=1 safSupportedCsType="safVersion=1\\,safCSType=AmfDemo1,safVersion=2,safCompType=AmfDemo1"

iii)

immcfg -a saAmfCompType=safVersion=2,safCompType=AmfDemo1 safComp=AmfDemo,safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1

iv)

immcfg -c SaAmfHealthcheckType -a saAmfHctDefPeriod=10000000000 -a saAmfHctDefMaxDuration=6000000000 safHealthcheckKey="AmfDemo,safVersion=2,safCompType=AmfDemo1"

v)

amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1
amf-adm unlock-in safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1

amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1
amf-adm unlock safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1

[Hans Nordebäck]

It seems an ctcstype association object can not be created if the comptype does not exist but it can be created if the cstype does not exist. The comptype can be created without an existing ctcstype, so if creating/modifying a comptype a related creation of a ctcstype and, if needed, creation of a cstype should be in the same ccb. I agree with the removal of the assert.

[Praveen]

changeset: 6577:7c09a2f59ad5
tag: tip
parent: 6574:4b5e2c837a13
user: praveen.malviya@oracle.com
date: Mon May 25 11:35:44 2015 +0530
summary: amfd: validate ctcstype relationship while modifying comptype in comp [#1302]

changeset: 6576:91f6d5c44742
branch: opensaf-4.6.x
parent: 6573:2b27523902a9
user: praveen.malviya@oracle.com
date: Mon May 25 11:35:34 2015 +0530
summary: amfd: validate ctcstype relationship while modifying comptype in comp [#1302]

changeset: 6575:67be46646536
branch: opensaf-4.5.x
parent: 6572:a3e9c7e7dff0
user: praveen.malviya@oracle.com
date: Mon May 25 11:34:59 2015 +0530
summary: amfd: validate ctcstype relationship while modifying comptype in comp [#1302]