#1553 AMF: Send CsiSetCallback immediately when CSI attributes are modified (loadbalancing, VM migration usecases)

[Mathi Naickan]

AMF is the 'workload' manager in OpenSAF. There are also loadbalancers external to OpenSAF that can choose to modify some attributes of an application component's workload which is currently assigned ACTIVE (without loss of service). In such scenarios it is possible (and necessary) for both AMF and the external load balancers to throttle each other. The reverse throttling(from AMF to the external load balancer) is possible based on any changes to the NodeCapacity limitations set in AMF, but that is for later/future considerations.

For eg:- Suppose an application component (say modelled in n-way active) is currently serving traffic originating from a source/target X, based on some network criteria the load balancer can choose to re-configure/modify that application to serve traffic from a different source/target Y. Such change in information related to that workload could be communicated to that application component at runtime without loss of service.

The above scenario is achieved if the admin (or a loadbalancer) modifies the attributes of a workload i.e. csiattribute name-value pair, and then the altered name-value information is communicated to that application or associated application components 'immediately' instead of waiting for the next admin operation!

Today AMF supports modifying csi attribute name-value without requiring that SI to be locked. But the modified attributes are sent only after a subsequent admin operation. This ticket is intended to evaluate and support sending CSIsetcallback (with the same CSI state information but modified csi attributes) immediately.

Implementation scope:
Introduction of new resources:
-saAmfInitialize_5() API,
-callback structure SaAmfCallbacksT_5 and
-CSI Attribute Change callback OsafCsiAttributeChangeCallbackT.
-new attribute osafAmfCSICommunicateCsiAttributeChange in class SaAmfCSI.
As a part of #1553, minor version (SAF) of AMF will be updated to 02 because of introduction of aobve mentioned resources in agent. So highest supported version now becomes B.04.02.

Description of new resources at agent:
A)OsafCsiAttributeChangeCallbackT
AMF will use this callback to communicate to a running component
any runtime change in the assigned CSI Attribute value.
A component can register this callback using new API saAmfInitialize_5().
Callback Characteristics:
-Arguments:
typedef void (OsafCsiAttributeChangeCalbackT)( SaInvocationT invocation,
const SaNameT csiName, SaAmfCSIAttributeListT csiAttr);
-After receving this callback, a component will have to respond to AMFND
using saAmfResponse_4() API with the same invocation number.
-Callback will be invoked in the context of registered process.
-Callback is meant for SA-Aware component, PROXIED PI component and
PROXIED NPI component. In cases of proxied it will be given to proxy.
-Callback timeout will be same as saAmfCompCSISetCallbackTimeout or
saAmfCtDefCallbackTimeout.
B)SaAmfCallbacksT_5 (new callback structure)
typedef struct {
SaAmfHealthcheckCallbackT saAmfHealthcheckCallback;
SaAmfComponentTerminateCallbackT saAmfComponentTerminateCallback;
SaAmfCSISetCallbackT saAmfCSISetCallback;
SaAmfCSIRemoveCallbackT saAmfCSIRemoveCallback;
SaAmfProtectionGroupTrackCallbackT_4 saAmfProtectionGroupTrackCallback;
SaAmfProxiedComponentInstantiateCallbackT saAmfProxiedComponentInstantiateCallback;
SaAmfProxiedComponentCleanupCallbackT saAmfProxiedComponentCleanupCallback;
SaAmfContainedComponentInstantiateCallbackT saAmfContainedComponentInstantiateCallback;
SaAmfContainedComponentCleanupCallbackT saAmfContainedComponentCleanupCallback;
OsafCsiAttributeChangeCallbackT osafCsiAttributeChangeCallback;
} SaAmfCallbacksT_5;
This is new callback structure consisting of all B.04.01 callbacks and newly
introduced OsafCsiAttributeChangeCallbackT. A component can register all
callbacks including new one with new API saAmfInitialize_5().
C)New API: SaAisErrorT saAmfInitialize_5( SaAmfHandleT amfHandle, const SaAmfCallbacksT_5
amfCallbacks, SaVersionT *version)
Applicable return codes are same as valid for saAmfInitialize_4().
New API for initializing with AMF service for all callbacks including the new one.
New saf version for this is B.04.02. A user will have to compulsory register
OsafCsiAttributeChangeCallbackT callback along with other mandatory callbacks to
initialize with AMF service using this API.
-Attribute osafAmfCSICommunicateCsiAttributeChange in class SaAmfCSI.
A NON Proxied NPI component neither registers with AMF nor it is registered via
any Proxy component. AMF manages only Life Cycle of this component through
CLC-CLI scripts. All CLC-CLI scripts are idempotents. For such a component, AMF
passes name-value pairs of CSI as environment variables to each CLC-CLI command.
If there is a change in CSI attribute value for a CSI assigned to NON PROXIED NPI
component, the change will be passed to the component as environment variables by
executing INSTANTIATED command provided osafAmfCSICommunicateCsiAttributeChange is
marked true for the assigned CSI.

[Mathi Naickan]

Praveen is pointing out that this ticket could be useful in scenarios involving VM migration too!
For eg:- As a part of the migration solution, before trigerring the migration, the CSI attributes can be modified first to let the application(VM) know about the coordinates(in the form of CSI attributes) of the destination VM.
This information could then be utilized to trigger the start of migration as a part of a AMF shutdown operation for example!

[Anders Widell]

Note: this change is not backwards compatible, so we will need to bump the minor version of the AMF API (in a similar way as we do in IMM) if we introduce this feature.

[Praveen]

Invoking CSI set callback whenever CSI attribute value is changed seems in-consistent with the HA state diagram mentioned on page 83 of AMF B0401 spec as CSI set callback is given only when there is a change in HA state of CSI. To avoid inconsistency introducing new callback and other resources. Please check the implementation scope in ticket description.

[Praveen]

Changes at AMFD (CSI Attribute Change Callback):
1)Support for new attribute osafAmfCSICommunicateCsiAttributeChange in class SaAmfCSI.
Dynamic modification of this attribute is supported. Default value will be 0.
2)New message AVSV_D2N_COMPCSI_ASSIGN_MSG from AMFD to AMFND for compcsi related
information. As of now only CSI attribute list will be sent to AMFND.
In future it can be used in the context of tickets like #538 (TERM_FAILED state) and #83.
3)Upon modification of CSI attribute value for a object of class "SaAmfCSIAttribute",
AMFD will send a message to AMFND with new list. For a NON PROXIED NPI component,
message will not be sent if osafAmfCSICommunicateCsiAttributeChange is false.
For such a component, change will come into effect only when its SI is locked
and unlocked or other such ways.
4)AMFD now also maintains MDS install version of all AMFNDs in
std::map<saclmnodeidt, mds_svc_pvt_sub_part_ver=""> nds_mds_ver_db.
It will be updated whenever AMFD gets MDS_UP and MDS_DOWN for AMFND.
Using this AMFD can decide whether message is meant for particular AMFND much
before encode callback given by MDS.</saclmnodeidt,>

Changes at AMFND (CSI Attribute Change Callback):
1)New internal AMF callback from AMFND to AMF agent corresponding to
OsafCsiAttributeChangeCallbackT for giving modified CSI Attribute
Name value pairs to component (AVSV_AMF_CSI_ATTR_CHANGE_PARAM).
2)AMFND processes AVSV_D2N_COMPCSI_ASSIGN_MSG from AMFD. Upong receving
CSI attribute list, it updates its databae with modified information.
If compoent has registered with new callback OsafCsiAttributeChangeCallbackT,
then it will send this modification through internal callback AVSV_AMF_CSI_ATTR_CHANGE_PARAM
to AMF agent.
3)AMFND now maintains MDS install version of AMF AGENTs in
std::map<mds_dest, mds_svc_pvt_sub_part_ver=""> agent_mds_ver_db.
It will be updated whenever AMFND gets MDS_UP and MDS_DOWN for AMF Agent.
Using this AMFND can decide whether message is meant for particular Agent much
before encode callback given by MDS.
4)AMFND maintains now SAF version for each registered component.
AMFND will get it from Agent through existing component registeration message.</mds_dest,>

Changes at AMF Agent (CSI Attribute Change Callback):
1)Because of different AMF callback struture correponding to each of B.01.01
B.04.01, introduced an internal callback structure OsafAmfCallbacksT.
It consists of callback from each AMF version. Since callback for each
intialization needs to be maintained internally in AVA_HDL_REC. With this
internal structure, same handle can used internally for any initialization
of AMF service. Here it is:
typedef struct osaf_amf_cbk {
SaAmfHealthcheckCallbackT saAmfHealthcheckCallback;
SaAmfComponentTerminateCallbackT saAmfComponentTerminateCallback;
SaAmfCSISetCallbackT saAmfCSISetCallback;
SaAmfCSIRemoveCallbackT saAmfCSIRemoveCallback;
SaAmfProtectionGroupTrackCallbackT saAmfProtectionGroupTrackCallback;
SaAmfProtectionGroupTrackCallbackT_4 saAmfProtectionGroupTrackCallback_4; //Only in B.04.01
SaAmfProxiedComponentInstantiateCallbackT saAmfProxiedComponentInstantiateCallback;
SaAmfProxiedComponentCleanupCallbackT saAmfProxiedComponentCleanupCallback;
SaAmfContainedComponentInstantiateCallbackT saAmfContainedComponentInstantiateCallback;
SaAmfContainedComponentCleanupCallbackT saAmfContainedComponentCleanupCallback;
OsafCsiAttributeChangeCallbackT osafCsiAttributeChangeCallback; //OpenSAF implementation.
} OsafAmfCallbacksT;
Also added utility function to copy callbacks from AMF callbacks structure to this internal one.
AMF agent will use OsafAmfCallbacksT internally instead of SaAmfCallbacksT_<#x>.
2)Implementation of saAmfInitialize_5() in ava_api.cc.
3)Now saAmfRegister() also sends SAF version to AMFND.

Will update more on AMFD and AMFND MDS versioning before publishing the patches.

[Praveen]

Published intial version with some TODOs:
1)Invocation of INSTANTIATE command for a Non Proxied NPI component.
2)Introduce saAmf_B_04_02.h for new resources at agent.
3)These patches are old (work done in 5.0), so
re-base over #1642(AMF long dn) and other C++ refactoring.

[Praveen]

Patches rebased on long dn patches 1553_longdn.tgz.
Also attached is 1553_longdn_rebased.patch (As a single patch rebased on lonng dn changes).

[Praveen]

Removed old attahcments as there were minor problem when applied. Attached are the new ones.

[Praveen]

changeset: 7977:3cdd554fbbd6
tag: tip
user: praveen.malviya@oracle.com
date: Mon Aug 29 17:51:26 2016 +0530
summary: samples/amf: update amf sa-aware demo for CSI attribute change callback. [#1553]

changeset: 7976:b5da43b1f1ce
user: praveen.malviya@oracle.com
date: Mon Aug 29 17:50:27 2016 +0530
summary: amfa: add support for csi attribute change callback at amfa. [#1553]

changeset: 7975:ddb4db46b4c8
user: praveen.malviya@oracle.com
date: Mon Aug 29 17:49:47 2016 +0530
summary: amfnd: add support for csi attribute change callback at amfnd. [#1553]

changeset: 7974:a22ce66187ff
user: praveen.malviya@oracle.com
date: Mon Aug 29 17:48:47 2016 +0530
summary: amfd: send msg to AMFND upon modification of safCsiAttr [#1553]

changeset: 7973:2c659b57d28d
user: praveen.malviya@oracle.com
date: Mon Aug 29 17:47:30 2016 +0530
summary: amf: add new D2ND message and new ND to Agent message. [#1553]

changeset: 7972:f2d92561c55c
user: praveen.malviya@oracle.com
date: Mon Aug 29 17:46:10 2016 +0530
summary: amf: amf: add saAmf_B_04_02.h for new intialize API, callback and callback structure [#1553]

changeset: 7971:b1503f8425a2
user: praveen.malviya@oracle.com
date: Mon Aug 29 17:43:04 2016 +0530
summary: amf: add README for implementation details [#1553]

Will update PR doc.

[Praveen]

Updated PR doc in :
changeset: 190:c4d93e4c37f7
tag: tip
user: praveen.malviya@oracle.com
date: Tue Sep 20 11:10:08 2016 +0530
summary: amf: update PR doc for enhancements [#1608], [#1454] and [#1553]