#2323 imm: CCB operations fail after SC absence (Headless)

[Hung Nguyen]

Reproduce steps:

1. Start SC-1
2. Commit some CCBs
    # immcfg -c Test test=0
    # immcfg -c Test test=1
    # immcfg -c Test test=2
    # immcfg -c Test test=3
3. Start PL-3
4. Restart SC-1
5. When SC-1 is back, it fails to add operations to CCB
    # immcfg -c Test test=10
    error - saImmOmCcbObjectCreate_2 FAILED with SA_AIS_ERR_FAILED_OPERATION (21)
    OI reports: IMM: Resource abort: CCB is not in an expected state
    error - saImmOmCcbApply FAILED: SA_AIS_ERR_FAILED_OPERATION (21)
    OI reports: IMM: Resource abort: CCB is not in an expected state

cb->mLatestCcbId was not updated on PL-3 when it joined the cluster so it still had value of zero.

When SC-1 was back from headless, IMMND on PL-3 sent re-introduce message to IMMD on SC-1 with cb->mLatestCcbId = 0.

IMMD failed to update cb->ccb_id_count so when new CCB is created, it will start from 0+1 instead of mLatestCcbId + 1.

That results in the conflict with the CCB in sCcbVector and the CCB operation failure.

Attached is logs and traces.

[Anders Bjornerstedt]

Added "headless" clarification because "SC absence" can be missunderstood as just one (out of normally two) SCs being absent.

[Hung Nguyen]

default (5.2) [staging:8cbe1c]
changeset: 8669:8cbe1c0a3335
user: Hung Nguyen hung.d.nguyen@dektech.com.au
date: Wed Mar 08 13:14:20 2017 +0700
summary: imm: Sync latest ccb-id to sync clients [#2323]

opensaf-5.1.x [staging:b642a1]
changeset: 8670:b642a1336cf3
user: Hung Nguyen hung.d.nguyen@dektech.com.au
date: Wed Mar 08 13:35:24 2017 +0700
summary: imm: Sync latest ccb-id to sync clients [#2323]

opensaf-5.0.x [staging:bf11d1]
changeset: 8671:bf11d1e7720a
user: Hung Nguyen hung.d.nguyen@dektech.com.au
date: Wed Mar 08 13:35:24 2017 +0700
summary: imm: Sync latest ccb-id to sync clients [#2323]