OpenSAF / Tickets / #1353 smf: step undoing is in progress forever until cluster reset

Anders Bjornerstedt - 2015-07-15

Milestone: future --> 4.6.1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anders Widell - 2015-11-02

Milestone: 4.6.1 --> 4.6.2
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Mathi Naickan - 2016-05-04

Milestone: 4.6.2 --> 4.7.2
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Madhurika Koppula - 2016-09-09

summary: smf: two hours is spent on step undoing state --> smf: step undoing is in progress forever until cluster reset

Attachments has changed:

Diff:

--- old +++ new @@ -1 +1,2 @@ +1353.tgz (475.2 kB; application/octet-stream) messages_step_undo (111.1 kB; application/octet-stream)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Madhurika Koppula - 2016-09-09

Steps to reproduce:

1) Execute middle-ware upgrade(5.0->5.1). Campaign is ran.
2) On the node (SC-2) which is being upgraded, the new rpms(5.1) are kept empty and the node came up without opensaf installation.

Observations:

1) Node(SC-2) went for reboot for upgrade.
2) As node SC-2 didnot join within 10 mins of time, step undoing is initiated. Rolling back of node reboot step is initiated by SMF.

Below is the snippet:

Sep 12 13:13:30 SLES-M-SLOT-1 osafamfd[2528]: NO Node 'SC-2' left the cluster
Sep 12 13:13:33 SLES-M-SLOT-1 osaffmd[2467]: NO Node Down event for node id 2020f:
Sep 12 13:13:33 SLES-M-SLOT-1 osaffmd[2467]: NO Current role: ACTIVE
Sep 12 13:13:33 SLES-M-SLOT-1 osaffmd[2467]: Rebooting OpenSAF NodeId = 131599 EE Name = , Reason: Received Node Down for peer controller, OwnNodeId = 131343, SupervisionTime = 60

Sep 12 13:23:38 SLES-M-SLOT-1 osafsmfd[2583]: NO SmfUpgradeStep::nodeReboot: the following nodes has not been correctly rebooted
Sep 12 13:23:38 SLES-M-SLOT-1 osafsmfd[2583]: NO Node safAmfNode=SC-2,safAmfCluster=myAmfCluster
Sep 12 13:23:38 SLES-M-SLOT-1 osafsmfd[2583]: ER Fails to reboot node safAmfNode=SC-2,safAmfCluster=myAmfCluster
Sep 12 13:23:38 SLES-M-SLOT-1 osafsmfd[2583]: ER Step execution failed, Try undoing the step

Sep 12 13:23:38 SLES-M-SLOT-1 osafsmfd[2583]: NO SmfStepStateUndoing::execute start undoing step.
Sep 12 13:23:38 SLES-M-SLOT-1 osafsmfd[2583]: NO STEP: Rolling back node reboot step safSmfStep=0002,safSmfProc=OpenSAF-upgrade,safSmfCampaign=UpgradeCampaign_7.0_7.1,safApp=safSmfService

3) Step undoing is in progress forever until cluster reset.

Attachments:

1) Syslog, smf traces of both the controllers.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anders Widell - 2016-09-20

Milestone: 4.7.2 --> 5.0.2
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Anders Widell - 2017-04-03

Milestone: 5.0.2 --> future
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Thuan Tran - 2018-09-25

status: unassigned --> accepted

assigned_to: Thuan

Blocker: --> False
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Thuan Tran - 2018-09-25

Milestone: future --> 5.18.09
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Thuan Tran - 2018-09-25

status: accepted --> review
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Gary Lee - 2018-09-29

commit e08db1131c686779e418fe1514deaecf666bf776
Author: thuan.tran thuan.tran@dektech.com.au
Date: Fri Sep 28 09:29:29 2018 +0000

smf: campaign is executing forever until cluster reset [#1353]

The function getNodeDestination() reset elapsedTime to zero cause
the node reboot timeout at waitForNodeDestination() never reach.
If scenario that node reboot cannot come back then campaign is stuck
in executing forever until cluster reset.

Related

Tickets: ~~#1353~~

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Gary Lee - 2018-09-29

status: review --> fixed
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

smf: step undoing is in progress forever until cluster reset

Milestone

Searches

Help

#1353 smf: step undoing is in progress forever until cluster reset

-------------------

Related

Discussion

Related