Menu

#3250 imm: immnd dectect headless state wrong

5.21.03
fixed
None
defect
imm
nd
major
False
2021-03-01
2021-01-28
No

from MDS thread, that receive in order event : IMMD RED_DOWN from standby , NCSMDS_DOWN, IMMD RED_DOWN from active. All event from MDS thread will be going to IMMND mail box. In event NCSMDS_DOWN was mark active node down.

IMMND receive IMMD RED_DOWN (standby) from MDS thread, then enter the headless state. IMMND enter the headless state due to ACTIVE and STANDBY is down. So the entering headless state is unexpected.

<143>1 2021-01-25T21:49:35.467747+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="115"] 4916:imm/common/immsv_evt.c:6569 T8 Received: IMMND_EVT_MDS_INFO (1) from 0
<143>1 2021-01-25T21:49:35.467758+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="116"] 4916:imm/immnd/immnd_evt.c:12290 T2 IMMD RED_DOWN EVENT 2010f role=2 ==> ACT:0 SBY:0
<140>1 2021-01-25T21:49:35.468554+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="117"] 4916:imm/immnd/immnd_evt.c:12292 WA Both Active & Standby DOWN, going to headless
<140>1 2021-01-25T21:49:35.468578+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="118"] 4916:imm/immnd/immnd_evt.c:12315 WA SC Absence IS allowed:900 IMMD service is DOWN ex_immd_node_id=2020f
<141>1 2021-01-25T21:49:35.468595+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="119"] 4916:imm/immnd/immnd_evt.c:12395 NO IMMD SERVICE IS DOWN, HYDRA IS CONFIGURED => UNREGISTERING IMMND form MDS
<143>1 2021-01-25T21:49:35.468602+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="120"] 4916:imm/immnd/immnd_mds.c:256 >> immnd_mds_unregister
...
<143>1 2021-01-25T21:49:36.495445+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="237"] 4916:imm/common/immsv_evt.c:6569 T8 Received: IMMND_EVT_MDS_INFO (1) from 0
<143>1 2021-01-25T21:49:36.49548+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="238"] 4916:imm/immnd/immnd_evt.c:12252 T2 NCSMDS_DOWN 0
<140>1 2021-01-25T21:49:36.495548+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="239"] 4916:imm/immnd/immnd_evt.c:12315 WA SC Absence IS allowed:900 IMMD service is DOWN ex_immd_node_id=2020f
<141>1 2021-01-25T21:49:36.495582+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="240"] 4916:imm/immnd/immnd_evt.c:12395 NO IMMD SERVICE IS DOWN, HYDRA IS CONFIGURED => UNREGISTERING IMMND form MDS
<143>1 2021-01-25T21:49:36.495595+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="241"] 4916:imm/immnd/immnd_mds.c:256 >> immnd_mds_unregister
...
<143>1 2021-01-25T21:49:37.49666+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="275"] 4916:imm/common/immsv_evt.c:6569 T8 Received: IMMND_EVT_MDS_INFO (1) from 0
<143>1 2021-01-25T21:49:37.496675+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="276"] 4916:imm/immnd/immnd_evt.c:12290 T2 IMMD RED_DOWN EVENT 2020f role=1 ==> ACT:0 SBY:0
<140>1 2021-01-25T21:49:37.496744+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="277"] 4916:imm/immnd/immnd_evt.c:12292 WA Both Active & Standby DOWN, going to headless
<140>1 2021-01-25T21:49:37.496786+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="278"] 4916:imm/immnd/immnd_evt.c:12315 WA SC Absence IS allowed:900 IMMD service is DOWN ex_immd_node_id=2020f
<141>1 2021-01-25T21:49:37.496809+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="279"] 4916:imm/immnd/immnd_evt.c:12395 NO IMMD SERVICE IS DOWN, HYDRA IS CONFIGURED => UNREGISTERING IMMND form MDS
<143>1 2021-01-25T21:49:37.496819+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId="280"] 4916:imm/immnd/immnd_mds.c:256 >> immnd_mds_unregister

Related

Wiki: ChangeLog-5.21.03

Discussion

  • Thien Minh Huynh

    • summary: imm: immnd dectect heas --> imm: immnd dectect headless state wrong
     
  • Thien Minh Huynh

    • Description has changed:

    Diff:

    --- old
    +++ new
    @@ -1,22 +1,21 @@
    -~~~
    -&lt;143&gt;1 2021-01-25T21:49:35.467747+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;115&#34;] 4916:imm/common/immsv_evt.c:6569 T8 Received: IMMND_EVT_MDS_INFO (1) from 0
    -&lt;143&gt;1 2021-01-25T21:49:35.467758+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;116&#34;] 4916:imm/immnd/immnd_evt.c:12290 T2 IMMD RED_DOWN EVENT 2010f role=2 ==&gt; ACT:0 SBY:0
    +IMMND enter headless state when receive IMMD RED_DOWN EVENT from SC-1 (standby). That is expected no headless state.
    +
    +&gt; &lt;143&gt;1 2021-01-25T21:49:35.467747+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;115&#34;] 4916:imm/common/immsv_evt.c:6569 T8 Received: IMMND_EVT_MDS_INFO (1) from 0
    +&lt;143&gt;1 2021-01-25T21:49:35.467758+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;116&#34;] 4916:imm/immnd/immnd_evt.c:12290 T2 **IMMD RED_DOWN EVENT 2010f role=2 ==&gt; ACT:0 SBY:0**
     &lt;140&gt;1 2021-01-25T21:49:35.468554+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;117&#34;] 4916:imm/immnd/immnd_evt.c:12292 WA Both Active &amp; Standby DOWN, going to headless
     &lt;140&gt;1 2021-01-25T21:49:35.468578+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;118&#34;] 4916:imm/immnd/immnd_evt.c:12315 WA SC Absence IS allowed:900 IMMD service is DOWN ex_immd_node_id=2020f
     &lt;141&gt;1 2021-01-25T21:49:35.468595+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;119&#34;] 4916:imm/immnd/immnd_evt.c:12395 NO IMMD SERVICE IS DOWN, HYDRA IS CONFIGURED =&gt; UNREGISTERING IMMND form MDS
     &lt;143&gt;1 2021-01-25T21:49:35.468602+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;120&#34;] 4916:imm/immnd/immnd_mds.c:256 &gt;&gt; immnd_mds_unregister 
     ...
     &lt;143&gt;1 2021-01-25T21:49:36.495445+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;237&#34;] 4916:imm/common/immsv_evt.c:6569 T8 Received: IMMND_EVT_MDS_INFO (1) from 0
    -&lt;143&gt;1 2021-01-25T21:49:36.49548+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;238&#34;] 4916:imm/immnd/immnd_evt.c:12252 T2 NCSMDS_DOWN 0
    +&lt;143&gt;1 2021-01-25T21:49:36.49548+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;238&#34;] 4916:imm/immnd/immnd_evt.c:12252 T2 **NCSMDS_DOWN 0**
     &lt;140&gt;1 2021-01-25T21:49:36.495548+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;239&#34;] 4916:imm/immnd/immnd_evt.c:12315 WA SC Absence IS allowed:900 IMMD service is DOWN ex_immd_node_id=2020f
     &lt;141&gt;1 2021-01-25T21:49:36.495582+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;240&#34;] 4916:imm/immnd/immnd_evt.c:12395 NO IMMD SERVICE IS DOWN, HYDRA IS CONFIGURED =&gt; UNREGISTERING IMMND form MDS
     &lt;143&gt;1 2021-01-25T21:49:36.495595+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;241&#34;] 4916:imm/immnd/immnd_mds.c:256 &gt;&gt; immnd_mds_unregister 
     ...
     &lt;143&gt;1 2021-01-25T21:49:37.49666+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;275&#34;] 4916:imm/common/immsv_evt.c:6569 T8 Received: IMMND_EVT_MDS_INFO (1) from 0
    -&lt;143&gt;1 2021-01-25T21:49:37.496675+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;276&#34;] 4916:imm/immnd/immnd_evt.c:12290 T2 IMMD RED_DOWN EVENT 2020f role=1 ==&gt; ACT:0 SBY:0
    +&lt;143&gt;1 2021-01-25T21:49:37.496675+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;276&#34;] 4916:imm/immnd/immnd_evt.c:12290 T2 **IMMD RED_DOWN EVENT 2020f role=1 ==&gt; ACT:0 SBY:0**
     &lt;140&gt;1 2021-01-25T21:49:37.496744+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;277&#34;] 4916:imm/immnd/immnd_evt.c:12292 WA Both Active &amp; Standby DOWN, going to headless
     &lt;140&gt;1 2021-01-25T21:49:37.496786+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;278&#34;] 4916:imm/immnd/immnd_evt.c:12315 WA SC Absence IS allowed:900 IMMD service is DOWN ex_immd_node_id=2020f
     &lt;141&gt;1 2021-01-25T21:49:37.496809+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;279&#34;] 4916:imm/immnd/immnd_evt.c:12395 NO IMMD SERVICE IS DOWN, HYDRA IS CONFIGURED =&gt; UNREGISTERING IMMND form MDS
     &lt;143&gt;1 2021-01-25T21:49:37.496819+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;280&#34;] 4916:imm/immnd/immnd_mds.c:256 &gt;&gt; immnd_mds_unregister 
    -
    -~~~
    
     
  • Thien Minh Huynh

    • status: accepted --> review
     
  • Thien Minh Huynh

    • Description has changed:

    Diff:

    --- old
    +++ new
    @@ -1,4 +1,6 @@
    -IMMND enter headless state when receive IMMD RED_DOWN EVENT from SC-1 (standby). That is expected no headless state.
    +from MDS thread, that receive in order event : **IMMD RED_DOWN** from standby , **NCSMDS_DOWN**, **IMMD RED_DOWN** from active. All event from MDS thread will be going to IMMND mail box. In event **NCSMDS_DOWN** was mark active node down. 
    +
    +IMMND receive  **IMMD RED_DOWN** (standby) from MDS thread, then enter the headless state. IMMND enter the headless state due to ACTIVE and STANDBY is down. So the entering headless state is unexpected.
    
     &gt; &lt;143&gt;1 2021-01-25T21:49:35.467747+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;115&#34;] 4916:imm/common/immsv_evt.c:6569 T8 Received: IMMND_EVT_MDS_INFO (1) from 0
     &lt;143&gt;1 2021-01-25T21:49:35.467758+11:00 PL-2-4 osafimmnd 4916 osafimmnd [meta sequenceId=&#34;116&#34;] 4916:imm/immnd/immnd_evt.c:12290 T2 **IMMD RED_DOWN EVENT 2010f role=2 ==&gt; ACT:0 SBY:0**
    
     
  • Thien Minh Huynh

    • status: review --> fixed
     
  • Thien Minh Huynh

    commit 54bb248ee217f5ba619a7c2389cb10e2b0fefaab (HEAD -> develop, origin/develop)
    Author: thien.m.huynh thien.m.huynh@dektech.com.au
    Date: Thu Jan 28 15:58:19 2021 +0700

    imm: fix immnd detect headless state wrong [#3250]
    
    From MDS thread, that receives in order event: IMMD_RED_DOWN from standby,
    NCSMDS_DOWN, IMMD_RED_DOWN from active. All event from MDS thread will be
    sent to the mail box. MDS thread marks active down when receive NCSMDS_DOWN
    event. In main thread the IMMND enter headless state when receive the first
    event IMMD_RED_DOWN. Headless state is unexpected.
    
    The solution just mark node down when receive NCSMDS_DOWN on the main thread.
    That is prevent IMMND detect headless state wrong.
    
     
  • Thien Minh Huynh

    • status: fixed --> review
     
  • Thien Minh Huynh

    • status: review --> fixed
     
  • Thien Minh Huynh

    commit 76b7a650effce95575add26d5b25d500b5a441fd (HEAD -> develop, origin/develop)
    Author: thien.m.huynh thien.m.huynh@dektech.com.au
    Date: Fri Feb 5 13:26:57 2021 +0700

    imm: ignore enter headless if already entered [#3250]
    
    immnd going to headless twice when receive event NCSMDS_DOWN, NCSMDS_RED_DOWN
    in the same time.
    
    The fix is prevent go to headless many time.
    
     
  • Gary Lee

    Gary Lee - 2021-03-01
    • Milestone: 5.20.11 --> 5.21.03
     

Log in to post a comment.