#2532 mds: TCP SVC_UP event is not received after subscribing

[Hung Nguyen]

MDS is successfully installed on IMMA and IMMA subscribed to IMMD successfully.
IMMND also received IMMA SVC_UP event but IMMA didn't receive SVC_UP event for IMMND.

<142>1 2017-07-20T13:00:36.072773+02:00 PL-4 immomtest 278 mds.log [meta sequenceId="14043"] MCM:API: svc_id = IMMA_OM(26) on VDEST id = 65535, SVC_PVT_VER = 0 Install Successfull
> ...
<142>1 2017-07-20T13:00:36.073091+02:00 PL-4 immomtest 278 mds.log [meta sequenceId="14074"] MCM:API: svc_subscribe :svc_id = IMMA_OM(26) on VDEST id = 65535 Subscription to svc_id = IMMND(25) Successful
> ...
<142>1 2017-07-20T13:00:36.073904+02:00 PL-4 osafimmnd 177 mds.log [meta sequenceId="96185"] MCM:API: svc_up : svc_id = IMMND(25) on DEST id = 65535 got UP for svc_id = IMMA_OM(26) on Adest = <node[4]:immomtest[278]>, rem_svc_pvt_ver=0, rem_svc_archword=10

IMMA waited for the SVC_UP event for 30 sec but didn't receive anything.

Jul 20 13:00:36.071465 imma [278:278:src/imm/agent/imma_init.cc:0263] >> imma_startup 
Jul 20 13:00:36.071474 imma [278:278:src/imm/agent/imma_init.cc:0273] TR use count 0
Jul 20 13:00:36.071484 imma [278:278:src/base/ncs_main_pub.c:0220] TR 
NCS:PROCESS_ID=278
Jul 20 13:00:36.071494 imma [278:278:src/base/sysf_def.c:0089] TR INITIALIZING LEAP ENVIRONMENT
Jul 20 13:00:36.071584 imma [278:278:src/base/sysf_def.c:0124] TR DONE INITIALIZING LEAP ENVIRONMENT
Jul 20 13:00:36.071832 imma [278:278:src/base/ncs_main_pub.c:0757] TR NCS:NODE_ID=0x0002040F
Jul 20 13:00:36.072329 imma [278:278:src/mbc/mbcsv_dl_api.c:0059] >> mbcsv_lib_req 
Jul 20 13:00:36.072350 imma [278:278:src/mbc/mbcsv_dl_api.c:0096] >> mbcsv_lib_init 
Jul 20 13:00:36.072378 imma [278:278:src/mbc/mbcsv_mbx.c:0174] >> mbcsv_initialize_mbx_list 
Jul 20 13:00:36.072389 imma [278:278:src/mbc/mbcsv_mbx.c:0189] << mbcsv_initialize_mbx_list 
Jul 20 13:00:36.072399 imma [278:278:src/mbc/mbcsv_pwe_anc.c:0158] >> mbcsv_initialize_peer_list 
Jul 20 13:00:36.072409 imma [278:278:src/mbc/mbcsv_pwe_anc.c:0173] << mbcsv_initialize_peer_list 
Jul 20 13:00:36.072419 imma [278:278:src/mbc/mbcsv_dl_api.c:0075] << mbcsv_lib_req 
Jul 20 13:00:36.072440 imma [278:278:src/base/ncs_main_pub.c:0389] TR 
MBCSV:MBCA:ON
Jul 20 13:00:36.073104 imma [278:278:src/imm/agent/imma_init.cc:0063] >> imma_sync_with_immnd 
Jul 20 13:00:36.073114 imma [278:278:src/imm/agent/imma_init.cc:0071] TR Blocking first client
Jul 20 13:01:06.102156 imma [278:278:src/imm/agent/imma_init.cc:0081] TR Blocking wait released
Jul 20 13:01:06.102375 imma [278:278:src/imm/agent/imma_init.cc:0091] << imma_sync_with_immnd 
Jul 20 13:01:06.102413 imma [278:278:src/imm/agent/imma_init.cc:0179] TR Client agent successfully initialized
Jul 20 13:01:06.102427 imma [278:278:src/imm/agent/imma_init.cc:0296] << imma_startup: use count 1

Attached is traces and logs.

[A V Mahesh (AVM)]

Hi Hung Nguyen,

I am not able to reproduce the problem with TIPC 2.0 , whic TIPC you are using ?

Do you have any any specific steps to reproduce ?

Not receiving delayed/No SVC_UP event is very fundamental problem ,
do you have any other real-time thread on your IMM application (except Opensaf) ?

-AVM

[A V Mahesh (AVM)]

Hi,

TCP was used, not TIPC.

Please do provide following :

Is the issue reproducible every time on multiple setups ?
Is the issue reproducible any other test other than executing immomtest ? ( logtest, ect ...)
Is the issue reproducible TIPC as well on same setups ? , this help us in isolating the problem to IMM or DTM/MDS.

Share setup details like VM or physical system or container setup ect..

[Hung Nguyen]

Hi,

It's very hard to reproduce the issue.

Only immomtest was run when the problem occured. The cluster was also in stable state, all nodes was not rebooting or starting.

We don't run tests with TIPC, so I don't know if this also happens with TIPC.

The cluster is lxc (Linux Containers).

Thanks,

[A V Mahesh (AVM)]

Hi,

I did run immomtest over night with TCP & TIPC, I haven't seen any issue , and also not able figureit out any thing form attached logs. as another try I will give by rebooting or starting nodes randomly while immomtestis running on another node as you were saying.

If still not reproducible, we no option that you need to run same test on TIPC on same setup where you observed the issue , this will help us isolating problem of IMM as if they are NO code changes in MDS TCP since last two release , we need IMMA traces to debug.

-AVM

[Hung Nguyen]

Hi,

In logs_n_traces.tgz file, I already included:

• IMMND trace (osafimmnd)
• IMMA trace (imma.trace)
• MDS log with MDS_LOG_LEVEL=5 for osafimmnd and immomtest (mds.log)
• MDS log with MDS_LOG_LEVEL=5 for immomtest (mds.log)

Just to be clear, are you requesting those traces for TIPC?

Thanks,

[A V Mahesh (AVM)]

Just to be clear, are you requesting those traces for TIPC?

I try to reproduce as well , as for your reproducible steps, but not able to reproduce on TCP or TIPC.
I did check logs and didn't get much clue , any how I will revisit the logs again and see if I get any clue.

If required we have to reproduce the issue with some additional IMMA/MDS debugging patch on same setup with TIPC or TCP where you observed the issue .

-AVM

[Zoran Milinkovic]

Hi Mahesh,

I just want to add that this issue is very hard to reproduce.

We see this issue at least once a day in our test environment.
I have never managed to reproduce the problem in my environment.

BR,
Zoran

[A V Mahesh (AVM)]

We see this issue at least once a day in our test environment

If so, can you try to run the same test with TIPC over night, with IMMA traces enabled.

[Zoran Milinkovic]

The tests have been started.
We'll come back when the problem is reproduced.