#2362 logd got crashed on Active controller during saLogStreamOpen_2

[Ritu Raj]

Environment details

OS : Suse 64bit
Changeset : 8634 ( 5.2.FC)
4 nodes setup(2 controller and 2 payload)

Summary

logd got crashed on Active controller during initializtion

Steps followed & Observed behaviour

1. Initially calling logsv saLogInitialize>saLogStreamOpen_2>saLogWriteLogAsync> saLogStreamClose>saLogFinalize with unsupported major version ('A',3,2) and got return value 'SA_AIS_ERR_VERSION' which is expected

2. Again, calling api saLogInitialize with version ('A',2,2) > saLogStreamOpen_2() on active controller and got crashed

Syslog:::
Mar 10 21:44:12 SO-SLOT-1 osafamfnd[2309]: NO 'safComp=LOG,safSu=SC-1,safSg=2N,safApp=OpenSAF' faulted due to 'avaDown' : Recovery is 'nodeFailfast'
Mar 10 21:44:12 SO-SLOT-1 osafamfnd[2309]: ER safComp=LOG,safSu=SC-1,safSg=2N,safApp=OpenSAF Faulted due to:avaDown Recovery is:nodeFailfast
Mar 10 21:44:12 SO-SLOT-1 osafamfnd[2309]: Rebooting OpenSAF NodeId = 131343 EE Name = , Reason: Component faulted: recovery is node failfast, OwnNodeId = 131343, SupervisionTime = 60

BT
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/lib64/opensaf/osaflogd --tracemask=0xffffffff'.
Program terminated with signal SIGSEGV, Segmentation fault.
0 0x00007f6e77d2e2a9 in free () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f6e76796700 (LWP 2470))]
(gdb) thread apply all bt

Thread 6 (Thread 0x7f6e79901720 (LWP 2269)):
0 0x00007f6e78033989 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
1 0x00007f6e7997b3ca in log_file_api(lgsf_apipar_t) ()
2 0x00007f6e799a6c2c in get_number_of_cfg_files_h(log_stream, char) ()
3 0x00007f6e799a3d6c in rotate_if_needed(log_stream) ()
4 0x00007f6e799a41cc in log_initiate_stream_files(log_stream) ()
5 0x00007f6e799a5af6 in log_stream_open_fileinit(log_stream) ()
6 0x00007f6e79978607 in proc_stream_open_msg(lgs_cb, lgsv_lgs_evt) ()
7 0x00007f6e799796a1 in process_api_evt(lgsv_lgs_evt) ()
8 0x00007f6e7997975b in lgs_process_mbx(unsigned int) ()
9 0x00007f6e79993fcd in main ()

Thread 5 (Thread 0x7f6e798deb00 (LWP 2273)):
0 0x00007f6e77d824f6 in poll () from /lib64/libc.so.6
1 0x00007f6e78a4133b in osaf_ppoll (io_fds=0x7f6e798de1c0, i_nfds=1, i_timeout_ts=0x7f6e798de180, i_sigmask=0x0) at src/base/osaf_poll.c:105
2 0x00007f6e78a41261 in osaf_poll (io_fds=0x7f6e798de1c0, i_nfds=1, i_timeout=30000) at src/base/osaf_poll.c:44
3 0x00007f6e78a41430 in osaf_poll_one_fd (i_fd=13, i_timeout=30000) at src/base/osaf_poll.c:128
4 0x00007f6e78a88360 in rda_read_msg(int, char, int) () from /usr/lib64/libopensaf_core.so.0
5 0x00007f6e78a87b40 in rda_callback_task(RDA_CALLBACK_CB) () from /usr/lib64/libopensaf_core.so.0
6 0x00007f6e7802f7b6 in start_thread () from /lib64/libpthread.so.0
7 0x00007f6e77d8b9cd in clone () from /lib64/libc.so.6
8 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f6e75f95700 (LWP 2921)):
0 0x00007f6e77d824f6 in poll () from /lib64/libc.so.6
1 0x00007f6e78a411d7 in osaf_poll_no_timeout (io_fds=0x7f6e79bdaa60 <applier_thread(void*)::fds>, i_nfds=2) at src/base/osaf_poll.c:32
2 0x00007f6e78a41233 in osaf_poll (io_fds=0x7f6e79bdaa60 <applier_thread(void*)::fds>, i_nfds=2, i_timeout=-1) at src/base/osaf_poll.c:42
3 0x00007f6e799926c1 in applier_thread(void*) ()
4 0x00007f6e7802f7b6 in start_thread () from /lib64/libpthread.so.0
5 0x00007f6e77d8b9cd in clone () from /lib64/libc.so.6
6 0x0000000000000000 in ?? ()</applier_thread(void*)::fds></applier_thread(void*)::fds>

Thread 3 (Thread 0x7f6e798feb00 (LWP 2272)):
0 0x00007f6e77d824f6 in poll () from /lib64/libc.so.6
1 0x00007f6e78a83958 in mdtm_process_recv_events () at src/mds/mds_dt_tipc.c:669
2 0x00007f6e7802f7b6 in start_thread () from /lib64/libpthread.so.0
3 0x00007f6e77d8b9cd in clone () from /lib64/libc.so.6
4 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f6e79933b00 (LWP 2271)):
0 0x00007f6e77d824f6 in poll () from /lib64/libc.so.6
1 0x00007f6e78a4133b in osaf_ppoll (io_fds=0x7f6e79933270, i_nfds=1, i_timeout_ts=0x7f6e799332a0, i_sigmask=0x0) at src/base/osaf_poll.c:105
2 0x00007f6e78a4b604 in ncs_tmr_wait () at src/base/sysf_tmr.c:406
3 0x00007f6e7802f7b6 in start_thread () from /lib64/libpthread.so.0
4 0x00007f6e77d8b9cd in clone () from /lib64/libc.so.6
5 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f6e76796700 (LWP 2470)):
0 0x00007f6e77d2e2a9 in free () from /lib64/libc.so.6
1 0x00007f6e7997e823 in get_number_of_cfg_files_hdl(void, void, unsigned long) ()
---Type <return> to continue, or q <return> to quit---
2 0x00007f6e7997aa78 in file_hndl_thread(void*) ()
3 0x00007f6e7802f7b6 in start_thread () from /lib64/libpthread.so.0
4 0x00007f6e77d8b9cd in clone () from /lib64/libc.so.6
5 0x0000000000000000 in ?? ()</return></return>

Notes:

1. Syslog of Active controller attached
2. logd trace attached

[Vu Minh Nguyen]

changeset: 8699:6c3291973c8e
tag: tip
user: Canh Van Truong canh.v.truong@dektech.com.au
date: Mon Mar 13 12:10:19 2017 +0100
summary: log: fix logd crash on Active side [#2362]