#2202 cpnd: osafckptnd core dump in high memory load

[Vo Minh Hoang]

Coredump occur while creating checkpoint section in high memory load, sharemem guarantee is not enable.

Core was generated by `/usr/lib64/opensaf/osafckptnd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f38f8513109 in __strtok_r_1c () from /lib64/libc.so.6
Missing separate debuginfos, use: zypper install opensaf-ckpt-nodedirector-debuginfo-5.1.0-690.0.d0f65c1.sle12.x86_64
(gdb) where
#0  0x00007f38f8513109 in __strtok_r_1c () from /lib64/libc.so.6
#1  0x00007f38f9fc074a in memcpy (__len=<optimized out>, __src=<optimized out>, 
    __dest=<optimized out>) at /usr/include/bits/string3.h:51
#2  ncs_os_posix_shm (req=req@entry=0x7fff5de1f6b0)
    at ../../../../../../opensaf/osaf/libs/core/leap/os_defs.c:858
#3  0x0000000000415f6f in cpnd_sec_hdr_update (sec_info=sec_info@entry=0x19dc880, 
    cp_node=cp_node@entry=0x19dc3e0)
    at ../../../../../../../opensaf/osaf/services/saf/cpsv/cpnd/cpnd_proc.c:1875
#4  0x000000000040673a in cpnd_ckpt_sec_add (cp_node=0x19dc3e0, id=0x7f38f0008a00, 
    exp_time=1478796221720867000, gen_flag=gen_flag@entry=0)
    at ../../../../../../../opensaf/osaf/services/saf/cpsv/cpnd/cpnd_db.c:456
#5  0x000000000040d718 in cpnd_evt_proc_ckpt_sect_create (cb=cb@entry=0x18337f0, 
    evt=evt@entry=0x7f38f000ad80, sinfo=sinfo@entry=0x7f38f000b3d8)
    at ../../../../../../../opensaf/osaf/services/saf/cpsv/cpnd/cpnd_evt.c:2244
#6  0x000000000040eff4 in cpnd_process_evt (evt=0x7f38f000ad70)
    at ../../../../../../../opensaf/osaf/services/saf/cpsv/cpnd/cpnd_evt.c:227
#7  0x0000000000410bcd in cpnd_main_process (cb=cb@entry=0x18337f0)
    at ../../../../../../../opensaf/osaf/services/saf/cpsv/cpnd/cpnd_init.c:579
#8  0x0000000000405a83 in main (argc=<optimized out>, argv=<optimized out>)
    at ../../../../../../../opensaf/osaf/services/saf/cpsv/cpnd/cpnd_main.c:79

[A V Mahesh (AVM)]

We are some how able to simulated one case where slimier core dump occurs.
their can some other case this core dump can occur , as soon as we fund root cause
we will provide the patch for that as well.
In this test case , If OSAF_CKPT_SHM_ALLOC_GUARANTEE is NOT set and SHM is 100% used in system ,
cpnd Segmentation fault (core dumped) at LEAP memcpy().
Following are the detailed steps how we reproduced ,
this test is generating same core dumb as below.
Test application : cpsv_shm_2202.c
==================================================================
1) /etc/init.d/opensafd stop
2) Change the defaults /dev/shm size to 3MB
# vi /etc/fstab tmpfs

And add following line

tmpfs /dev/shm tmpfs defaults,size=3m 0 0
3) Remount /dev/shm

mount -o remount /dev/shm

4) Check /dev/shm reflected with new value

df -k /dev/shm/

Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 3072 0 3072 0% /dev/shm
5) set ulimit to unlimited

ulimit -c unlimited

6) #/etc/init.d/opensafd start
7) Compile & run attached test application ( cpsv_shm_2202.c )

gcc cpsv_shm_2202.c -o ckpt_shm -lSaCkpt

./ckpt_shm

8) Once /dev/shm/ reach 100% Use you will see core dump same as yours

df -k /dev/shm/

7) Then we applied the patch test again with no core dump
saCkptSectionCreate 1 returned 18. ( no core dump )

[A V Mahesh (AVM)]

changeset: 8395:67980ad1a5c9
branch: opensaf-5.1.x
user: A V Mahesh mahesh.valla@oracle.com
date: Thu Dec 01 14:34:35 2016 +0530
summary: mqsv : update mqsv error handing based on leap changes [#2202]

changeset: 8394:1c5e9ad7410b
branch: opensaf-5.1.x
user: A V Mahesh mahesh.valla@oracle.com
date: Thu Dec 01 14:34:25 2016 +0530
summary: cpsv : update cpsv error handing based on leap changes [#2202]

changeset: 8393:ea7d1df6002a
branch: opensaf-5.1.x
parent: 8383:f82d380b667f
user: A V Mahesh mahesh.valla@oracle.com
date: Thu Dec 01 14:34:15 2016 +0530
summary: leap: leap library ensure shm availability before writing [#2202]

changeset: 8392:119ad64e95b0
user: A V Mahesh mahesh.valla@oracle.com
date: Thu Dec 01 14:33:34 2016 +0530
summary: mqsv : update mqsv error handing based on leap changes [#2202]

changeset: 8391:19b261b7199f
user: A V Mahesh mahesh.valla@oracle.com
date: Thu Dec 01 14:33:02 2016 +0530
summary: cpsv : update cpsv error handing based on leap changes [#2202]

changeset: 8390:181b9f179980
user: A V Mahesh mahesh.valla@oracle.com
date: Thu Dec 01 14:32:21 2016 +0530
summary: leap: leap library ensure shm availability before writing [#2202]