#86 ncks multislab extraction segfault

[Tom Hilinski]

I have a lat/lon/value netcdf file with ~61k records. I'm trying to extract ~20 intervals of records (ie. delete ~20 records) using ncks -d... Works up to a certain number of -d options; adding more causes a segfault. Here are the command-lines:

works:
ncks -4 -h -a --msa_user_order -d record,0,34319 -d record,34321,34617 -d record,34619,38722 -d record,38724,42582 -d record,42585,42898 -d record,42900,48391 -d record,48394,48865 -d record,48867,48545 $ncFileName $newNcFileName

segfault:
ncks -4 -h -a --msa_user_order -d record,0,34319 -d record,34321,34617 -d record,34619,38722 -d record,38724,42582 -d record,42585,42898 -d record,42900,48391 -d record,48394,48865 -d record,48867,48545 -d record,48548,49106 $ncFileName $newNcFileName

I made a debug build of NCO and ran the latter in gdb: here's the traceback and debug session after the segfault:

---

Starting program: /usr/local/bin/ncks -4 -h -a --msa_user_order -d record,0,34319 -d record,34321,34617 -d record,34619,38722 -d record,38724,42582 -d record,42585,42898 -d record,42900,48391 -d record,48394,48865 -d record,48867,48545 -d record,48548,49106 Sites.nc Sites_cleaned.nc
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.19-17.4.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7b5141b in nco_msa_clc_cnt_trv (dmn_trv=0x715ef0) at nco_msa.c:957
957 for(idx=0;idx<size;idx++) cnt+="dmn_trv-">lmt_msa.lmt_dmn[idx]->cnt;
Missing separate debuginfos, use: zypper install gsl-debuginfo-1.16-6.1.x86_64 krb5-debuginfo-1.12.1-24.1.x86_64 libcom_err2-debuginfo-1.42.11-10.2.x86_64 libcurl4-debuginfo-7.37.0-7.1.x86_64 libexpat1-debuginfo-2.1.0-15.4.x86_64 libgcc_s1-debuginfo-5.2.1+r226025-4.2.x86_64 libgomp1-debuginfo-5.2.1+r226025-4.2.x86_64 libidn11-debuginfo-1.28-4.1.x86_64 libkeyutils1-debuginfo-1.5.9-4.3.x86_64 libldap-2_4-2-debuginfo-2.4.41-11.1.x86_64 libopenssl1_0_0-debuginfo-1.0.1i-9.1.x86_64 libpciaccess0-debuginfo-0.13.4-1.1.x86_64 libpcre1-debuginfo-8.33-3.5.x86_64 libsasl2-3-debuginfo-2.1.26-8.1.x86_64 libselinux1-debuginfo-2.3-5.1.x86_64 libssh2-1-debuginfo-1.4.3-10.1.x86_64 libstdc++6-debuginfo-5.2.1+r226025-4.2.x86_64 libz1-debuginfo-1.2.8-6.4.x86_64</size;idx++)>

(gdb) bt

#0  0x00007ffff7b5141b in nco_msa_clc_cnt_trv (dmn_trv=0x715ef0) at nco_msa.c:957
#1  0x00007ffff7b32105 in nco_bld_lmt (nc_id=65536, MSA_USR_RDR=1, lmt_nbr=9, lmt=0x716f70, 
    FORTRAN_IDX_CNV=0, trv_tbl=0x61d330) at nco_grp_utl.c:6878
#2  0x00007ffff7b30af3 in nco_bld_trv_tbl (nc_id=65536, grp_pth=0x7ffffffe76e0 "/", lmt_nbr=9, 
    lmt_arg=0x7fffffff76f0, aux_nbr=0, aux_arg=0x7fffffffb700, MSA_USR_RDR=1, FORTRAN_IDX_CNV=0, 
    grp_lst_in=0x0, grp_lst_in_nbr=0, var_lst_in=0x0, var_xtr_nbr=0, EXTRACT_ALL_COORDINATES=0, flg_unn=0, 
    GRP_XTR_VAR_XCL=0, EXCLUDE_INPUT_LIST=0, EXTRACT_ASSOCIATED_COORDINATES=1, nco_pck_plc=0, 
    flg_dne=0x7ffffffe7608, trv_tbl=0x61d330) at nco_grp_utl.c:6530
#3  0x0000000000404da8 in main (argc=25, argv=0x7fffffffdac8) at ncks.c:887

(gdb) l
952 return;
953 } / end if /
954
955 / If slabs remain in user-specified order /
956 if(dmn_trv->lmt_msa.MSA_USR_RDR){
957 for(idx=0;idx<size;idx++) cnt+="dmn_trv-">lmt_msa.lmt_dmn[idx]->cnt;
958 dmn_trv->lmt_msa.dmn_cnt=cnt;
959 }else{
960 indices=(long )nco_malloc(sizesizeof(long));
961 mnm=(nco_bool )nco_malloc(sizesizeof(nco_bool));</size;idx++)>

(gdb) info locals
idx = 9
cnt = 109880
size = 10
indices = 0x715ef0
mnm = 0x71d446

(gdb) p dmn_trv->lmt_msa
$3 = {
dmn_nm = 0x715fe0 "record",
dmn_sz_org = 61343,
dmn_cnt = 61343,
NON_HYP_DMN = 0,
WRP = 0,
MSA_USR_RDR = 1,
lmt_dmn_nbr = 10,
lmt_dmn = 0x71d460,
lmt_crr = 10
}

(gdb) p dmn_trv->lmt_msa.lmt_dmn[idx]
$6 = (lmt_sct *) 0x0 <-- segfault

---

Any suggestions for a fix or workaround?

---

netdf file header:

netcdf Sites {
dimensions:
record = 61343 ;
count = 364 ;
variables:
float lat(record) ;
float lon(record) ;
uint id(record) ;
byte siteType(record) ;
string name(record) ;
string description(record) ;
float values(record, count) ;
string names(count) ;
}

---

NCO version 4.5.4

[Tom Hilinski]

Following this in gdb, this is what I learned:

file: nco_msa.c
In function:
nco_msa_wrp_splt_trv
breaking at line 908:
dmn_trv->lmt_msa.lmt_dmn=(lmt_sct )nco_realloc(dmn_trv->lmt_msa.lmt_dmn,(lmt_dmn_nbr+1)*sizeof(lmt_sct *));
some vars:

  size (loop limit) = 9
  lmt_dmn_nbr = 9
  lmt_new_idx = 8
  idx = 7

but alloc is for lmt_dmn_nbr + 1 == 10 elements

After alloc:
(gdb) p dmn_trv->lmt_msa.lmt_dmn[9]
$19 = (lmt_sct ) 0x0
(gdb) p dmn_trv->lmt_msa.lmt_dmn[8]
$20 = (lmt_sct ) 0x718630

lines 908-918:

  dmn_trv->lmt_msa.lmt_dmn=(lmt_sct )nco_realloc(dmn_trv->lmt_msa.lmt_dmn,(lmt_dmn_nbr+1)*sizeof(lmt_sct *));

  /* Alloc the extra limit  */
  dmn_trv->lmt_msa.lmt_dmn[lmt_new_idx]=(lmt_sct *)nco_malloc(sizeof(lmt_sct));

  /* Initialize the extra limit */
  (void)nco_lmt_init(dmn_trv->lmt_msa.lmt_dmn[lmt_new_idx]);

  /* Insert the limits to table (allocated; idx was already there; lmt_new_idx was alloced here)   */
  (void)nco_lmt_cpy(&lmt_wrp[0],dmn_trv->lmt_msa.lmt_dmn[idx]);
  (void)nco_lmt_cpy(&lmt_wrp[1],dmn_trv->lmt_msa.lmt_dmn[lmt_new_idx]);

lmt_dmn[9] is never initialzed because the init lines use different indices into that array, and the init index (lmt_new_idx) is less than (the number of elements allocated - 1).

[Charlie Zender]

Hi Tom,
This does appear to be a bug, thanks for reporting it. We'll post again when we've made progress in fixing it.
cz

[Pedro Vicente]

Tom
Can you put that file "Sites.nc", or any other that reproduces the bug, somewhere where we can get it?
-Pedro

[Pedro Vicente]

Tom
I was able to reproduce the issue, that happens only for "wrapped" limits, that are limits where the first index is greater than the second, like this one
-d record,48867,48545
Just wanted to make sure that this is indeed what you need
-Pedro

[Charlie Zender]

There has been no activity on this for over a year. Pedro showed that it was due to combingin multislabs with wrapped indices, which may have been a typo on the part of the user. While it shouldn't segfault, this is too much of a corner case to spend precious developer time on. Can be re-opened if the wheel gets squeaky...