CMU Sphinx / Forums / Help: program crashes after adding more than 4 words with ps_add

Hi,
so this problem I have may not be so easy to describe.
I have created a python script and use pocketsphinx 0.1.0 for python.
I created a little additional dictionary that contains some words and their pronunciation. At the start of my program and after the configuration of the decoder I call the function add_word(word,pron,update) in a loop with the words and pronunciations from the lines of my additional text file dict. I choose update=True only for the last word added.
This seems to work fine, the words can be recognized if I decode files with these spoken words in them. Also with lookup_word they are found.

Now there is one problem, I narrowed my problem down step by step and it is this:
If I add more than 4 words with add_word and then decode an audiofile where one of the words, being the 5th or 5+Nth word added, are recognized (or should be recognized), my program just crashes and in my case windows says "python has stopped working".
I reinstalled python3.5.2 and pocketshpinx but there's no difference. Through the narrowing down the problem I can see that the SAME word in the SAME audiofile is being recognized if the word is added via add_word e.g. as the 4th new word.

I find this very confusing, the log also is not that interesting because the program crashes but I will post it anyway.

Current configuration:
[NAME]          [DEFLT]     [VALUE]
-agc            none        none
-agcthresh      2.0     2.000000e+00
-allphone               
-allphone_ci        no      no
-alpha          0.97        9.700000e-01
-ascale         20.0        2.000000e+01
-aw         1       1
-backtrace      no      no
-beam           1e-48       1.000000e-48
-bestpath       yes     yes
-bestpathlw     9.5     9.500000e+00
-ceplen         13      13
-cmn            live        batch
-cmninit        40,3,-1     40,3,-1
-compallsen     no      no
-debug                  0
-dict                   ./models/ger2/voxforge.dic
-dictcase       no      no
-dither         no      no
-doublebw       no      no
-ds         1       1
-fdict                  
-feat           1s_c_d_dd   1s_c_d_dd
-featparams             
-fillprob       1e-8        1.000000e-08
-frate          100     100
-fsg                    
-fsgusealtpron      yes     yes
-fsgusefiller       yes     yes
-fwdflat        yes     yes
-fwdflatbeam        1e-64       1.000000e-64
-fwdflatefwid       4       4
-fwdflatlw      8.5     8.500000e+00
-fwdflatsfwin       25      25
-fwdflatwbeam       7e-29       7.000000e-29
-fwdtree        yes     yes
-hmm                    ./models/ger2/voxforge.cd_cont_3000
-input_endian       little      little
-jsgf                   
-keyphrase              
-kws                    
-kws_delay      10      10
-kws_plp        1e-1        1.000000e-01
-kws_threshold      1       1.000000e+00
-latsize        5000        5000
-lda                    
-ldadim         0       0
-lifter         0       22
-lm                 ./models/ger2/voxforge.lm.dmp
-lmctl                  
-lmname                 
-logbase        1.0001      1.000100e+00
-logfn                  
-logspec        no      no
-lowerf         133.33334   1.300000e+02
-lpbeam         1e-40       1.000000e-40
-lponlybeam     7e-29       7.000000e-29
-lw         6.5     6.500000e+00
-maxhmmpf       30000       30000
-maxwpf         -1      -1
-mdef                   
-mean                   
-mfclogdir              
-min_endfr      0       0
-mixw                   
-mixwfloor      0.0000001   1.000000e-07
-mllr                   
-mmap           yes     yes
-ncep           13      13
-nfft           512     512
-nfilt          40      25
-nwpen          1.0     1.000000e+00
-pbeam          1e-48       1.000000e-48
-pip            1.0     1.000000e+00
-pl_beam        1e-10       1.000000e-10
-pl_pbeam       1e-10       1.000000e-10
-pl_pip         1.0     1.000000e+00
-pl_weight      3.0     3.000000e+00
-pl_window      5       5
-rawlogdir              
-remove_dc      no      no
-remove_noise       yes     yes
-remove_silence     yes     yes
-round_filters      yes     yes
-samprate       16000       1.600000e+04
-seed           -1      -1
-sendump                
-senlogdir              
-senmgau                
-silprob        0.005       5.000000e-03
-smoothspec     no      no
-svspec                 
-tmat                   
-tmatfloor      0.0001      1.000000e-04
-topn           4       4
-topn_beam      0       0
-toprule                
-transform      legacy      dct
-unit_area      yes     yes
-upperf         6855.4976   6.800000e+03
-uw         1.0     1.000000e+00
-vad_postspeech     50      50
-vad_prespeech      20      20
-vad_startspeech    10      10
-vad_threshold      2.0     2.000000e+00
-var                    
-varfloor       0.0001      1.000000e-04
-varnorm        no      no
-verbose        no      no
-warp_params                
-warp_type      inverse_linear  inverse_linear
-wbeam          7e-29       7.000000e-29
-wip            0.65        6.500000e-01
-wlen           0.025625    2.562500e-02

INFO: feat.c(715): Initializing feature stream to type: '1s_c_d_dd', ceplen=13, CMN='batch', VARNORM='no', AGC='none'
INFO: cmn.c(97): mean[0]= 12.00, mean[1..12]= 0.0
INFO: acmod.c(156): Reading linear feature transformation from ./models/ger2/voxforge.cd_cont_3000/feature_transform
INFO: mdef.c(518): Reading model definition: ./models/ger2/voxforge.cd_cont_3000/mdef
INFO: bin_mdef.c(181): Allocating 82313 * 8 bytes (643 KiB) for CD tree
INFO: tmat.c(206): Reading HMM transition probability matrices: ./models/ger2/voxforge.cd_cont_3000/transition_matrices
INFO: acmod.c(117): Attempting to use PTM computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: ./models/ger2/voxforge.cd_cont_3000/means
INFO: ms_gauden.c(242): 3198 codebook, 1 feature, size: 
INFO: ms_gauden.c(244):  16x29
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: ./models/ger2/voxforge.cd_cont_3000/variances
INFO: ms_gauden.c(242): 3198 codebook, 1 feature, size: 
INFO: ms_gauden.c(244):  16x29
INFO: ms_gauden.c(304): 3398 variance values floored
INFO: ptm_mgau.c(804): Number of codebooks exceeds 256: 3198
INFO: acmod.c(119): Attempting to use semi-continuous computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: ./models/ger2/voxforge.cd_cont_3000/means
INFO: ms_gauden.c(242): 3198 codebook, 1 feature, size: 
INFO: ms_gauden.c(244):  16x29
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: ./models/ger2/voxforge.cd_cont_3000/variances
INFO: ms_gauden.c(242): 3198 codebook, 1 feature, size: 
INFO: ms_gauden.c(244):  16x29
INFO: ms_gauden.c(304): 3398 variance values floored
INFO: acmod.c(121): Falling back to general multi-stream GMM computation
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: ./models/ger2/voxforge.cd_cont_3000/means
INFO: ms_gauden.c(242): 3198 codebook, 1 feature, size: 
INFO: ms_gauden.c(244):  16x29
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: ./models/ger2/voxforge.cd_cont_3000/variances
INFO: ms_gauden.c(242): 3198 codebook, 1 feature, size: 
INFO: ms_gauden.c(244):  16x29
INFO: ms_gauden.c(304): 3398 variance values floored
INFO: ms_senone.c(149): Reading senone mixture weights: ./models/ger2/voxforge.cd_cont_3000/mixture_weights
INFO: ms_senone.c(200): Truncating senone logs3(pdf) values by 10 bits
INFO: ms_senone.c(207): Not transposing mixture weights in memory
INFO: ms_senone.c(268): Read mixture weights for 3198 senones: 1 features x 16 codewords
INFO: ms_senone.c(320): Mapping senones to individual codebooks
INFO: ms_mgau.c(144): The value of topn: 4
INFO: phone_loop_search.c(114): State beam -225 Phone exit beam -225 Insertion penalty 0
INFO: dict.c(320): Allocating 31724 * 20 bytes (619 KiB) for word entries
INFO: dict.c(333): Reading main dictionary: ./models/ger2/voxforge.dic
INFO: dict.c(213): Dictionary size 27625, allocated 261 KiB for strings, 453 KiB for phones
INFO: dict.c(336): 27625 words read
INFO: dict.c(358): Reading filler dictionary: ./models/ger2/voxforge.cd_cont_3000/noisedict
INFO: dict.c(213): Dictionary size 27628, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(361): 3 words read
INFO: dict2pid.c(396): Building PID tables for dictionary
INFO: dict2pid.c(406): Allocating 66^3 * 2 bytes (561 KiB) for word-initial triphones
INFO: dict2pid.c(132): Allocated 52536 bytes (51 KiB) for word-final triphones
INFO: dict2pid.c(196): Allocated 52536 bytes (51 KiB) for single-phone word triphones
INFO: ngram_model_trie.c(354): Trying to read LM in trie binary format
INFO: ngram_search_fwdtree.c(74): Initializing search tree
INFO: ngram_search_fwdtree.c(101): 605 unique initial diphones
INFO: ngram_search_fwdtree.c(186): Creating search channels
INFO: ngram_search_fwdtree.c(323): Max nonroot chan increased to 88448
INFO: ngram_search_fwdtree.c(333): Created 605 root, 88320 non-root channels, 3 single-phone words
INFO: ngram_search_fwdflat.c(157): fwdflat: min_ef_width = 4, max_sf_win = 25
paula --- P AW L AA no update
wäscht --- V EHH SH T no update
persil --- P ERH Z IIH L no update
überquere --- Q YYH B EX K V EEH RR AX no update
kraut --- K RR AW T no update
INFO: ngram_search_fwdtree.c(74): Initializing search tree
INFO: ngram_search_fwdtree.c(101): 605 unique initial diphones
INFO: ngram_search_fwdtree.c(186): Creating search channels
INFO: ngram_search_fwdtree.c(333): Created 605 root, 88332 non-root channels, 3 single-phone words
INFO: phone_loop_search.c(114): State beam -225 Phone exit beam -225 Insertion penalty 0
flieder --- F L IIH D EX UPDATE
INFO: cmn.c(137): CMN: 43.20  9.09  4.01  7.81 -1.89 -9.08 -10.76 -6.70 -1.88 -2.02  4.28 -4.94 -4.49

There you can see that at some point at the end I print out the added words with an information if the update is done or not. If I then decode a file in which normally the word "kraut" (the 5th word) or "flieder" (the 6th word) are recognized (e.g. if they are added as the first two new words) the program crashes after this last line of the log.

I thought it may have something to do with the language model update, because with lookup_word the words seem to be existent.

Thanks for your help!!!

Last edit: Jonas Helm 2016-09-21

I will try it. At the moment I have this problem:

GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mingw32".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from C:\Users\jmh\AppData\Local\Programs\Python\Python35-32\pyth
on.exe...(no debugging symbols found)...done.
(gdb) run
Starting program: C:\Users\jmh\AppData\Local\Programs\Python\Python35-32/python.
exe recognition_test_gdb.py
[New Thread 8488.0xca4]
[New Thread 8488.0x1850]
../../gdb-7.6.1/gdb/minsyms.c:862: internal-error: sect_index_data not initializ
ed
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) n

../../gdb-7.6.1/gdb/minsyms.c:862: internal-error: sect_index_data not initializ
ed
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n)

I will try to fix this and come back to you!

program crashes after adding more than 4 words with ps_add_word and decoding them

Speech Recognition Toolkit

Forums

Help

program crashes after adding more than 4 words with ps_add_word and decoding them

program crashes after adding more than 4 words with ps_add_word and decoding them

Speech Recognition Toolkit

Forums

Help

program crashes after adding more than 4 words with ps_add_word and decoding them document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

program crashes after adding more than 4 words with ps_add_word and decoding them