Re: [Kaldi-users] wsj/s3 problem: make_denlats_lda_etc.sh - gmm-latgen-faster - KALDI_ASSERT - tran

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

I just committed a fix to all the denlat-generating scripts so they check
that HCLG.fst is newer than the model we're using, before skipping its
creation.  Thanks for pointing out the relevant code.
I have not tested this fix but it's pretty simple:

 echo "Compiling decoding graph in $dir/dengraph"
-if [ -s $dir/dengraph/HCLG.fst ]; then
+if [ -s $dir/dengraph/HCLG.fst ] && [ $dir/dengraph/HCLG.fst -nt
$srcdir/final.mdl ]; then
    echo "Graph $dir/dengraph/HCLG.fst already exists: skipping graph
creation."

Dan

On Tue, Jun 18, 2013 at 3:03 AM, Mailing list used for User Communication
and Updates <kal...@li...> wrote:

> Thank you for your prompt and accurate replies. The script was indeed
> finding an incompatible HCLG.fst from a previous run and would not update
> it:
>
> if [ -s $dir/dengraph/HCLG.fst ]; then
>     echo Not creating denominator graph $dir/dengraph/HCLG.fst since it
> already exists.
>  else
>   scripts/mkgraph.sh $dir/lang $alidir $dir/dengraph || exit 1;
>  fi
>
> I removed the old fst and so it was rebuilt … Things are working fine now.
>
> nassos
>
> On Jun 17, 2013, at 11:26 PM, Mailing list used for User Communication and
> Updates wrote:
>
> Is not this a mismatch between feature and model dimensions?
> Petr
>
>
>
> 2013/6/17 Mailing list used for User Communication and Updates <
> kal...@li...>
>
>> It looks to me like your HCLG.fst may be out of date with respect to your
>> model-- perhaps you changed your model and the HCLG.fst did not get rebuilt?
>> Dan
>>
>>
>>
>> On Mon, Jun 17, 2013 at 1:51 PM, Mailing list used for User Communication
>> and Updates <kal...@li...> wrote:
>>
>>> Hi all,
>>>
>>> I'm trying to run the s3 recipe for WSJ and I'm running into a problem I
>>> was wondering whether you could shed any light upon:
>>>
>>> The script run.sh works fine up to steps/align_lda_mllt.sh. However,
>>> when I am running:
>>>
>>> steps/make_denlats_lda_etc.sh --num-jobs 4 --cmd "$train_cmd" \
>>>    data/train_si84 data/lang exp/tri2b_ali_si84 exp/tri2b_denlats_si84
>>>
>>> I get a KALDI_ASSERT error. I updated to the most recent version of the
>>> trunk  and retried but with no effect.
>>>
>>> More specifically, I get the following output in one of the log files
>>> (I'm running the recipe on a cluster and I'm submitting to a queue using 20
>>> jobs):
>>>
>>> =====================================
>>> >> cat wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>>
>>> Running on ro
>>> Started at Tue Jun 18 05:27:00 EEST 2013
>>> gmm-latgen-faster --beam=13.0 --lattice-beam=7.0 --acoustic-scale=0.1
>>> --max-mem=20000000 --max-active=5000
>>> --word-symbol-table=data/lang/words.txt exp/tri2b_ali_si84/final.mdl
>>> exp/tri2b_denlats_si84/dengraph/HCLG.fst 'ark:apply-cmvn --norm-vars=false
>>> --utt2spk=ark:data/train_si84/split20/24/utt2spk
>>> ark:exp/tri2b_ali_si84/24.cmvn scp:data/train_si84/split20/24/feats.scp
>>> ark:- | splice-feats ark:- ark:- | transform-feats
>>> exp/tri2b_ali_si84/final.mat ark:- ark:- |' 'ark:|gzip -c
>>> >exp/tri2b_denlats_si84/lat.24.gz'
>>> splice-feats ark:- ark:-
>>> apply-cmvn --norm-vars=false
>>> --utt2spk=ark:data/train_si84/split20/24/utt2spk
>>> ark:exp/tri2b_ali_si84/24.cmvn scp:data/train_si84/split20/24/feats.scp
>>> ark:-
>>> transform-feats exp/tri2b_ali_si84/final.mat ark:- ark:-
>>> KALDI_ASSERT: at
>>> gmm-latgen-faster:TransitionIdToPdf:hmm/transition-model.h:309, failed:
>>> static_cast<size_t>(trans_id) < id2state_.size()
>>> Stack trace is:
>>> kaldi::KaldiGetStackTrace()
>>> kaldi::KaldiAssertFailure_(char const*, char const*, int, char const*)
>>> kaldi::TransitionModel::TransitionIdToPdf(int) const
>>> kaldi::DecodableAmDiagGmmScaled::LogLikelihood(int, int)
>>> kaldi::LatticeFasterDecoder::ProcessEmitting(kaldi::DecodableInterface*,
>>> int)
>>> kaldi::LatticeFasterDecoder::Decode(kaldi::DecodableInterface*)
>>> kaldi::DecodeUtteranceLatticeFaster(kaldi::LatticeFasterDecoder&,
>>> kaldi::DecodableInterface&, fst::SymbolTable const*, std::string, double,
>>> bool, bool, kaldi::TableWriter<kaldi::BasicVectorHolder<int> >*,
>>> kaldi::TableWriter<kaldi::BasicVectorHolder<int> >*,
>>> kaldi::TableWriter<kaldi::CompactLatticeHolder>*,
>>> kaldi::TableWriter<kaldi::LatticeHolder>*, double*)
>>> gmm-latgen-faster(main+0xc3b) [0x58dad6]
>>> /lib64/libc.so.6(__libc_start_main+0xe6) [0x2ba2f7d9cc16]
>>> gmm-latgen-faster() [0x58cd11]
>>> /rmt/programs/gridengine_new/default/spool/ro/job_scripts/10778: line 6:
>>> 26822 Aborted                 (core dumped) ( gmm-latgen-faster --beam=13.0
>>> --lattice-beam=7.0 --acoustic-scale=0.1 --max-mem=20000000
>>> --max-active=5000 --word-symbol-table=data/lang/words.txt
>>> exp/tri2b_ali_si84/final.mdl exp/tri2b_denlats_si84/dengraph/HCLG.fst
>>> "ark:apply-cmvn --norm-vars=false
>>> --utt2spk=ark:data/train_si84/split20/24/utt2spk
>>> ark:exp/tri2b_ali_si84/24.cmvn scp:data/train_si84/split20/24/feats.scp
>>> ark:- | splice-feats ark:- ark:- | transform-feats
>>> exp/tri2b_ali_si84/final.mat ark:- ark:- |" "ark:|gzip -c
>>> >exp/tri2b_denlats_si84/lat.24.gz" ) 2>>
>>> /rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>> >>
>>> /rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>>
>>> =====================================
>>>
>>> I've started looking into the code in further detail but I guess
>>> debugging in this way will take a while since I have very little experience
>>> with kaldi. So, any ideas or suggestions will be greatly appreciated.
>>>
>>> Thank you,
>>> nassos
>>>
>>>
>>>
>>>
>>> PS: The decode_den.24.sh script:
>>>
>>> =====================================
>>>
>>> #!/bin/bash
>>> cd /rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3
>>> . path.sh
>>> echo Running on `hostname`
>>> >/rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>> echo Started at `date`
>>> >>/rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>>  ( gmm-latgen-faster --beam=13.0 --lattice-beam=7.0 --acoustic-scale=0.1
>>> --max-mem=20000000 --max-active=5000
>>> --word-symbol-table=data/lang/words.txt exp/tri2b_ali_si84/final.mdl
>>> exp/tri2b_denlats_si84/dengraph/HCLG.fst "ark:apply-cmvn --norm-vars=false
>>> --utt2spk=ark:data/train_si84/split20/24/utt2spk
>>> ark:exp/tri2b_ali_si84/24.cmvn scp:data/train_si84/split20/24/feats.scp
>>> ark:- | splice-feats ark:- ark:- | transform-feats
>>> exp/tri2b_ali_si84/final.mat ark:- ark:- |" "ark:|gzip -c
>>> >exp/tri2b_denlats_si84/lat.24.gz"  )
>>> 2>>/rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>> >>/rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>> ret=$?
>>> echo
>>>  >>/rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>> echo Finished at `date`
>>> >>/rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>> exit $ret
>>> ## submitted with:
>>> # qsub -S /bin/bash -sync y -j y -o
>>> /rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/decode_den.24.log
>>> -l mem_free=700M
>>>  /rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/q/
>>> decode_den.24.sh>>/rmt/work/audio_asr/kaldi/kaldi-trunk/egs/wsj/s3/exp/tri2b_denlats_si84/q/queue.log
>>> 2>&1
>>> =====================================
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> This SF.net email is sponsored by Windows:
>>>
>>> Build for Windows Store.
>>>
>>> http://p.sf.net/sfu/windows-dev2dev
>>> _______________________________________________
>>> Kaldi-users mailing list
>>> Kal...@li...
>>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> This SF.net email is sponsored by Windows:
>>
>> Build for Windows Store.
>>
>> http://p.sf.net/sfu/windows-dev2dev
>> _______________________________________________
>> Kaldi-users mailing list
>> Kal...@li...
>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>
>>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
>
> http://p.sf.net/sfu/windows-dev2dev_______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>
>

Re: [Kaldi-users] wsj/s3 problem: make_denlats_lda_etc.sh - gmm-latgen-faster - KALDI_ASSERT - tran

Re: [Kaldi-users] wsj/s3 problem: make_denlats_lda_etc.sh - gmm-latgen-faster - KALDI_ASSERT - transition-model.h