[Kaldi-users] discriminative LSTM training

[Gupta V. <vis...@cr...>]

Hi,


When I try to do discriminative LSTM training I get the following error:


If I use train_mpe.sh, it runs for a few thousand utterances and then I get the following error:


ERROR (nnet-train-mpe-sequential:LatticeForwardBackwardMpeVariants():lattice-functions.cc:833) Total forward score over lattice = -nan, while total backward score = 0
and then the program crashes.


If I use train_mmi.sh then after few thousand utterances I get logs with "nan":


 VLOG[1] (nnet-train-mmi-sequential:main():nnet-train-mmi-sequential.cc:346) Utterance 20080401_170000_bbcone_bbc_news_spk-0025_seg-0150897:0151494: Average MMI obj. value = nan over 595 frames. (Avg. den-posterior on ali -nan)

However, the program keeps on running.
Is there a workaround for that?


Thanks,

Vishwa