Re: [Kaldi-users] DNN Hybrid Training & Decoding of TIMIT example

[Daniel P. <dp...@gm...>]

Possibly you did not compile for CUDA.  The logs should say which GPU you
are using (look in the dir, for *.log).  If the configure script does not
see nvcc on the command line, it will not use CUDA.  Grep for CUDA in
kaldi.mk to see.

Dan


On Thu, Oct 23, 2014 at 10:17 PM, Xingyu Na <asr...@gm...> wrote:

> Hi, I'm new in this community.
> I am running the TIMIT example s5, all the way to DNN Hybrid Training &
> Decoding part.
> The script "steps/nnet/pretrain_dbn.sh" was called yesterday, and still
> running.
> I checked the script and found that it stuck at calling nnet-forward for
> "Renormalizing MLP input features into
> exp/dnn4_pretrain-dbn/tr_splice5-1_cmvn-g.nnet"
> The program has been running more then 24 hours.
> 'nvidia-smi' said 'nnet-forward' is still running on a Tesla K20m...
> How long does it normally take? Is there something going wrong?
> Please help.
>
> The log is posted below.
> Thank you
> Xingyu
>
>
> ============================================================================
>
>                 DNN Hybrid Training & Decoding (Karel's recipe)
>
> ============================================================================
>
> steps/nnet/make_fmllr_feats.sh --nj 10 --cmd run.pl --transform-dir
> exp/tri3/decode_test data-fmllr-tri3/test data/test exp/tri3
> data-fmllr-tri3/test/log data-fmllr-tri3/test/data
> steps/nnet/make_fmllr_feats.sh: feature type is lda_fmllr
> steps/nnet/make_fmllr_feats.sh: Done!, type lda_fmllr, data/test -->
> data-fmllr-tri3/test, using : raw-trans None, gmm exp/tri3, trans
> exp/tri3/decode_test
> steps/nnet/make_fmllr_feats.sh --nj 10 --cmd run.pl --transform-dir
> exp/tri3/decode_dev data-fmllr-tri3/dev data/dev exp/tri3
> data-fmllr-tri3/dev/log data-fmllr-tri3/dev/data
> steps/nnet/make_fmllr_feats.sh: feature type is lda_fmllr
> steps/nnet/make_fmllr_feats.sh: Done!, type lda_fmllr, data/dev -->
> data-fmllr-tri3/dev, using : raw-trans None, gmm exp/tri3, trans
> exp/tri3/decode_dev
> steps/nnet/make_fmllr_feats.sh --nj 10 --cmd run.pl --transform-dir
> exp/tri3_ali data-fmllr-tri3/train data/train exp/tri3
> data-fmllr-tri3/train/log data-fmllr-tri3/train/data
> steps/nnet/make_fmllr_feats.sh: feature type is lda_fmllr
> steps/nnet/make_fmllr_feats.sh: Done!, type lda_fmllr, data/train -->
> data-fmllr-tri3/train, using : raw-trans None, gmm exp/tri3, trans
> exp/tri3_ali
> utils/subset_data_dir_tr_cv.sh data-fmllr-tri3/train
> data-fmllr-tri3/train_tr90 data-fmllr-tri3/train_cv10
> /nobackup/s1/asr/naxingyu/exps/kaldi/egs/timit/utils/subset_data_dir.sh:
> reducing #utt from 3696 to 3320
> /nobackup/s1/asr/naxingyu/exps/kaldi/egs/timit/utils/subset_data_dir.sh:
> reducing #utt from 3696 to 376
> # steps/nnet/pretrain_dbn.sh --hid-dim 1024 --rbm-iter 20
> data-fmllr-tri3/train exp/dnn4_pretrain-dbn
> # Started at Wed Oct 22 16:11:09 CST 2014
> #
> steps/nnet/pretrain_dbn.sh --hid-dim 1024 --rbm-iter 20
> data-fmllr-tri3/train exp/dnn4_pretrain-dbn
> # INFO
> steps/nnet/pretrain_dbn.sh : Pre-training Deep Belief Network as a stack
> of RBMs
>       dir       : exp/dnn4_pretrain-dbn
>       Train-set : data-fmllr-tri3/train
>
> # PREPARING FEATURES
> Preparing train/cv lists
> 3696 exp/dnn4_pretrain-dbn/train.scp
> copy-feats scp:exp/dnn4_pretrain-dbn/train.scp_non_local
> ark,scp:/tmp/tmp.3ctodczOzO/train.ark,exp/dnn4_pretrain-dbn/train.scp
> LOG (copy-feats:main():copy-feats.cc:100) Copied 3696 feature matrices.
> apply_cmvn disabled (per speaker norm. on input features)
> Getting feature dim : copy-feats scp:exp/dnn4_pretrain-dbn/train.scp ark:-
> WARNING (feat-to-dim:Close():kaldi-io.cc:446) Pipe copy-feats
> scp:exp/dnn4_pretrain-dbn/train.scp ark:- | had nonzero return status 13
> 40
> Using splice ± 5 , step 1
> Renormalizing MLP input features into
> exp/dnn4_pretrain-dbn/tr_splice5-1_cmvn-g.nnet
> compute-cmvn-stats ark:- -
> cmvn-to-nnet - -
> nnet-concat --binary=false exp/dnn4_pretrain-dbn/tr_splice5-1.nnet -
> exp/dnn4_pretrain-dbn/tr_splice5-1_cmvn-g.nnet
> LOG (nnet-concat:main():nnet-concat.cc:53) Reading
> exp/dnn4_pretrain-dbn/tr_splice5-1.nnet
> LOG (nnet-concat:main():nnet-concat.cc:65) Concatenating -
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>