CMU Sphinx / Forums / Help: MMIE training constant is infinite problem

Hello Nicolay,

In the MODULE: 65 MMIE Training I have encountered the following error:

/home/monika/workspace/training/bin/norm \
 -mmie yes \
 -accumdir /home/monika/workspace/training/bwaccumdir/boolware_buff_2 /home/monika/workspace/training/bwaccumdir/boolware_buff_1 /home/monika/workspace/training/bwaccumdir/boolware_buff_4 /home/monika/workspace/training/bwaccumdir/boolware_buff_3 \
 -inmeanfn /home/monika/workspace/training/model_parameters/boolware.cd_cont_8000/means \
 -invarfn /home/monika/workspace/training/model_parameters/boolware.cd_cont_8000/variances \
 -constE 3.0 \
 -meanfn /home/monika/workspace/training/model_parameters/boolware.cd_cont_8000_mmie/iter_1/means \
 -fullvar no

(...)

INFO: main.c(675): No -varfn specified, will skip if any
INFO: main.c(680): read original density mean parameters from /home/monika/workspace/training/model_parameters/boolware.cd_cont_8000/means
INFO: s3gau_io.c(166): Read /home/monika/workspace/training/model_parameters/boolware.cd_cont_8000/means [8190x1x16 array]
INFO: main.c(695): read original density variance parameters from /home/monika/workspace/training/model_parameters/boolware.cd_cont_8000/variances
INFO: s3gau_io.c(166): Read /home/monika/workspace/training/model_parameters/boolware.cd_cont_8000/variances [8190x1x16 array]
INFO: main.c(710): Reading and accumulating counts from /home/monika/workspace/training/bwaccumdir/boolware_buff_2
INFO: s3gau_io.c(379): Read /home/monika/workspace/training/bwaccumdir/boolware_buff_2/numlat_gauden_counts with means with vars [8190x1x16 vector arrays]
INFO: s3gau_io.c(379): Read /home/monika/workspace/training/bwaccumdir/boolware_buff_2/denlat_gauden_counts with means with vars [8190x1x16 vector arrays]
INFO: main.c(710): Reading and accumulating counts from /home/monika/workspace/training/bwaccumdir/boolware_buff_1
INFO: s3gau_io.c(379): Read /home/monika/workspace/training/bwaccumdir/boolware_buff_1/numlat_gauden_counts with means with vars [8190x1x16 vector arrays]
INFO: s3gau_io.c(379): Read /home/monika/workspace/training/bwaccumdir/boolware_buff_1/denlat_gauden_counts with means with vars [8190x1x16 vector arrays]
INFO: main.c(710): Reading and accumulating counts from /home/monika/workspace/training/bwaccumdir/boolware_buff_4
INFO: s3gau_io.c(379): Read /home/monika/workspace/training/bwaccumdir/boolware_buff_4/numlat_gauden_counts with means with vars [8190x1x16 vector arrays]
INFO: s3gau_io.c(379): Read /home/monika/workspace/training/bwaccumdir/boolware_buff_4/denlat_gauden_counts with means with vars [8190x1x16 vector arrays]
INFO: main.c(710): Reading and accumulating counts from /home/monika/workspace/training/bwaccumdir/boolware_buff_3
INFO: s3gau_io.c(379): Read /home/monika/workspace/training/bwaccumdir/boolware_buff_3/numlat_gauden_counts with means with vars [8190x1x16 vector arrays]
INFO: s3gau_io.c(379): Read /home/monika/workspace/training/bwaccumdir/boolware_buff_3/denlat_gauden_counts with means with vars [8190x1x16 vector arrays]
INFO: s3gau_io.c(166): Read /home/monika/workspace/training/model_parameters/boolware.cd_cont_8000/means [8190x1x16 array]
INFO: main.c(778): Normalizing mean for n_mgau= 8190, n_stream= 1, n_density= 16
FATAL_ERROR: "gauden.c", line 2106: Constant D:-nan (gau:3327 feat:0 density:0) is INFINITE
Thu Sep  9 21:55:22 2010

Any ideas what could cause this?

Thanks,
Marek

Nickolay V. Shmyrev - 2010-09-10

Values go out-of-bound. That may be caused by broken LM weights in lattices as
well as by other reason. In the code in gauden.c:

/ compute constant D, which controls the convergence speed and accuracy /
d_mmi = cal_constD(in_mean_, wt_num_mean_, wt_den_mean_,
in_var_, wt_num_var_, wt_den_var_,
num_dnom_, den_dnom_, veclen, constE);
if (!finite(d_mmi))
E_FATAL("Constant D:%f (gau:%d feat:%d density:%d) is INFINITE\n", d_mmi, i,
j, k);

Try to print all the arguments to find out which looks wrong.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

the result is:

INFO: main.c(778): Normalizing mean for n_mgau= 8190, n_stream= 1, n_density= 16

i=3327 j=0 k=0
in_mean=1472968912 wt_num_mean=1441511632 wt_den_mean=1408481488
in_var=1457764560 wt_num_var=1425258704 wt_den_var=1392228560
num_dnom=29 den_dnom=24364384 vclen=24784216 constE=2147483647
FATAL_ERROR: "gauden.c", line 2109: Constant D:-nan (gau:3327 feat:0 density:0) is INFINITE

Nickolay V. Shmyrev - 2010-09-10

Your constE doesn't look right. It must be 3.0. It's specified as an argument
to bw. Are you running script with proper arguments? Probably it's because of
. in locale.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-10

As you can see norm is called with -constE 3.0 option. Could you elaborate
influence of in locale?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nickolay V. Shmyrev - 2010-09-11

Ups, it seems you printed them with %d in printf format while they are all
floats. Can you change %d to %g?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-11

Sorry I didn't check that. Now it looks like:

i=3327 j=0 k=0 in_mean=0 wt_num_mean=-nan wt_den_mean=3 in_var=0 wt_num_var=1.51278e-316 wt_den_var=1.4822e-323 num_dnom=1.64376e-320 den_dnom=4.94066e-324 vclen=6.90011e-310 constE=6.9001e-310
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nickolay V. Shmyrev - 2010-09-11

Still the output looks strange. In the previous logs on stage 50, is this
senone (3327) unseen in a training data? in_mean = 0 and in_var = 0 doesn't
look right for me.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-11

unfortunately no norm log in whole logdir contains info that senone 3327
is never observed (there are other senones unseen but not 3327)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-11

is it Ok that wt_num_mean is NaN?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-13

Just crossed my mind, maybe this senone could be treated as "not observed" in
this norm to allow finishing this training step? It should have only little
affect on training, am I right?

Marek

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nickolay V. Shmyrev - 2010-09-14

Yes, it's very easy to skip the update of this component and proceed with
others, but I'm afraid something goes wrong globally. Nans shouldn't appear
there. Probably if you could share all required files (model and accumdir) I
could run it myself and check what's going on there.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-14

what files and directories do you need? Only required to run norm in this step
or more?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nickolay V. Shmyrev - 2010-09-14

Just the files required to run the norm binary.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-15

Files you require are available at
http://www.mediafire.com/file/0g3l7v6pogry1eu/bwaccumdir1.zip
http://www.mediafire.com/file/h09ao3rdboayk3m/bwaccumdir2.zip
http://www.mediafire.com/file/5gn9sy5uu56lve4/cd_cont_8000.zip
in case of any problem please let me know

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nickolay V. Shmyrev - 2010-09-15

Hm, nans appeared on previous stage, they are accumulated in gauden_counts. I
means that bw is responsible in creating them. Can you share mmie bw log? Can
you also check if there are any nans in lattice folder which was created on
stage 62. You need to unpack them I suppose and then grep.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-15

http://www.mediafire.com/file/3clp205s5a4tbhd/65.mmie_train_logdir.zip contains bw
logs for mime. I will post results as soon as I'll manage analyze lattice.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-15

word "nan" is not present in pruned_denlat directory (the only with compressed
files) as well as in the lattice directory. Should I search for other strings
or in other files?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-21

Hello Nicolay,
I do not intend to be pushy or officious, but I would be grateful for any info
on status of this problem
thanks,
Marek

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nickolay V. Shmyrev - 2010-09-22

Hi Marek

Sorry, need some more time to check the sources.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-09-22

Hi Nicolay,
No problem, thx for info.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

marekl - 2010-10-06

Hi Nicolay,
Is there any progress on this issue?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nickolay V. Shmyrev - 2010-10-07

Sorry, still didn't have time to look.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

MMIE training constant is infinite problem

Speech Recognition Toolkit

Forums

Help

MMIE training constant is infinite problem document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

MMIE training constant is infinite problem