Yet another newbie problem... Sorry for bugging you so much.
Here's the thing:
I've set all the resources for a new project and started training. BW
iteration 1 started fine and I think worked fine - the log was created in the
logdir (for 120 files the final state was not reached though).
The last thing I saw in on the screen was something like 80% 90%
The when I came back the screen was filled with these lines:
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
...
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
After a while I finally ctrl+C the process.
I've created an archive (http://ge.tt/3fRqqew) with
the ./etc and ./logdir folders. Can you help me understand what's going on?
Horia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have no idea...
I've downloaded and used these files (as recommended by one of my colleagues):
drwxr-xr-x 12 cucu geod 4096 2011-03-18 10:25 sphinx3
-rw-r--r-- 1 cucu geod 26650469 2009-11-07 05:08 sphinx3.nightly.tar.gz
drwxr-xr-x 11 cucu geod 4096 2011-03-18 10:24 sphinxbase
-rw-r--r-- 1 cucu geod 2758493 2009-11-07 05:08 sphinxbase.nightly.tar.gz
drwxr-xr-x 14 cucu geod 4096 2011-03-24 11:25 SphinxTrain
-rw-r--r-- 1 cucu geod 9362262 2009-11-07 05:08 SphinxTrain.nightly.tar.gz
How can I find out the version of the tools? (is there some "<tool> -v" that I
can run?
How did you find out the training failed with a segmentation error? Where do I
look? </tool>
Horia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No, instead you need to run norm script after first iteration from the command
line and see what arguments are attached to norm and invoke norm binary with
those arguments.
Your friend is recommended you a wrong location but you can proceed with it.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ok, I've figured it out :)
I've rerun the step to see if there are any error messages before the "Waiting
for ..." line and there was one:
cucu@argos:~/projectHoriaRoot/project/spk01-20_rec00-04$
./scripts_pl/20.ci_hmm/slave_convg.pl
MODULE: 20 Training Context Independent models
Phase 1: Cleaning up directories:
accumulator...logs...qmanager...models...
Phase 2: Flat initialize
WARNING: This step had 0 ERROR messages and 4 WARNING messages. Please check
the log file for details.
WARNING: This step had 0 ERROR messages and 4 WARNING messages. Please check
the log file for details.
Phase 3: Forward-Backward
Baum welch starting for 1 Gaussian(s), iteration: 1 (1 of 1)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
This step had 240 ERROR messages and 4 WARNING messages. Please check the log
file for details. Failed to open /home/cucu/projectHoriaRoot/resources/sphinxTrainDefault.params: Aucun fichier ou répertoire de ce type at /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/scripts_pl/20.ci_hmm/../lib/SphinxTrain/Util.pm line 583.
Current Overall Likelihood Per Frame = -12.5045710136583
Baum welch starting for 1 Gaussian(s), iteration: 2 (1 of 1)
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
...
Horia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
Yet another newbie problem... Sorry for bugging you so much.
Here's the thing:
I've set all the resources for a new project and started training. BW
iteration 1 started fine and I think worked fine - the log was created in the
logdir (for 120 files the final state was not reached though).
The last thing I saw in on the screen was something like 80% 90%
The when I came back the screen was filled with these lines:
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
...
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
After a while I finally ctrl+C the process.
I've created an archive (http://ge.tt/3fRqqew) with
the ./etc and ./logdir folders. Can you help me understand what's going on?
Horia
It seems that your "norm" binary failed with segmentation fault. Which
Sphinxtrain version are you using?
I have no idea...
I've downloaded and used these files (as recommended by one of my colleagues):
drwxr-xr-x 12 cucu geod 4096 2011-03-18 10:25 sphinx3
-rw-r--r-- 1 cucu geod 26650469 2009-11-07 05:08 sphinx3.nightly.tar.gz
drwxr-xr-x 11 cucu geod 4096 2011-03-18 10:24 sphinxbase
-rw-r--r-- 1 cucu geod 2758493 2009-11-07 05:08 sphinxbase.nightly.tar.gz
drwxr-xr-x 14 cucu geod 4096 2011-03-24 11:25 SphinxTrain
-rw-r--r-- 1 cucu geod 9362262 2009-11-07 05:08 SphinxTrain.nightly.tar.gz
How can I find out the version of the tools? (is there some "<tool> -v" that I
can run?
How did you find out the training failed with a segmentation error? Where do I
look? </tool>
Horia
Should I download and install a different version of SphinxTrain?
No, instead you need to run norm script after first iteration from the command
line and see what arguments are attached to norm and invoke norm binary with
those arguments.
Your friend is recommended you a wrong location but you can proceed with it.
Ok, I've figured it out :)
I've rerun the step to see if there are any error messages before the "Waiting
for ..." line and there was one:
cucu@argos:~/projectHoriaRoot/project/spk01-20_rec00-04$
./scripts_pl/20.ci_hmm/slave_convg.pl
MODULE: 20 Training Context Independent models
Phase 1: Cleaning up directories:
accumulator...logs...qmanager...models...
Phase 2: Flat initialize
WARNING: This step had 0 ERROR messages and 4 WARNING messages. Please check
the log file for details.
WARNING: This step had 0 ERROR messages and 4 WARNING messages. Please check
the log file for details.
Phase 3: Forward-Backward
Baum welch starting for 1 Gaussian(s), iteration: 1 (1 of 1)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
This step had 240 ERROR messages and 4 WARNING messages. Please check the log
file for details.
Failed to open /home/cucu/projectHoriaRoot/resources/sphinxTrainDefault.params: Aucun fichier ou répertoire de ce type at /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/scripts_pl/20.ci_hmm/../lib/SphinxTrain/Util.pm line 583.
Current Overall Likelihood Per Frame = -12.5045710136583
Baum welch starting for 1 Gaussian(s), iteration: 2 (1 of 1)
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
Waiting for /home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/model_parame
ters/spk01-20_rec00-04.ci_cont/mixture_weights
...
Horia