My training process failed at module 50 in the last part (BW 16 gaussains) due
to lack of HDD space:
Normalization for iteration: 8
Current Overall Likelihood Per Frame = 0.44122194097241
Convergence Ratio = 0.120813569476258
Baum welch starting for 8 Gaussian(s), iteration: 9 (1 of 1)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
This step had 1696 ERROR messages and 4 WARNING messages. Please check the log
file for details.
Normalization for iteration: 9
Current Overall Likelihood Per Frame = 0.476100284769684
Convergence Ratio = 0.0790494319489325
Baum welch starting for 8 Gaussian(s), iteration: 10 (1 of 1)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
This step had 1698 ERROR messages and 4 WARNING messages. Please check the log
file for details.
Normalization for iteration: 10
Split Gaussians, increase by 8
Current Overall Likelihood Per Frame = 0.501348271154987
Convergence Ratio = 0.0530308155507969
Baum welch starting for 16 Gaussian(s), iteration: 1 (1 of 1)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
This step had 2810 ERROR messages and 5 WARNING messages. Please check the log
file for details.
Training failed in iteration 1
Something failed: (/home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/scrip
ts_pl/50.cd_hmm_tied/slave_convg.pl)
It there any way to restart it using the already trained models (8 Gaussians)
or do I have to restart the whole module (module 50)? How do I accomplish any
of these two?
Horia
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
My training process failed at module 50 in the last part (BW 16 gaussains) due
to lack of HDD space:
Normalization for iteration: 8
Current Overall Likelihood Per Frame = 0.44122194097241
Convergence Ratio = 0.120813569476258
Baum welch starting for 8 Gaussian(s), iteration: 9 (1 of 1)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
This step had 1696 ERROR messages and 4 WARNING messages. Please check the log
file for details.
Normalization for iteration: 9
Current Overall Likelihood Per Frame = 0.476100284769684
Convergence Ratio = 0.0790494319489325
Baum welch starting for 8 Gaussian(s), iteration: 10 (1 of 1)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
This step had 1698 ERROR messages and 4 WARNING messages. Please check the log
file for details.
Normalization for iteration: 10
Split Gaussians, increase by 8
Current Overall Likelihood Per Frame = 0.501348271154987
Convergence Ratio = 0.0530308155507969
Baum welch starting for 16 Gaussian(s), iteration: 1 (1 of 1)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
This step had 2810 ERROR messages and 5 WARNING messages. Please check the log
file for details.
Training failed in iteration 1
Something failed: (/home/cucu/projectHoriaRoot/project/spk01-20_rec00-04/scrip
ts_pl/50.cd_hmm_tied/slave_convg.pl)
It there any way to restart it using the already trained models (8 Gaussians)
or do I have to restart the whole module (module 50)? How do I accomplish any
of these two?
Horia
You can use undocumented arguments to perl scripts:
The argument 8 means the number of gaussians. As usual see perl script for
details.