So, long story short, we've adapted Sphinx4 and SphinxTrain to work
recognizing Urdu handwriting, and are getting some decent performance figures
( > 90% accuracy with a relatively small vocab).
Anyway, I'm working on "speaker" adaptation at the moment, and I have MLLR
transforms working, but am not seeing any performance increase. So I'm trying
to get MAP adaptation working... and I've come across the good ol:
ERROR: "backward.c", line 431: final state not reached
and sometimes the more interesting:
ERROR: "forward.c", line 595: All 4 active states, 606 615 618 N(28,4), zero
at time 1
0 ERROR: "baum_welch.c", line 331: word1/124-1 ignored
This happens for all words, and for adaptation sets sized from 30-200 words (I
wish I had more than 200 words for each person, but as of now that's not
really realistic). For reference, the model is a cd tied with 600 seno, and
our training set consists of ~200 handwritten words x 115 people.
I don't understand how it could be an alignment issue; I'm using the same .mfc
files as I did to train my overall model, and that it trains just fine. And
I've checked the .lsn file, it looks fine to me (it IS supposed to be the same
as the transcription file for overall training, just containing only those for
the person you're adapting to right? My lines look like this:
4_41_7_41 <sil> (word1/104-1)
Where the 4_41_7_41 is a particular Urdu word, and (word1/104-1) points to
writer 104's instance of the word. </sil>
I know there's been a lot of posts about these types of errors in the past,
but I've looked through them to no avail.
I appreciate any help!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
My lines look like this:
4_41_7_41 <sil> (word1/104-1)
Where the 4_41_7_41 is a particular Urdu word, and (word1/104-1) points to
writer 104's instance of the word. </sil>
Hi
For bw specifically I remember a problems because of missing and .
It's better to add them. Otherwise it's hard to guess, it would be probably
helpful to have files to debug this issue.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ah. Well that could be helpful, that'll take two seconds to add to my
transcripts.
Unfortunately I'm not at that computer anymore and it's been turned off for
the weekend =p
I'll try it out Monday and get back to ya.
I am interested in what exactly those would mean in my context though, I'll
have to look at the whitepaper doc and the code over the weekend.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So what could it mean that I can train a CD tied model, getting about 100
skipped files (out of 11k) due to errors such as "ERROR: "backward.c", line
1024: alpha(2.459268e-01) <> sum of alphas * betas (0.000000e+00) in frame 38"
or "final state not reached" that performs decently, but when I try to perform
MAP adaptation, it skips every file on the second iteration of BW?
As far as I can tell, all it should be doing is running a couple of iterations
of BW and re-estimating the mixture weights. If the data I'm trying to adapt
to works for making the overall model, why wouldn't it work for updating the
model?
Is there some essential difference between the two processes that might cause
BW for adaptation to fail, but not BW for overall training???
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ok, just to prove I'm not going crazy... here's the top of a log from a
successful BW iteration from training, followed by the same portion of a log
file for the failed adaptation. As far as I can see it should be executing an
identical process, with the only difference being the substitution of a single
writer's lsn and ctl files for the overall transcript and file listing.
with the only difference being the substitution of a single writer's lsn and
ctl files for the overall transcript and file listing.
From your past it's not the only one. In the first case it's cd_cont model, in
the second it's sat_cont. So probably there is issue with the model, not with
the transcription format.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I thought sat_cont was just a copy of the overall model that the Sphinxtrain
scripts create; the means file etc in it are the exact same size as the
overall model.
I'm going to try increasing the adaptation set size by adding in files from
other writers to see if I can get it to converge... at least then I'll know if
it's just a question of needing more data.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've narrowed it down to the map.pl script; it's output is bad, and since that
output is used for the second iteration of BW, it causes adaptation to fail.
Now to fool around with it to see if I can get anything interesting.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I passed the arg -bayesmean => "no" to the map_adapt program, and it works
now... or at the very least, it successfully completes adaptation!
I guess for whatever reason that portion of map_adapt is broken (for me
anyways).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
So, long story short, we've adapted Sphinx4 and SphinxTrain to work
recognizing Urdu handwriting, and are getting some decent performance figures
( > 90% accuracy with a relatively small vocab).
Anyway, I'm working on "speaker" adaptation at the moment, and I have MLLR
transforms working, but am not seeing any performance increase. So I'm trying
to get MAP adaptation working... and I've come across the good ol:
ERROR: "backward.c", line 431: final state not reached
and sometimes the more interesting:
ERROR: "forward.c", line 595: All 4 active states, 606 615 618 N(28,4), zero
at time 1
0 ERROR: "baum_welch.c", line 331: word1/124-1 ignored
This happens for all words, and for adaptation sets sized from 30-200 words (I
wish I had more than 200 words for each person, but as of now that's not
really realistic). For reference, the model is a cd tied with 600 seno, and
our training set consists of ~200 handwritten words x 115 people.
I don't understand how it could be an alignment issue; I'm using the same .mfc
files as I did to train my overall model, and that it trains just fine. And
I've checked the .lsn file, it looks fine to me (it IS supposed to be the same
as the transcription file for overall training, just containing only those for
the person you're adapting to right? My lines look like this:
4_41_7_41 <sil> (word1/104-1)
Where the 4_41_7_41 is a particular Urdu word, and (word1/104-1) points to
writer 104's instance of the word. </sil>
I know there's been a lot of posts about these types of errors in the past,
but I've looked through them to no avail.
I appreciate any help!
Could it have to do with the fact that I only have like 60 frames, whereas a
speech sample has a lot more?
Hi
For bw specifically I remember a problems because of missing
and.It's better to add them. Otherwise it's hard to guess, it would be probably
helpful to have files to debug this issue.
Ah. Well that could be helpful, that'll take two seconds to add to my
transcripts.
Unfortunately I'm not at that computer anymore and it's been turned off for
the weekend =p
I'll try it out Monday and get back to ya.
I am interested in what exactly those would mean in my context though, I'll
have to look at the whitepaper doc and the code over the weekend.
I added in the
and, but it made no difference as far as I can tell.I'm going to post the log files for the attempted adaptation: BW1, MAP1, BW2
/home/jpapon/TestProject/0.4_8/bin/map_adapt \
So yeah, it ignores all the words in the second iteration still.
So what could it mean that I can train a CD tied model, getting about 100
skipped files (out of 11k) due to errors such as "ERROR: "backward.c", line
1024: alpha(2.459268e-01) <> sum of alphas * betas (0.000000e+00) in frame 38"
or "final state not reached" that performs decently, but when I try to perform
MAP adaptation, it skips every file on the second iteration of BW?
As far as I can tell, all it should be doing is running a couple of iterations
of BW and re-estimating the mixture weights. If the data I'm trying to adapt
to works for making the overall model, why wouldn't it work for updating the
model?
Is there some essential difference between the two processes that might cause
BW for adaptation to fail, but not BW for overall training???
As another note, I've made sure that the vars passed to RunTool such as abeam
and bbeam are the same for the overall training and adaptation modules.
Ok, just to prove I'm not going crazy... here's the top of a log from a
successful BW iteration from training, followed by the same portion of a log
file for the failed adaptation. As far as I can see it should be executing an
identical process, with the only difference being the substitution of a single
writer's lsn and ctl files for the overall transcript and file listing.
And the failed run's log header:
first line from Control file for main run
word1/12-1
first line from transcript file for main run
4_41_7_41(12-1)first line from Control file for adaptation
word2/79-2
first line from transcript file for adaptation
33_1_34(79-2)From your past it's not the only one. In the first case it's cd_cont model, in
the second it's sat_cont. So probably there is issue with the model, not with
the transcription format.
I thought sat_cont was just a copy of the overall model that the Sphinxtrain
scripts create; the means file etc in it are the exact same size as the
overall model.
I'm going to try increasing the adaptation set size by adding in files from
other writers to see if I can get it to converge... at least then I'll know if
it's just a question of needing more data.
I've narrowed it down to the map.pl script; it's output is bad, and since that
output is used for the second iteration of BW, it causes adaptation to fail.
Now to fool around with it to see if I can get anything interesting.
I passed the arg -bayesmean => "no" to the map_adapt program, and it works
now... or at the very least, it successfully completes adaptation!
I guess for whatever reason that portion of map_adapt is broken (for me
anyways).