Re: [Kaldi-users] reproducing numbers in wsj/s5

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

There seems to be something broken with the current WSJ setup. I reran
the setup today and I am getting similar results as Hao. For example,
an older setup gets 34.98% on dev93, while the system trained today
gets 47.79%. The problem is not features, since the older model with
the new features is 34.76%. The problem is also not the dict/lm since
the old model with new dict/lm and old features gets 34.98%. It is
unlikely to be the new model, since the log-likelihood numbers are in
the same ballpark. My current best guess is that some setting in the
decode/scoring scripts has changed, which is causing the problem. I am
looking into this in more detail.
-Arnab

On Tue, Jun 4, 2013 at 8:02 AM, Mailing list used for User
Communication and Updates <kal...@li...> wrote:
> Here are the log for the last update with different number of jobs. I
> don't see much of a difference.
>
> In terms of likelihood shown below, they are pretty consistent.
>
> nj=2 -109.265 -> -95.5657
> nj=4 -109.265 -> -95.5599
> nj=8 -109.265 -> -95.556
>
> Hao
>
> On Mon, Jun 3, 2013 at 5:29 PM, Mailing list used for User
> Communication and Updates <kal...@li...> wrote:
>> OK, I have an idea what might have happened.  It's possible that because you
>> are using such a small number of parallel jobs (2), there is a problem due
>> to numerical roundoff.  This should not be the case because we accumulate in
>> double, but to see whether this is a factor, could you please run the
>> training with more jobs?  E.g. 4 or 8? See if the likelihoods change.  Also,
>> show me the full log from the last update, both before and after you make
>> the change.
>> Dan
>>
>>
>>
>> On Mon, Jun 3, 2013 at 6:23 PM, Mailing list used for User Communication and
>> Updates <kal...@li...> wrote:
>>>
>>> They are almost identical.
>>>
>>> 011c0201  [
>>>   46.88603 -32.66241 -11.9018 -14.99819 -4.834829 -2.596024 -5.20842
>>> 2.553219 3.935668 10.76861 9.013732 4.780495 -3.26797
>>>   46.34384 -33.52037 -10.28797 -11.47937 -5.18444 -6.867837 -12.17628
>>> -6.932252 -7.423642 -1.106863 3.337547 -0.5728683 0.6971088
>>>   46.81014 -31.05773 -7.440363 -11.46406 -5.926405 -2.568082 -3.351704
>>> -5.60004 -8.281031 -9.46069 6.799883 6.513657 -7.97058
>>>   46.8651 -30.11357 -5.052508 -11.06986 -6.155907 -2.664125 -10.907
>>> -8.487434 -3.194182 -6.367419 6.390572 0.01187897 -0.1876552
>>>   46.92208 -33.49314 -9.207784 -12.05287 -5.494797 -4.269251 -9.91317
>>> -1.722169 -0.7680084 -2.61649 9.212611 7.324253 12.41928
>>>   46.8317 -32.19164 -8.810924 -12.31202 -3.736232 -1.335111 -8.361964
>>> -2.103669 -3.199154 -6.140819 -8.036694 -8.49004 2.942402
>>>   47.34614 -33.04692 -7.571927 -8.986547 -3.523783 1.130056 -3.151051
>>> 3.572649 3.381185 -3.117306 -2.670765 2.165989 0.3512498
>>>   46.63126 -33.57331 -11.13035 -7.506401 2.113542 3.268812 -1.814186
>>> 2.770102 -4.381135 -3.264607 -3.549407 2.463478 -0.7675163
>>>   46.31366 -33.00954 -6.964884 -6.109343 -7.121595 -4.326707 -10.80855
>>> -6.96949 6.597637 4.109591 1.001096 2.592361 6.154362
>>>
>>> Hao
>>>
>>> On Mon, Jun 3, 2013 at 5:11 PM, Mailing list used for User
>>> Communication and Updates <kal...@li...> wrote:
>>> > OK, everything there looks pretty normal.  Let's check the features.
>>> > Try
>>> > doing
>>> >
>>> >  path.sh; copy-feats scp:data/train_si284/feats.scp ark,t:- | head
>>> > copy-feats scp:data/train_si284/feats.scp ark,t:-
>>> > 011c0201  [
>>> >   46.88603 -32.66241 -11.9018 -14.99819 -4.834829 -2.596024 -5.20842
>>> > 2.553219 3.935668 10.76861 9.013732 4.780495 -3.267961
>>> >   46.34384 -33.52037 -10.28797 -11.47937 -5.184441 -6.867837 -12.17628
>>> > -6.93225 -7.423645 -1.106866 3.337547 -0.5728683 0.6970833
>>> >   46.81014 -31.05773 -7.440367 -11.46406 -5.926404 -2.568073 -3.351702
>>> > -5.60004 -8.281028 -9.460709 6.799894 6.513657 -7.970572
>>> >   46.8651 -30.11357 -5.052508 -11.06986 -6.155907 -2.664125 -10.907
>>> > -8.487434 -3.194182 -6.367419 6.390572 0.01187897 -0.1876552
>>> >   46.92208 -33.49314 -9.207782 -12.05287 -5.4948 -4.269254 -9.913167
>>> > -1.722165 -0.7680123 -2.616493 9.212611 7.324253 12.41928
>>> >   46.8317 -32.19164 -8.810924 -12.31202 -3.736232 -1.335111 -8.361964
>>> > -2.103669 -3.199154 -6.140819 -8.036694 -8.49004 2.942402
>>> >   47.34614 -33.04692 -7.571927 -8.986545 -3.523783 1.130052 -3.151051
>>> > 3.572651 3.381188 -3.117314 -2.670754 2.165966 0.3512328
>>> >   46.63126 -33.57331 -11.13035 -7.506401 2.113541 3.268804 -1.814186
>>> > 2.770107 -4.381127 -3.264618 -3.549418 2.463501 -0.7675107
>>> >   46.31366 -33.00954 -6.964884 -6.109342 -7.121596 -4.326709 -10.80855
>>> > -6.96948 6.597637 4.109591 1.001096 2.592361 6.154356
>>> >
>>> > and show me the otuput
>>> > Dan
>>> >
>>> >
>>> >
>>> > On Mon, Jun 3, 2013 at 5:59 PM, Mailing list used for User Communication
>>> > and
>>> > Updates <kal...@li...> wrote:
>>> >>
>>> >> Here is the result for
>>> >>
>>> >>     ls -l data/lang  data/lang/phones exp/mono0a exp/mono0a/log
>>> >> exp/mono0a/decode/log
>>> >>
>>> >> Hao
>>> >>
>>> >> On Mon, Jun 3, 2013 at 4:49 PM, Mailing list used for User
>>> >> Communication and Updates <kal...@li...> wrote:
>>> >> > My corresponding numbers are -108.535 ->  -95.116, and the difference
>>> >> > is
>>> >> > about the same.  [It's strange that the absolute numbers differ
>>> >> > slightly,
>>> >> > though.]
>>> >> > Could you do
>>> >> > ls -l data/lang  data/lang/phones exp/mono0a exp/mono0a/log
>>> >> > exp/mono0a/decode/log
>>> >> > and send the output?
>>> >> >
>>> >> > Dan
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Mon, Jun 3, 2013 at 5:42 PM, Mailing list used for User
>>> >> > Communication
>>> >> > and
>>> >> > Updates <kal...@li...> wrote:
>>> >> >>
>>> >> >> Copied from exp/mono0a/log/update.39.log
>>> >> >>
>>> >> >>     LOG (gmm-sum-accs:main():gmm-sum-accs.cc:60) Summed 2 stats,
>>> >> >> total
>>> >> >> count 849568, avg like/frame -95.5657
>>> >> >>
>>> >> >> Hao
>>> >> >>
>>> >> >> On Mon, Jun 3, 2013 at 4:29 PM, Mailing list used for User
>>> >> >> Communication and Updates <kal...@li...> wrote:
>>> >> >> > OK, that is quite weird.  What is the avg like/frame on the final
>>> >> >> > update.??.log?
>>> >> >> > Dan
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > On Mon, Jun 3, 2013 at 5:00 PM, Mailing list used for User
>>> >> >> > Communication
>>> >> >> > and
>>> >> >> > Updates <kal...@li...> wrote:
>>> >> >> >>
>>> >> >> >> Unfortunately those are absolute differences. Here is what I get
>>> >> >> >> for
>>> >> >> >> the
>>> >> >> >> mono0a.
>>> >> >> >>
>>> >> >> >>     %WER 40.44 [ 2282 / 5643, 118 ins, 537 del, 1627 sub ]
>>> >> >> >> exp/mono0a/decode_tgpr_eval92/wer_9
>>> >> >> >>
>>> >> >> >> And below is the reported one in RESULTS.
>>> >> >> >>
>>> >> >> >>     exp/mono0a/decode_tgpr_eval92/wer_10:%WER 25.54 [ 1441 /
>>> >> >> >> 5643,
>>> >> >> >> 149
>>> >> >> >> ins, 191 del, 1101 sub ]
>>> >> >> >>
>>> >> >> >> I will try to checkout the same version as the RESULTS file and
>>> >> >> >> run
>>> >> >> >> the experiments again.
>>> >> >> >>
>>> >> >> >> Here is the total data count from exp/mono0a/log/update.0.log.
>>> >> >> >>
>>> >> >> >>     LOG (gmm-sum-accs:main():gmm-sum-accs.cc:60) Summed 2 stats,
>>> >> >> >> total
>>> >> >> >> count 849568, avg like/frame -109.265
>>> >> >> >>
>>> >> >> >> Thanks for your help.
>>> >> >> >>
>>> >> >> >> Hao
>>> >> >> >>
>>> >> >> >> On Mon, Jun 3, 2013 at 3:43 PM, Mailing list used for User
>>> >> >> >> Communication and Updates <kal...@li...>
>>> >> >> >> wrote:
>>> >> >> >> > Assuming those are relative numbers, they're probably within
>>> >> >> >> > the
>>> >> >> >> > margin
>>> >> >> >> > of
>>> >> >> >> > error.  See if it's the same on other test sets.  Because
>>> >> >> >> > various
>>> >> >> >> > algorithms
>>> >> >> >> > call rand(), and different machines implement this differently,
>>> >> >> >> > results
>>> >> >> >> > aren't fully reproducible.
>>> >> >> >> >
>>> >> >> >> > But to check that it's not some code change that hurts the
>>> >> >> >> > results,
>>> >> >> >> > you
>>> >> >> >> > could try checking out a copy with the same revision number as
>>> >> >> >> > the
>>> >> >> >> > RESULTS
>>> >> >> >> > file, and running again and seeing what results you get.  If
>>> >> >> >> > there
>>> >> >> >> > is
>>> >> >> >> > a
>>> >> >> >> > difference, I'd like to know.
>>> >> >> >> >
>>> >> >> >> > It's also possible some files were missing in your WSJ
>>> >> >> >> > distribution--
>>> >> >> >> > let me
>>> >> >> >> > know the data count reported in one the */log/update.log files
>>> >> >> >> > and
>>> >> >> >> > I'll
>>> >> >> >> > compare with a local copy.
>>> >> >> >> >
>>> >> >> >> > Dan
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > On Mon, Jun 3, 2013 at 4:34 PM, Mailing list used for User
>>> >> >> >> > Communication
>>> >> >> >> > and
>>> >> >> >> > Updates <kal...@li...> wrote:
>>> >> >> >> >>
>>> >> >> >> >> Hi,
>>> >> >> >> >>
>>> >> >> >> >> I'm following the WSJ s5 recipe, and I wasn't able to
>>> >> >> >> >> reproduce
>>> >> >> >> >> the
>>> >> >> >> >> results stated in the RESULTS file. Are we supposed to get
>>> >> >> >> >> exactly
>>> >> >> >> >> the
>>> >> >> >> >> same numbers with the (hyper-)parameters given in the recipe?
>>> >> >> >> >>
>>> >> >> >> >> Just to give a sense of how far I'm off. My monophone models
>>> >> >> >> >> perform
>>> >> >> >> >> 10%
>>> >> >> >> >> worse than the reported results on eval92 and the resulting
>>> >> >> >> >> triphone
>>> >> >> >> >> models perform 4% worse.
>>> >> >> >> >>
>>> >> >> >> >> Thanks,
>>> >> >> >> >> Hao
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >>
>>> >> >> >> >> ------------------------------------------------------------------------------
>>> >> >> >> >> How ServiceNow helps IT people transform IT departments:
>>> >> >> >> >> 1. A cloud service to automate IT design, transition and
>>> >> >> >> >> operations
>>> >> >> >> >> 2. Dashboards that offer high-level views of enterprise
>>> >> >> >> >> services
>>> >> >> >> >> 3. A single system of record for all IT processes
>>> >> >> >> >> http://p.sf.net/sfu/servicenow-d2d-j
>>> >> >> >> >> _______________________________________________
>>> >> >> >> >> Kaldi-users mailing list
>>> >> >> >> >> Kal...@li...
>>> >> >> >> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > ------------------------------------------------------------------------------
>>> >> >> >> > How ServiceNow helps IT people transform IT departments:
>>> >> >> >> > 1. A cloud service to automate IT design, transition and
>>> >> >> >> > operations
>>> >> >> >> > 2. Dashboards that offer high-level views of enterprise
>>> >> >> >> > services
>>> >> >> >> > 3. A single system of record for all IT processes
>>> >> >> >> > http://p.sf.net/sfu/servicenow-d2d-j
>>> >> >> >> > _______________________________________________
>>> >> >> >> > Kaldi-users mailing list
>>> >> >> >> > Kal...@li...
>>> >> >> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>> >> >> >> >
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> ------------------------------------------------------------------------------
>>> >> >> >> How ServiceNow helps IT people transform IT departments:
>>> >> >> >> 1. A cloud service to automate IT design, transition and
>>> >> >> >> operations
>>> >> >> >> 2. Dashboards that offer high-level views of enterprise services
>>> >> >> >> 3. A single system of record for all IT processes
>>> >> >> >> http://p.sf.net/sfu/servicenow-d2d-j
>>> >> >> >> _______________________________________________
>>> >> >> >> Kaldi-users mailing list
>>> >> >> >> Kal...@li...
>>> >> >> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > ------------------------------------------------------------------------------
>>> >> >> > How ServiceNow helps IT people transform IT departments:
>>> >> >> > 1. A cloud service to automate IT design, transition and
>>> >> >> > operations
>>> >> >> > 2. Dashboards that offer high-level views of enterprise services
>>> >> >> > 3. A single system of record for all IT processes
>>> >> >> > http://p.sf.net/sfu/servicenow-d2d-j
>>> >> >> > _______________________________________________
>>> >> >> > Kaldi-users mailing list
>>> >> >> > Kal...@li...
>>> >> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>> >> >> >
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> ------------------------------------------------------------------------------
>>> >> >> How ServiceNow helps IT people transform IT departments:
>>> >> >> 1. A cloud service to automate IT design, transition and operations
>>> >> >> 2. Dashboards that offer high-level views of enterprise services
>>> >> >> 3. A single system of record for all IT processes
>>> >> >> http://p.sf.net/sfu/servicenow-d2d-j
>>> >> >> _______________________________________________
>>> >> >> Kaldi-users mailing list
>>> >> >> Kal...@li...
>>> >> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > ------------------------------------------------------------------------------
>>> >> > How ServiceNow helps IT people transform IT departments:
>>> >> > 1. A cloud service to automate IT design, transition and operations
>>> >> > 2. Dashboards that offer high-level views of enterprise services
>>> >> > 3. A single system of record for all IT processes
>>> >> > http://p.sf.net/sfu/servicenow-d2d-j
>>> >> > _______________________________________________
>>> >> > Kaldi-users mailing list
>>> >> > Kal...@li...
>>> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> ------------------------------------------------------------------------------
>>> >> How ServiceNow helps IT people transform IT departments:
>>> >> 1. A cloud service to automate IT design, transition and operations
>>> >> 2. Dashboards that offer high-level views of enterprise services
>>> >> 3. A single system of record for all IT processes
>>> >> http://p.sf.net/sfu/servicenow-d2d-j
>>> >> _______________________________________________
>>> >> Kaldi-users mailing list
>>> >> Kal...@li...
>>> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>> >>
>>> >
>>> >
>>> >
>>> > ------------------------------------------------------------------------------
>>> > How ServiceNow helps IT people transform IT departments:
>>> > 1. A cloud service to automate IT design, transition and operations
>>> > 2. Dashboards that offer high-level views of enterprise services
>>> > 3. A single system of record for all IT processes
>>> > http://p.sf.net/sfu/servicenow-d2d-j
>>> > _______________________________________________
>>> > Kaldi-users mailing list
>>> > Kal...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>> >
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> How ServiceNow helps IT people transform IT departments:
>>> 1. A cloud service to automate IT design, transition and operations
>>> 2. Dashboards that offer high-level views of enterprise services
>>> 3. A single system of record for all IT processes
>>> http://p.sf.net/sfu/servicenow-d2d-j
>>> _______________________________________________
>>> Kaldi-users mailing list
>>> Kal...@li...
>>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>
>>
>>
>> ------------------------------------------------------------------------------
>> How ServiceNow helps IT people transform IT departments:
>> 1. A cloud service to automate IT design, transition and operations
>> 2. Dashboards that offer high-level views of enterprise services
>> 3. A single system of record for all IT processes
>> http://p.sf.net/sfu/servicenow-d2d-j
>> _______________________________________________
>> Kaldi-users mailing list
>> Kal...@li...
>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. A cloud service to automate IT design, transition and operations
> 2. Dashboards that offer high-level views of enterprise services
> 3. A single system of record for all IT processes
> http://p.sf.net/sfu/servicenow-d2d-j
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>