regarding changes from 0.6.1 to 0.7

[Pankaj]

Hello Nicolay

Due to some peculiar requirements my application code is very much dependent
on pocketsphinx internals.
Hence I was comparing the code of 0.6.1 and 0.7 release of pocketsphinx. I
have noticed some code changes
which are relevant to my application. It would be of great help to me if I can
know the reasoning behind
some of those changes.

1. In the new release wherever logs2prob is used it is shifted right by SENSCR_SHIFT. Similarly this shift is also happening wherever the result of logmath_log() function is used. In the repository there is a comment related to it
   "Remove superfluous shifting from HMM evaluation. 16-bit path scores will now
   work with appropriate renormalization (a bug with that exists mush fix it
   first)", but I could not figure out what bug it is referring to.

Also while computing ps_latlink_prob, out_ascr is shifted left by SENSCR_SHIFT
in 0.7. Any reason for this change.

1. In the fsg_model the null and the non null transition lists are now implemented using a hash table. Is it done with the intention of speeding up the computation or for memory optimization or for an increase in recognition accuracy.

2. In sphinxbase in feat.c feat_cmn() and feat_agc() are now being performed before padding of incoming features. I guess this fixes some corner cases.

Please correct me if I am wrong.

If possible could you point to some other code change specifically related to
improvement in accuracy? Is some readme kind of thing available indicating the
reasoning behind code changes from 0.6.1 to 0.7.

Regards
Pankaj

[Nickolay V. Shmyrev]

Hello

1. In the new release wherever logs2prob is used it is shifted right by
   SENSCR_SHIFT. Similarly this shift is also happening wherever the result of
   logmath_log() function is used. In the repository there is a comment related
   to it
   "Remove superfluous shifting from HMM evaluation. 16-bit path scores will now
   work with appropriate renormalization (a bug with that exists mush fix it
   first)", but I could not figure out what bug it is referring to.

I suppose it's related to

r10669 | dhdfu | 2010-11-25 20:31:45 +0300 (Чтв, 25 Ноя 2010) | 3 lines
Correctly unscale acoustic scores when outputting lattices, so that
posteriors computed externally actually make sense

But I'm not sure

Also while computing ps_latlink_prob, out_ascr is shifted left by
SENSCR_SHIFT in 0.7. Any reason for this change.

To be compatible with previous releases scores are shifted back so they are
valid probabilities in logbase. Internally all scores are shifted to 16-bit
(very helpful to save memory) but when we return them out we need to shirt
them back.

1. In the fsg_model the null and the non null transition lists are now
   implemented using a hash table. Is it done with the intention of speeding up
   the computation or for memory optimization or for an increase in recognition
   accuracy.

It's a huge speedup for large grammars. It has no relation on accuracy I
suppose but of course testing of this would be appreciated.

1. In sphinxbase in feat.c feat_cmn() and feat_agc() are now being performed
   before padding of incoming features. I guess this fixes some corner cases.

Yes, I had to change this. I needed that after CMN the mean should be exactly
zero. With previous approach it was different. Now its more or less inline
with what happens during the training.

If possible could you point to some other code change specifically related
to improvement in accuracy? Is some readme kind of thing available indicating
the reasoning behind code changes from 0.6.1 to 0.7.

See NEWS file. There was no thing to improve accuracy specifically, most of
changes were speed and memory optimizations. Since they are traded for
accuracy you can count them as accuracy imporvemnets as well

[Pankaj]

Hello Nicole,

Thanks for the reply. I have some more questions.

Nicole wrote
Since they are traded for accuracy you can count them as accuracy
imporvemnets as well.
I suppose that you meant change in accuracy and not literally improvements.
By shifting the logs2prob to right we are reducing the resolution and hence
some decline in accuracy may be expected ???.
Feature computation now being more identical to sphinxtrain , some improvement
in accuracy can be expected ???

In future I wish to implement n-best for FSG model. Can it be implemented on
top of the code of 0.6.1 or is there any dependency on 0.7 code. I would
appreciate your opinion on this.

The NEWS file in pocketsphinx package mentions the following bugfix
" Fixes very important regression with NULL transitions in fsg search". I am
unable to figure out what exactly it is referring to.

With regards
Pankaj

[Nickolay V. Shmyrev]

I suppose that you meant change in accuracy and not literally improvements.
By shifting the logs2prob to right we are reducing the resolution and hence
some decline in accuracy may be expected ???.

There is no degradation comparing to previous version because previously it
was also shifted. There is a small degradation compared to Sphinx4 for
example.

Feature computation now being more identical to sphinxtrain , some
improvement in accuracy can be expected ???

I don't think it's significant

In future I wish to implement n-best for FSG model. Can it be implemented on
top of the code of 0.6.1 or is there any dependency on 0.7 code. I would
appreciate your opinion on this.

If you want to maintain those changes it's better to keep them up-to-date with
the latest changes from the early beginning

" Fixes very important regression with NULL transitions in fsg search". I am
unable to figure out what exactly it is referring to.

Change in revision 10641