This project, currently in its early stages, aims to implement N-gram counting, smoothing, and interpolation. The project may eventually incorporate a number of smoothing methods, but initially it will incorporate just one, which is not a standard method. It's similar in spirit but modified, interpolated Kneser-Ney, but uses the principle that the amount you discount should always be the amount that gets given to the backoff state (which makes the traditional count-of-counts stuff and the associated formulas for obtaining smoothing constants unusuable).
This gives slightly better results than KN for standard LM building, but has larger advantages for interpolation and pruning. See <a href=freelm.sf.net> here </a> for documentation.
Categories
LinguisticsLicense
Apache License V2.0Follow FreeLM
You Might Also Like
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of FreeLM!