lm_combine usage weight

[Anurag Jain]

Hello everyone,
I want to combine two different language models to form a single language
model.
I found that lm_combine does this job, but how can I decide how much weight
should be given as an argument.
Is there a standard way or I have to choose arbitrarily

Regards
Anurag

[Nickolay V. Shmyrev]

Congratulations you made it

As for join weight, it's usually optimized on a development set. You select a
small domain-specific set as a development set. Then join with some wieght,
evaluate perplexity, tune weight and evaluate perplexity again until you will
find the weight which maximizes perplexity on a development set.

Other toolkits automate this process. You can try mitlm which have this
functionality implemented.

[Anurag Jain]

Hi nsh,
You said I have to choose weight which maximises perplexity. But when I
combined the language models and computed perplexity, it came out to be
infinite.
How can I fix this?
I also read somewhere that low perplexity indicates high accuracy, but you
said I have to maximise perplexity. Can you highlight me in this?

Regards
Anurag

[Nickolay V. Shmyrev]

But when I combined the language models and computed perplexity, it came out
to be infinite.

If perplexity is infinite something went wrong. It should be finite number
around 200-1000.

but you said I have to maximise perplexity. Can you highlight me in
this?[/quote

I was wrong here, of course you need to minimize it.