How do we calculate perplexity for LM which is of closed vocab_type?
I'm using CMU Tool kit to do using evallm but it says "this is a closed vocabulary model ".
any other tool or we can do it in CMU tool kit. I have created LM using CMU tool kit
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I have a train data of 4000 sentences as per the documentation test data should be of 1 to 10 proposition. So I have now 400 sentences. I calculated perplexity for test data using SRILM tool.
so here is the result:
400 sentences, 3184 words, 3148 oovs
0 zeroprobs, logprob= -452.076 ppl= 10.8861 ppl1 3.61124e+12
Does this makes any sense ? ppl with 10.8861 ??
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It is good perplexity. For large vocabulary the perplexity is about 100, for very large up to 200. For medium vocabulary domains perplexity is usually 20-50.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Just an observation: your oovs (out-of-vocbulary words) seems too large. You should check that test data has not much new words as compared to train data. Otherwise your perplexity estimate is meaningless
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
How do we calculate perplexity for LM which is of closed vocab_type?
I'm using CMU Tool kit to do using evallm but it says "this is a closed vocabulary model ".
any other tool or we can do it in CMU tool kit. I have created LM using CMU tool kit
With SRILM
Hi
the test.txt should have start and end tags for the sentences or it should be without tags?
Last edit: sumitraj 2016-08-09
It does not matter. For simplicity I recommend you to have text without tags. There is no need to add them.
Hi,
I have a train data of 4000 sentences as per the documentation test data should be of 1 to 10 proposition. So I have now 400 sentences. I calculated perplexity for test data using SRILM tool.
so here is the result:
400 sentences, 3184 words, 3148 oovs
0 zeroprobs, logprob= -452.076 ppl= 10.8861 ppl1 3.61124e+12
Does this makes any sense ? ppl with 10.8861 ??
It is good perplexity. For large vocabulary the perplexity is about 100, for very large up to 200. For medium vocabulary domains perplexity is usually 20-50.
Just an observation: your oovs (out-of-vocbulary words) seems too large. You should check that test data has not much new words as compared to train data. Otherwise your perplexity estimate is meaningless
Hi all,
I had done a bit mistake my LM was in Uppercase and test data was in smaller case. So now I have changed it to Uppercase and this is new output.
400 sentences, 3184 words, 37 oovs
0 zeroprobs, logprob= -5256.67 ppl= 30.3392 ppl1 46.8139
Last edit: sumitraj 2016-08-16
now it's much more realistic