Re: [infomap-nlp-users] missing words

[Dominic W. <wi...@ma...>]

Hi Gabriel,

If it's a really big corpus, then 500 occurrences may not be enough for=20=

a word to get into the top 20000, which I think is the default number=20
of rows in the cooccurrence matrix.

The first thing you should try is building a model with more rows,=20
which I think you can do by changing the
"admin/default_params" file.

Try a "grep -n falklands" on the ".dic" file in your model directory if=20=

you want to get a sense of how many rows you should include before you=20=

get to the word "falklands". If it looks like the word should have been=20=

in anyway, then the problem is something else.

Hope this helps.
Best wishes,
Dominic

On Oct 20, 2005, at 8:01 PM, Gabriel Murray wrote:

> I built an Infomap model using a very large corpus of newspaper=20
> articles (100+ million words). I can use associate to query words, but=20=

> I find that some words that were contained in the corpus and were NOT=20=

> stopwords are for some reason not contained in the model, i.e. I get a=20=

> response of "no word vector for X." Is there some frequency threshold=20=

> set? For example, "falklands" doesn't appear in the model even though=20=

> it appeared more than 500 times in the corpus.
> =A0
> If there is some threshold, can I turn it off?
> Thanks,
> Gabriel Murray=