#2 class-based language models

[Nicola Bertoldi]

created by Nicola Bertoldi on behalf of Fabienne Cap
(see mail to me of February 13, 2015 3:42:5)

Hi Nicola,

in the meantime I updated to Moses version3 and IRSTLM 5.80.07. Unfortunately, the problems I had remained. To tell from the tuning log file it seems as if the mapping file was not even used.

Loading LM0
In LanguageModelIRST::Load: nGramOrder = 5
Language Model Type of /mount/arbeitsdaten9/projekte/morphosynt/fritzife/SYNTHETIC_PHRASES/lm/5.08.07/config_dummy_map_arpa is 2
Language Model Type is 2
no selected field: the whole string is used
collapse is disabled
lmfilename:/mount/arbeitsdaten9/projekte/morphosynt/fritzife/SYNTHETIC_PHRASES/lm/5.08.07/de.eacl.split_apprart.BIG.lc.lm.arpa
mapfilename:/mount/arbeitsdaten9/projekte/morphosynt/fritzife/SYNTHETIC_PHRASES/lm/5.08.07/dummy_mapfile
\data\
loadtxt_ram()
1-grams: reading 1721109 entries
done level 1
2-grams: reading 19358476 entries
...done level 2
3-grams: reading 12169880 entries
..done level 3
4-grams: reading 11843353 entries
..done level 4
5-grams: reading 8234106 entries
.done level 5
done
OOV code is 1721108
OOV code is 1721108
OOV code is 1721108
Reading map /mount/arbeitsdaten9/projekte/morphosynt/fritzife/SYNTHETIC_PHRASES/lm/5.08.07/dummy_mapfile...
starting to use OOV words [´]
...done
OOV code is 0
OOV code is 0
IRST: m_unknownId=0

It seems wrong that the OOV code of the mapfile is "0". And in "starting to use OOV words [']" the square brackets contain the first word of the mapping file that seems also weird.

Attached you find the following files:

• tuning_irstlm_fabienne.sh = the script we used for tuning
• tuning_irstlm_fabienne.LOG = the log file of the tuning
• moses_dummy_arpa.ini = the moses file called for tuning
• config_dummy_map_arpa = the configuration file called from moses_dummy_arpa.ini
• dummy_mapfile = the mapfile in which each word is mapped to itself
• train_irstlm_arpa.sh = the script we used for language model training

Thanks again for any help, hints and comments!

Cheers,

Fabienne