Re: [Kaldi-users] Memory requirement for FSTs

[Michael R. <ri...@go...>]

I don't know much about how ngram FSTs are used in Kaldi or the
characteristics of their implementations, but I do know that for sheer
OpenFst representation:

  VectorFst and ConstFst (the standard mutable and immutable OpenFst reps)
use about 20 bytes per state and 16 bytes per arc while NGramFst, specific
to ngram models (see
here<http://www.openfst.org/twiki/bin/view/FST/FstExtensions>),
use about 8 bytes per state and 8 bytes per arc. The last, for representing
at the word level, is the most compact and is used, in general, in a
rescoring or on-the-fly composition mode.  There are also various compact
FST formats (see here<http://www.openfst.org/twiki/bin/view/FST/FstExtensions>)
that can represent other specific FSTs (and is extensible).

  With 64 bit compilation, you are limited by how much memory you have for
those reps and of course. Hope that helps.

On Fri, Oct 18, 2013 at 12:06 AM, E <oth...@ao...> wrote:

> Thanks a lot for the answers and possible solutions.
>
>  Few questions-
>
>  What is the maximum size trigram language model supported by FST? I
> tried to use Gigaword LM (64k vocab), mkgraph.sh ran for a long time but
> crashed afterwords. So I want to know if there is a theoretical limit on
> size of language model that can be integrated with Kaldi.
>
>  I will try to make HCLG.fst with gigaword again (with triphone AM), but
> has anyone tried to build it with LM of this size successfully, if so, what
> were the system requirements (RAM) and final FST size in mega/gigabytes?
>
>
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
> from
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>
>