Re: [Kaldi-developers] Problems Running Kaldi Tests

[Joaquin A. R. <ja...@co...>]

Thanks!

On Mon, Dec 29, 2014 at 11:50 PM, Daniel Povey <dp...@gm...> wrote:

> OK, I just committed a fix (by removing a slash in a script) so this
> will work on a mac.
> Dan
>
>
> On Mon, Dec 29, 2014 at 8:46 PM, Joaquin Antonio Ruales
> <ja...@co...> wrote:
> > Hmm, it seems to be another Mac problem:
> >
> http://serverfault.com/questions/11518/recursive-cp-copies-folder-contents-instead-of-folder-on-os-x
> >
> > I believe that changing
> >
> > for f in phones.txt words.txt phones.txt L.fst L_disambig.fst phones/; do
> >     cp -r data/lang/$f $test
> > done
> >
> > for
> >
> > for f in phones.txt words.txt phones.txt L.fst L_disambig.fst phones; do
> >     cp -r data/lang/$f $test
> > done
> >
> > should do the trick (if "phones" has a slash after it, Mac computers copy
> > the contents of the folder rather than the folder itself, so this case
> > requires ommiting the /, for OSX).
> >
> > Thanks,
> > Joaquín
> >
> > On Mon, Dec 29, 2014 at 11:30 PM, Daniel Povey <dp...@gm...> wrote:
> >>
> >> local/voxforge_format_data.sh creates that directory.  See if you can
> >> figure out what happened.
> >> Dan
> >>
> >>
> >> On Mon, Dec 29, 2014 at 8:28 PM, Joaquin Antonio Ruales
> >> <ja...@co...> wrote:
> >> > There is a "data/lang" folder, and there is a "data/lang_test" folder.
> >> > The
> >> > "data/lang" folder does have the "phones" subfolder, but the
> >> > "data/lang_test" folder only has the "tmp" folder inside it. All the
> >> > files
> >> > that were expected to be inside "phones" are directly inside
> >> > "data/lang_test".
> >> >
> >> > On Mon, Dec 29, 2014 at 11:22 PM, Daniel Povey <dp...@gm...>
> wrote:
> >> >>
> >> >> That file should be in the phones/ subdirectory.  let me look into
> it.
> >> >> Dan
> >> >>
> >> >>
> >> >> On Mon, Dec 29, 2014 at 8:21 PM, Joaquin Antonio Ruales
> >> >> <ja...@co...> wrote:
> >> >> > Hi Daniel,
> >> >> >
> >> >> > All tests passed as expected.
> >> >> > I am now trying to run the Voxforge recipe, and after train_mono.sh
> >> >> > succeeds, I get the error:
> >> >> >
> >> >> > mkgraph.sh: expected data/lang_test/phones/silence.csl to exist
> >> >> >
> >> >> > I have checked the directories and that file is in fact missing.
> >> >> > However, I
> >> >> > have found a "silence.csl" file as well as other expected files
> >> >> > inside
> >> >> > data/lang_test/silence.csl (one level above from the expected
> >> >> > directory). I
> >> >> > have thus modified mkgraph.sh to reference the expected file with
> the
> >> >> > line:
> >> >> >
> >> >> > required="$lang/L.fst $lang/G.fst $lang/phones.txt $lang/words.txt
> >> >> > $lang/silence.csl $lang/disambig.int $model $tree"
> >> >> >
> >> >> > which now references the file one directory higher. Is this the
> >> >> > correct
> >> >> > thing to do? or am I referencing the wrong file?
> >> >> >
> >> >> > After I made this change I reran ./run.sh, and the recipe didn't
> >> >> > throw
> >> >> > the
> >> >> > error mentioned above anymore. However, I obtain more errors later
> >> >> > on:
> >> >> >
> >> >> > fstcomposecontext --context-size=1 --central-position=0
> >> >> > --read-disambig-syms=data/lang_test/phones/disambig.int
> >> >> > --write-disambig-syms=data/lang_test/tmp/disambig_ilabels_1_0.int
> >> >> > data/lang_test/tmp/ilabels_1_0
> >> >> >
> >> >> > ERROR (fstcomposecontext:main():fstcomposecontext.cc:126)
> >> >> > fstcomposecontext:
> >> >> > Could not read disambiguation symbols from
> >> >> > data/lang_test/phones/disambig.int
> >> >> >
> >> >> > ERROR (fstcomposecontext:main():fstcomposecontext.cc:126)
> >> >> > fstcomposecontext:
> >> >> > Could not read disambiguation symbols from
> >> >> > data/lang_test/phones/disambig.int
> >> >> >
> >> >> > Again, the file data/lang_test/phones/disambig.int does not exist,
> >> >> > but
> >> >> > the
> >> >> > file data/lang_test/disambig.int (one folder above) does exist.
> Is it
> >> >> > correct to change the referenced disambig.int to be the one on the
> >> >> > parent
> >> >> > folder?
> >> >> > I want to make sure that I am not making changes that don't make
> >> >> > sense
> >> >> > with
> >> >> > mkgraph.sh and disambig.int as mentioned above.
> >> >> >
> >> >> > Thanks,
> >> >> > Joaquín
> >> >> >
> >> >> > On Thu, Dec 4, 2014 at 3:05 PM, Daniel Povey <dp...@gm...>
> >> >> > wrote:
> >> >> >>
> >> >> >> I have modified that test program and it should pass now.
> >> >> >>
> >> >> >> We normally don't get so many issues discovered by one person- it
> >> >> >> only
> >> >> >> happens when we encounter a new platform that hasn't been properly
> >> >> >> tested before.
> >> >> >>
> >> >> >> Dan
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Dec 4, 2014 at 4:34 AM, Joaquin Antonio Ruales
> >> >> >> <ja...@co...> wrote:
> >> >> >> > Yep! now the matrix test works. Now I'm getting an error on the
> >> >> >> > mle-full-gmm-test. could it be another problem only relevant to
> >> >> >> > OSX
> >> >> >> > Mavericks and Yosemite? Here are the last few lines of output of
> >> >> >> > ./mle-full-gmm-test:
> >> >> >> >
> >> >> >> > Condition number of random matrix large 1.558446e+02, trying
> again
> >> >> >> > (this
> >> >> >> > is
> >> >> >> > normal)
> >> >> >> >
> >> >> >> > Avg log-like per frame [full-cov, 1-mix] should be:
> -2.535968e+01
> >> >> >> >
> >> >> >> > Total log-like [full-cov, 1-mix] should be: -3.550355e+04
> >> >> >> >
> >> >> >> > Cov eigs are  [ 1.639155e+01 1.397001e+01 1.149592e+01
> >> >> >> > 1.090052e+01
> >> >> >> > 1.049452e+01 8.523492e+00 7.486929e+00 7.382205e+00 6.812490e+00
> >> >> >> > 5.214448e+00 ]
> >> >> >> >
> >> >> >> > LOG (UnitTestEstimateFullGmm():mle-full-gmm-test.cc:361) Testing
> >> >> >> > natural<>normal conversion
> >> >> >> >
> >> >> >> > KALDI_ASSERT: at
> UnitTestEstimateFullGmm:mle-full-gmm-test.cc:374,
> >> >> >> > failed:
> >> >> >> > std::abs(gmm->means_invcovars().Row(0)(d) -
> >> >> >> > rgmm.means_invcovars().Row(0)(d)) < prec_v
> >> >> >> >
> >> >> >> > Stack trace is:
> >> >> >> >
> >> >> >> > 0   mle-full-gmm-test                   0x000000010b873bb7
> >> >> >> > _ZN5kaldi18KaldiGetStackTraceEv + 71
> >> >> >> >
> >> >> >> > 1   mle-full-gmm-test                   0x000000010b873f86
> >> >> >> > _ZN5kaldi19KaldiAssertFailure_EPKcS1_iS1_ + 230
> >> >> >> >
> >> >> >> > 2   mle-full-gmm-test                   0x000000010b78f8c6
> >> >> >> > _Z23UnitTestEstimateFullGmmv + 4774
> >> >> >> >
> >> >> >> > 3   mle-full-gmm-test                   0x000000010b790a48 main
> +
> >> >> >> > 40
> >> >> >> >
> >> >> >> > 4   libdyld.dylib                       0x000000010bb725fd
> start +
> >> >> >> > 1
> >> >> >> >
> >> >> >> > 5   ???                                 0x0000000000000001 0x0
> + 1
> >> >> >> >
> >> >> >> > Abort trap: 6
> >> >> >> >
> >> >> >> >
> >> >> >> > On Wed, Dec 3, 2014 at 8:10 PM, Dogan Can <dog...@us...>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Hi Dan,
> >> >> >> >>
> >> >> >> >> It turns out this was due to a bug in our RandGauss2 function
> >> >> >> >> that
> >> >> >> >> accepts
> >> >> >> >> double arguments. I’m committing the following fix to kaldi
> trunk
> >> >> >> >> right
> >> >> >> >> away.
> >> >> >> >>
> >> >> >> >> Cheers,
> >> >> >> >> Dogan
> >> >> >> >>
> >> >> >> >> diff --git a/src/base/kaldi-math.cc b/src/base/kaldi-math.cc
> >> >> >> >> index 5b30320..0de73bf 100644
> >> >> >> >> --- a/src/base/kaldi-math.cc
> >> >> >> >> +++ b/src/base/kaldi-math.cc
> >> >> >> >> @@ -147,7 +147,7 @@ void RandGauss2(double *a, double *b,
> >> >> >> >> RandomState
> >> >> >> >> *state)
> >> >> >> >>    float a_float, b_float;
> >> >> >> >>    // Just because we're using doubles doesn't mean we need
> >> >> >> >> super-high-quality
> >> >> >> >>    // random numbers, so we just use the floating-point version
> >> >> >> >> internally.
> >> >> >> >> -  RandGauss2(&a_float, &b_float);
> >> >> >> >> +  RandGauss2(&a_float, &b_float, state);
> >> >> >> >>    *a = a_float; *b = b_float;
> >> >> >> >>  }
> >> >> >> >>
> >> >> >> >> On Dec 3, 2014, at 2:23 AM, Dogan Can <dog...@us...>
> wrote:
> >> >> >> >>
> >> >> >> >> Hi Dan
> >> >> >> >>
> >> >> >> >> I don’t have a Mavericks setup but I can replicate the failure
> on
> >> >> >> >> Yosemite
> >> >> >> >> with clang. When everything is compiled with gcc,
> matrix-lib-test
> >> >> >> >> finishes
> >> >> >> >> successfully.
> >> >> >> >>
> >> >> >> >> I looked a bit into the root cause of why the test was going
> into
> >> >> >> >> an
> >> >> >> >> infinite loop. The random 3x3 matrices generated during the
> >> >> >> >> failing
> >> >> >> >> test end
> >> >> >> >> up having the same first and third columns and fail the
> condition
> >> >> >> >> number
> >> >> >> >> test, hence the generation loop never terminates. I did some
> >> >> >> >> sleuthing
> >> >> >> >> to
> >> >> >> >> figure out how that was happening and it turns out the value of
> >> >> >> >> rstate.seed
> >> >> >> >> (see the snippet below from matrix/kaldi-matrix.cc) is the same
> >> >> >> >> before
> >> >> >> >> and
> >> >> >> >> after the call to RandGauss2 (note that inner for loop executes
> >> >> >> >> only
> >> >> >> >> once).
> >> >> >> >> It seems like we are hitting a race condition in the system
> >> >> >> >> provided
> >> >> >> >> rand_r
> >> >> >> >> implementation or a compiler bug.
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> 1093 template<typename Real>
> >> >> >> >> 1094 void MatrixBase<Real>::SetRandn() {
> >> >> >> >> 1095   kaldi::RandomState rstate;
> >> >> >> >> 1096   for (MatrixIndexT row = 0; row < num_rows_; row++) {
> >> >> >> >> 1097     Real *row_data = this->RowData(row);
> >> >> >> >> 1098     MatrixIndexT nc = (num_cols_ % 2 == 1) ? num_cols_ -
> 1 :
> >> >> >> >> num_cols_;
> >> >> >> >> 1099     for (MatrixIndexT col = 0; col < nc; col += 2) {
> >> >> >> >> 1100       kaldi::RandGauss2(row_data + col, row_data + col +
> 1,
> >> >> >> >> &rstate);
> >> >> >> >> 1101     }
> >> >> >> >> 1102     if (nc != num_cols_) row_data[nc] =
> >> >> >> >> static_cast<Real>(kaldi::RandGauss(&rstate));
> >> >> >> >> 1103   }
> >> >> >> >> 1104 }
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Cheers,
> >> >> >> >> Dogan
> >> >> >> >>
> >> >> >> >> On Dec 2, 2014, at 10:13 PM, Daniel Povey <dp...@gm...>
> >> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >> According to
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> http://stackoverflow.com/questions/19554439/gdb-missing-in-os-x-mavericks
> >> >> >> >> gdb is no longer supported in mavericks and you have to use
> lldb.
> >> >> >> >> I've
> >> >> >> >> never used that so I don't know what to tell you to do.
> >> >> >> >> Dogan, do you have a mavericks setup, and can you see if you
> get
> >> >> >> >> the
> >> >> >> >> infinite loop that he gets when you test?
> >> >> >> >> Joaqin: I would just continue through the tutorial for now.
> >> >> >> >> Probably
> >> >> >> >> it's not going to be a problem.
> >> >> >> >> Dan
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Wed, Dec 3, 2014 at 1:09 AM, Daniel Povey <dp...@gm...
> >
> >> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >> Hm.  That makes me think there may be a problem with your
> gcc/gdb
> >> >> >> >> installation- maybe you could try uninstalling it and
> installing
> >> >> >> >> it
> >> >> >> >> again.  Did you get it from MacPorts?
> >> >> >> >> Dan
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Wed, Dec 3, 2014 at 1:05 AM, Joaquin Antonio Ruales
> >> >> >> >> <ja...@co...> wrote:
> >> >> >> >>
> >> >> >> >> Now I get:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Program received signal SIGINT, Interrupt.
> >> >> >> >>
> >> >> >> >> 0x00007fff8c755e9a in ?? ()
> >> >> >> >>
> >> >> >> >> (gdb) b matrix-lib-test.cc:39
> >> >> >> >>
> >> >> >> >> Cannot access memory at address 0xab670
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Wed, Dec 3, 2014 at 12:57 AM, Daniel Povey <
> dp...@gm...>
> >> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Probably when you did ctrl-c it was in the Atlas library,
> >> >> >> >> sometimes
> >> >> >> >> it
> >> >> >> >> can't get a proper backtrace.  Instead, after doing ctrl-c do:
> >> >> >> >> (gdb) b matrix-lib-test.cc:39
> >> >> >> >> (gdb) c
> >> >> >> >>
> >> >> >> >> to continue, and when it breaks, show me a backtrace.
> >> >> >> >> Dan
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Wed, Dec 3, 2014 at 12:48 AM, Joaquin Antonio Ruales
> >> >> >> >> <ja...@co...> wrote:
> >> >> >> >>
> >> >> >> >> This is what I get:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Program received signal SIGINT, Interrupt.
> >> >> >> >>
> >> >> >> >> 0x00007fff8b2c6342 in ?? ()
> >> >> >> >>
> >> >> >> >> (gdb) bt
> >> >> >> >>
> >> >> >> >> #0  0x00007fff8b2c6342 in ?? ()
> >> >> >> >>
> >> >> >> >> #1  0x00007fff5fbfdbc8 in ?? ()
> >> >> >> >>
> >> >> >> >> #2  0x0000000000000001 in ?? ()
> >> >> >> >>
> >> >> >> >> #3  0x00007fff5fbfdfdc in ?? ()
> >> >> >> >>
> >> >> >> >> #4  0x00007fff5fbfdfe0 in ?? ()
> >> >> >> >>
> >> >> >> >> #5  0x00007fff5fbfdbbe in ?? ()
> >> >> >> >>
> >> >> >> >> #6  0x0000000100102672 in ?? ()
> >> >> >> >>
> >> >> >> >> #7  0x00007fff5fbfdc10 in ?? ()
> >> >> >> >>
> >> >> >> >> #8  0x00007fff8b00eac4 in ?? ()
> >> >> >> >>
> >> >> >> >> #9  0x00007fff5fbfda90 in ?? ()
> >> >> >> >>
> >> >> >> >> #10 0x00007fff8e2bf272 in ?? ()
> >> >> >> >>
> >> >> >> >> #11 0x00000002001a0a00 in ?? ()
> >> >> >> >>
> >> >> >> >> #12 0x00000001001a0a00 in ?? ()
> >> >> >> >>
> >> >> >> >> #13 0x000000010019d000 in ?? ()
> >> >> >> >>
> >> >> >> >> #14 0x00000001001a0a00 in ?? ()
> >> >> >> >>
> >> >> >> >> #15 0x00000001003002ca in ?? ()
> >> >> >> >>
> >> >> >> >> #16 0x0000000000000000 in ?? ()
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Wed, Dec 3, 2014 at 12:42 AM, Daniel Povey <
> dp...@gm...>
> >> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> That isn't normal and it doesn't happen when I compile on a
> Mac.
> >> >> >> >> Can you run it in gdb by doing
> >> >> >> >> gdb ./matrix-lib-test
> >> >> >> >> (gdb) r
> >> >> >> >>
> >> >> >> >> and then do ctrl-c when it gets in the loop and type "bt" and
> >> >> >> >> show
> >> >> >> >> me
> >> >> >> >> the backtrace?
> >> >> >> >> Dan
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Wed, Dec 3, 2014 at 12:36 AM, Joaquin Antonio Ruales
> >> >> >> >> <ja...@co...> wrote:
> >> >> >> >>
> >> >> >> >> Hi Dan,
> >> >> >> >>
> >> >> >> >> Thanks for the quick reply. I have "svn up"ed and recompiled
> >> >> >> >> everything,
> >> >> >> >> but
> >> >> >> >> now the same matrix test runs into an infinite loop. Would it
> be
> >> >> >> >> safe
> >> >> >> >> to
> >> >> >> >> ignore the test results and continue with the tutorial? Here
> are
> >> >> >> >> the
> >> >> >> >> few
> >> >> >> >> lines of output from the test:
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 7.50727e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 1.91265e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 7.70738e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 2.32488e+17, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 8.44565e+17, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 1.83557e+17, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 2.61736e+17, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 8.67803e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 1.80053e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 1.67101e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 1.49485e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 8.31813e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 1.6285e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 2.48172e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition
> number
> >> >> >> >> of
> >> >> >> >> random
> >> >> >> >> matrix large 2.1569e+16, trying again (this is normal)
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Tue, Dec 2, 2014 at 4:35 PM, Daniel Povey <dp...@gm...
> >
> >> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> I think I had noticed this before, and I fixed the test by
> >> >> >> >> changing
> >> >> >> >> the threshold.
> >> >> >> >> If you do "svn up" and recompile, it should pass.
> >> >> >> >> Dan
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Tue, Dec 2, 2014 at 4:20 PM, Joaquín Ruales
> >> >> >> >> <ja...@co...>
> >> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >> Hi Kaldi Team,
> >> >> >> >>
> >> >> >> >> I'm running into trouble when running the Kaldi tests (make
> test)
> >> >> >> >> described
> >> >> >> >> in the Kaldi tutorial. I'm running it on a Mac (Mavericks) and
> >> >> >> >> the
> >> >> >> >> error
> >> >> >> >> is
> >> >> >> >> in matrix-lib-test: KALDI_ASSERT: at
> >> >> >> >> UnitTestLinearCgd:matrix-lib-test.cc:3118, failed: error <
> >> >> >> >> 1.0e-05
> >> >> >> >> *
> >> >> >> >> b.Norm(2.0)
> >> >> >> >>
> >> >> >> >> Has anyone faced this problem before or have any suggestions?
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> Joaquín
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> ------------------------------------------------------------------------------
> >> >> >> >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT
> Server
> >> >> >> >> from Actuate! Instantly Supercharge Your Business Reports and
> >> >> >> >> Dashboards
> >> >> >> >> with Interactivity, Sharing, Native Excel Exports, App
> >> >> >> >> Integration
> >> >> >> >> &
> >> >> >> >> more
> >> >> >> >> Get technology previously reserved for billion-dollar
> >> >> >> >> corporations,
> >> >> >> >> FREE
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> >> >> >> >> _______________________________________________
> >> >> >> >> Kaldi-developers mailing list
> >> >> >> >> Kal...@li...
> >> >> >> >> https://lists.sourceforge.net/lists/listinfo/kaldi-developers
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>