Menu
▾
▴
Re: [Kaldi-developers] Problems Running Kaldi Tests
|
From: Daniel P. <dp...@gm...> - 2014-12-04 20:05:49
|
I have modified that test program and it should pass now.
We normally don't get so many issues discovered by one person- it only
happens when we encounter a new platform that hasn't been properly
tested before.
Dan
On Thu, Dec 4, 2014 at 4:34 AM, Joaquin Antonio Ruales
<ja...@co...> wrote:
> Yep! now the matrix test works. Now I'm getting an error on the
> mle-full-gmm-test. could it be another problem only relevant to OSX
> Mavericks and Yosemite? Here are the last few lines of output of
> ./mle-full-gmm-test:
>
> Condition number of random matrix large 1.558446e+02, trying again (this is
> normal)
>
> Avg log-like per frame [full-cov, 1-mix] should be: -2.535968e+01
>
> Total log-like [full-cov, 1-mix] should be: -3.550355e+04
>
> Cov eigs are [ 1.639155e+01 1.397001e+01 1.149592e+01 1.090052e+01
> 1.049452e+01 8.523492e+00 7.486929e+00 7.382205e+00 6.812490e+00
> 5.214448e+00 ]
>
> LOG (UnitTestEstimateFullGmm():mle-full-gmm-test.cc:361) Testing
> natural<>normal conversion
>
> KALDI_ASSERT: at UnitTestEstimateFullGmm:mle-full-gmm-test.cc:374, failed:
> std::abs(gmm->means_invcovars().Row(0)(d) -
> rgmm.means_invcovars().Row(0)(d)) < prec_v
>
> Stack trace is:
>
> 0 mle-full-gmm-test 0x000000010b873bb7
> _ZN5kaldi18KaldiGetStackTraceEv + 71
>
> 1 mle-full-gmm-test 0x000000010b873f86
> _ZN5kaldi19KaldiAssertFailure_EPKcS1_iS1_ + 230
>
> 2 mle-full-gmm-test 0x000000010b78f8c6
> _Z23UnitTestEstimateFullGmmv + 4774
>
> 3 mle-full-gmm-test 0x000000010b790a48 main + 40
>
> 4 libdyld.dylib 0x000000010bb725fd start + 1
>
> 5 ??? 0x0000000000000001 0x0 + 1
>
> Abort trap: 6
>
>
> On Wed, Dec 3, 2014 at 8:10 PM, Dogan Can <dog...@us...> wrote:
>>
>> Hi Dan,
>>
>> It turns out this was due to a bug in our RandGauss2 function that accepts
>> double arguments. I’m committing the following fix to kaldi trunk right
>> away.
>>
>> Cheers,
>> Dogan
>>
>> diff --git a/src/base/kaldi-math.cc b/src/base/kaldi-math.cc
>> index 5b30320..0de73bf 100644
>> --- a/src/base/kaldi-math.cc
>> +++ b/src/base/kaldi-math.cc
>> @@ -147,7 +147,7 @@ void RandGauss2(double *a, double *b, RandomState
>> *state)
>> float a_float, b_float;
>> // Just because we're using doubles doesn't mean we need
>> super-high-quality
>> // random numbers, so we just use the floating-point version
>> internally.
>> - RandGauss2(&a_float, &b_float);
>> + RandGauss2(&a_float, &b_float, state);
>> *a = a_float; *b = b_float;
>> }
>>
>> On Dec 3, 2014, at 2:23 AM, Dogan Can <dog...@us...> wrote:
>>
>> Hi Dan
>>
>> I don’t have a Mavericks setup but I can replicate the failure on Yosemite
>> with clang. When everything is compiled with gcc, matrix-lib-test finishes
>> successfully.
>>
>> I looked a bit into the root cause of why the test was going into an
>> infinite loop. The random 3x3 matrices generated during the failing test end
>> up having the same first and third columns and fail the condition number
>> test, hence the generation loop never terminates. I did some sleuthing to
>> figure out how that was happening and it turns out the value of rstate.seed
>> (see the snippet below from matrix/kaldi-matrix.cc) is the same before and
>> after the call to RandGauss2 (note that inner for loop executes only once).
>> It seems like we are hitting a race condition in the system provided rand_r
>> implementation or a compiler bug.
>>
>>
>> 1093 template<typename Real>
>> 1094 void MatrixBase<Real>::SetRandn() {
>> 1095 kaldi::RandomState rstate;
>> 1096 for (MatrixIndexT row = 0; row < num_rows_; row++) {
>> 1097 Real *row_data = this->RowData(row);
>> 1098 MatrixIndexT nc = (num_cols_ % 2 == 1) ? num_cols_ - 1 :
>> num_cols_;
>> 1099 for (MatrixIndexT col = 0; col < nc; col += 2) {
>> 1100 kaldi::RandGauss2(row_data + col, row_data + col + 1, &rstate);
>> 1101 }
>> 1102 if (nc != num_cols_) row_data[nc] =
>> static_cast<Real>(kaldi::RandGauss(&rstate));
>> 1103 }
>> 1104 }
>>
>>
>> Cheers,
>> Dogan
>>
>> On Dec 2, 2014, at 10:13 PM, Daniel Povey <dp...@gm...> wrote:
>>
>> According to
>> http://stackoverflow.com/questions/19554439/gdb-missing-in-os-x-mavericks
>> gdb is no longer supported in mavericks and you have to use lldb. I've
>> never used that so I don't know what to tell you to do.
>> Dogan, do you have a mavericks setup, and can you see if you get the
>> infinite loop that he gets when you test?
>> Joaqin: I would just continue through the tutorial for now. Probably
>> it's not going to be a problem.
>> Dan
>>
>>
>> On Wed, Dec 3, 2014 at 1:09 AM, Daniel Povey <dp...@gm...> wrote:
>>
>> Hm. That makes me think there may be a problem with your gcc/gdb
>> installation- maybe you could try uninstalling it and installing it
>> again. Did you get it from MacPorts?
>> Dan
>>
>>
>> On Wed, Dec 3, 2014 at 1:05 AM, Joaquin Antonio Ruales
>> <ja...@co...> wrote:
>>
>> Now I get:
>>
>>
>> Program received signal SIGINT, Interrupt.
>>
>> 0x00007fff8c755e9a in ?? ()
>>
>> (gdb) b matrix-lib-test.cc:39
>>
>> Cannot access memory at address 0xab670
>>
>>
>> On Wed, Dec 3, 2014 at 12:57 AM, Daniel Povey <dp...@gm...> wrote:
>>
>>
>> Probably when you did ctrl-c it was in the Atlas library, sometimes it
>> can't get a proper backtrace. Instead, after doing ctrl-c do:
>> (gdb) b matrix-lib-test.cc:39
>> (gdb) c
>>
>> to continue, and when it breaks, show me a backtrace.
>> Dan
>>
>>
>> On Wed, Dec 3, 2014 at 12:48 AM, Joaquin Antonio Ruales
>> <ja...@co...> wrote:
>>
>> This is what I get:
>>
>>
>> Program received signal SIGINT, Interrupt.
>>
>> 0x00007fff8b2c6342 in ?? ()
>>
>> (gdb) bt
>>
>> #0 0x00007fff8b2c6342 in ?? ()
>>
>> #1 0x00007fff5fbfdbc8 in ?? ()
>>
>> #2 0x0000000000000001 in ?? ()
>>
>> #3 0x00007fff5fbfdfdc in ?? ()
>>
>> #4 0x00007fff5fbfdfe0 in ?? ()
>>
>> #5 0x00007fff5fbfdbbe in ?? ()
>>
>> #6 0x0000000100102672 in ?? ()
>>
>> #7 0x00007fff5fbfdc10 in ?? ()
>>
>> #8 0x00007fff8b00eac4 in ?? ()
>>
>> #9 0x00007fff5fbfda90 in ?? ()
>>
>> #10 0x00007fff8e2bf272 in ?? ()
>>
>> #11 0x00000002001a0a00 in ?? ()
>>
>> #12 0x00000001001a0a00 in ?? ()
>>
>> #13 0x000000010019d000 in ?? ()
>>
>> #14 0x00000001001a0a00 in ?? ()
>>
>> #15 0x00000001003002ca in ?? ()
>>
>> #16 0x0000000000000000 in ?? ()
>>
>>
>> On Wed, Dec 3, 2014 at 12:42 AM, Daniel Povey <dp...@gm...> wrote:
>>
>>
>> That isn't normal and it doesn't happen when I compile on a Mac.
>> Can you run it in gdb by doing
>> gdb ./matrix-lib-test
>> (gdb) r
>>
>> and then do ctrl-c when it gets in the loop and type "bt" and show me
>> the backtrace?
>> Dan
>>
>>
>> On Wed, Dec 3, 2014 at 12:36 AM, Joaquin Antonio Ruales
>> <ja...@co...> wrote:
>>
>> Hi Dan,
>>
>> Thanks for the quick reply. I have "svn up"ed and recompiled
>> everything,
>> but
>> now the same matrix test runs into an infinite loop. Would it be safe
>> to
>> ignore the test results and continue with the tutorial? Here are the
>> few
>> lines of output from the test:
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 7.50727e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 1.91265e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 7.70738e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 2.32488e+17, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 8.44565e+17, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 1.83557e+17, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 2.61736e+17, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 8.67803e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 1.80053e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 1.67101e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 1.49485e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 8.31813e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 1.6285e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 2.48172e+16, trying again (this is normal)
>>
>> LOG (RandPosdefSpMatrix():matrix-lib-test.cc:39) Condition number of
>> random
>> matrix large 2.1569e+16, trying again (this is normal)
>>
>>
>> On Tue, Dec 2, 2014 at 4:35 PM, Daniel Povey <dp...@gm...>
>> wrote:
>>
>>
>> I think I had noticed this before, and I fixed the test by changing
>> the threshold.
>> If you do "svn up" and recompile, it should pass.
>> Dan
>>
>>
>> On Tue, Dec 2, 2014 at 4:20 PM, Joaquín Ruales
>> <ja...@co...>
>> wrote:
>>
>> Hi Kaldi Team,
>>
>> I'm running into trouble when running the Kaldi tests (make test)
>> described
>> in the Kaldi tutorial. I'm running it on a Mac (Mavericks) and the
>> error
>> is
>> in matrix-lib-test: KALDI_ASSERT: at
>> UnitTestLinearCgd:matrix-lib-test.cc:3118, failed: error < 1.0e-05
>> *
>> b.Norm(2.0)
>>
>> Has anyone faced this problem before or have any suggestions?
>>
>> Thanks,
>> Joaquín
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and
>> Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration
>> &
>> more
>> Get technology previously reserved for billion-dollar
>> corporations,
>> FREE
>>
>>
>>
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Kaldi-developers mailing list
>> Kal...@li...
>> https://lists.sourceforge.net/lists/listinfo/kaldi-developers
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
|
