Re: [Kaldi-users] matrix-lib-test test fail with MKL 11.0 update5

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

I previously spoke to Michael Deisher at Intel about this, he says he
has asked the relevant team.  So I think they are aware of the
problem.
Dan

On Mon, Jul 29, 2013 at 12:06 PM, Mailing list used for User
Communication and Updates <kal...@li...> wrote:
> It has been reported on intel dev forum.
> http://software.intel.com/en-us/forums/topic/402436
> http://software.intel.com/en-us/forums/topic/402473
>
> The problem is caused by  the incorrect query result of workspace size given
> by dgesvd().
>
> 11.0.5 seems to be a very buggy version.
>
>
>
> On Mon, Jul 29, 2013 at 11:39 PM, Mailing list used for User Communication
> and Updates <kal...@li...> wrote:
>>
>> I just checked different versions of MKL and indeed, this was introduced
>> in the  update 5. Anyone with the commercial MKL support cares to submit a
>> ticket to Intel?
>> y.
>>
>>
>> On Wed, Jul 24, 2013 at 12:42 PM, Mailing list used for User Communication
>> and Updates <kal...@li...> wrote:
>>>
>>> You are right.... I just downgraded to MKL 11.0update1 and passed the
>>> test.
>>>
>>>
>>>
>>> On Thu, Jul 25, 2013 at 12:28 AM, Mailing list used for User
>>> Communication and Updates <kal...@li...> wrote:
>>>>
>>>> I saw some changes regarding improving performance of *GESVD in the MKL
>>>> 11.0 update5  changelog. I wonder if they messed something up? How about
>>>> trying slightly "older" version of the MKL? If you are on the subscription
>>>> of Intel Software Network, you can download older versions of the MKL at no
>>>> charge...
>>>>
>>>> We are using Intel 11.0 and the tests pass without any problem...
>>>>
>>>> yenda
>>>>
>>>>
>>>>
>>>> On Wed, Jul 24, 2013 at 12:14 PM, Mailing list used for User
>>>> Communication and Updates <kal...@li...> wrote:
>>>>>
>>>>> I can't see any problems in the Kaldi code.  Perhaps this is an older
>>>>> version of MKL?  It could be a bug in MKL itself.  MKL tests OK for
>>>>> me.
>>>>> Dan
>>>>>
>>>>>
>>>>> On Wed, Jul 24, 2013 at 11:55 AM, Mailing list used for User
>>>>> Communication and Updates <kal...@li...> wrote:
>>>>> > here's the dbg info copied from eclipse.
>>>>> > PS: I can pass the test with atlas provided by centos repo
>>>>> > atlas-sse3.x86_64                          3.8.4-2.el6
>>>>> > @base
>>>>> > atlas-sse3-devel.x86_64                    3.8.4-2.el6
>>>>> > @base
>>>>> >
>>>>> >
>>>>> >
>>>>> > stack:
>>>>> >
>>>>> > matrix-lib-test [C/C++ Application]
>>>>> > matrix-lib-test [11312] [cores: 1]
>>>>> > Thread [1] 11312 [core: 1] (Suspended : Signal : SIGABRT:Aborted)
>>>>> > raise() at 0x33090328a5
>>>>> > abort() at 0x3309034085
>>>>> > kaldi::KaldiAssertFailure_() at kaldi-error.cc:136 0x4cf401
>>>>> > kaldi::MatrixBase<double>::LapackGesvd() at kaldi-matrix.cc:401
>>>>> > 0x48fdd3
>>>>> > kaldi::MatrixBase<double>::DestructiveSvd() at kaldi-matrix.cc:1,330
>>>>> > 0x48b902
>>>>> > kaldi::MatrixBase<double>::Svd() at kaldi-matrix.cc:1,352 0x48b9bb
>>>>> > kaldi::MatrixBase<double>::Svd() at kaldi-matrix.h:315 0x468e50
>>>>> > kaldi::MatrixBase<double>::Cond() at kaldi-matrix.cc:1,222 0x48bf47
>>>>> > kaldi::InitRand<double>() at matrix-lib-test.cc:74 0x42dd7e
>>>>> > kaldi::UnitTestCompressedMatrix<double>() at matrix-lib-test.cc:3,532
>>>>> > 0x43055a
>>>>> > kaldi::MatrixUnitTest<double>() at matrix-lib-test.cc:3,705 0x42de8b
>>>>> > main() at matrix-lib-test.cc:3,829 0x42dac7
>>>>> >
>>>>> >
>>>>> >
>>>>> > variable list after invoking the following    @line393
>>>>> > kaldi-matrix.cc
>>>>> >   clapack_Xgesvd(v_job, u_job,
>>>>> >                  &M, &N, data_, &LDA,
>>>>> >                  s->Data(),
>>>>> >                  V->Data(), &V_stride,
>>>>> >                  U->Data(), &U_stride,
>>>>> >                  p_work, &l_work,
>>>>> >                  &result);
>>>>> >
>>>>> >
>>>>> > __func__ char [12] 0x4d5418
>>>>> > M KaldiBlasInt 1
>>>>> > LDA KaldiBlasInt 2
>>>>> > U kaldi::MatrixBase<double> * 0x7ffffffface0
>>>>> > data_ double * 0x7413b0
>>>>> > *data_ double 0
>>>>> > num_cols_ kaldi::MatrixIndexT 1
>>>>> > num_rows_ kaldi::MatrixIndexT 10
>>>>> > stride_ kaldi::MatrixIndexT 2
>>>>> > l_work KaldiBlasInt 5
>>>>> > p_work double * 0x741460
>>>>> > *p_work double 1.0829823019173015e-312
>>>>> > temp void * 0x741460
>>>>> > N KaldiBlasInt 10
>>>>> > V kaldi::MatrixBase<double> * 0x7fffffffacc0
>>>>> > data_ double * 0x7406a0
>>>>> > *data_ double 0
>>>>> > num_cols_ kaldi::MatrixIndexT 1
>>>>> > num_rows_ kaldi::MatrixIndexT 1
>>>>> > stride_ kaldi::MatrixIndexT 2
>>>>> > result KaldiBlasInt -13
>>>>> > tmpU kaldi::Matrix<double> {...}
>>>>> > V_stride KaldiBlasInt 2
>>>>> > work_query double 5
>>>>> > u_job char * 0x4d4fea "N"
>>>>> > tmpV kaldi::Matrix<double> {...}
>>>>> > U_stride KaldiBlasInt 2
>>>>> > v_job char * 0x4d4fea "N"
>>>>> > this kaldi::MatrixBase<double> * const 0x7fffffffaf80
>>>>> > data_ double * 0x741300
>>>>> > *data_ double 0.78239572048187256
>>>>> > num_cols_ kaldi::MatrixIndexT 1
>>>>> > num_rows_ kaldi::MatrixIndexT 10
>>>>> > stride_ kaldi::MatrixIndexT 2
>>>>> > s kaldi::VectorBase<double> * 0x7fffffffb190
>>>>> > data_ double * 0x7404a0
>>>>> > *data_ double 0
>>>>> > dim_ kaldi::MatrixIndexT 1
>>>>> > U_in kaldi::MatrixBase<double> * 0x0
>>>>> > V_in kaldi::MatrixBase<double> * 0x0
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Jul 24, 2013 at 11:04 PM, Mailing list used for User
>>>>> > Communication
>>>>> > and Updates <kal...@li...> wrote:
>>>>> >>
>>>>> >> Could you please run this in a debugger and show me the values of
>>>>> >> p_work, l_work and result, e.g.
>>>>> >>
>>>>> >> gdb matrix-lib
>>>>> >> (gdb) r
>>>>> >> [wait till it crashes]
>>>>> >> (gdb) up
>>>>> >> (gdb) up    # not sure how many times
>>>>> >> (gdb) up
>>>>> >> (gdb) p l_work
>>>>> >> (gdb) p p_work
>>>>> >> (gdb) p result
>>>>> >>
>>>>> >> Dan
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Jul 24, 2013 at 10:34 AM, Mailing list used for User
>>>>> >> Communication and Updates <kal...@li...> wrote:
>>>>> >> > Hi,
>>>>> >> >
>>>>> >> > I tried latest trunk version with MKL. I have no problem when
>>>>> >> > running
>>>>> >> > `make
>>>>> >> > all`, but get a FAIL when testing. Can anyone help? Thanks.
>>>>> >> >
>>>>> >> >
>>>>> >> > svn revision 2720 trunk
>>>>> >> >
>>>>> >> >
>>>>> >> > configuration line:
>>>>> >> > $ ./configure --threaded-math=yes --mathlib=MKL --use-cuda=yes
>>>>> >> > --mkl-root=$MKLROOT --omp-libdir=/opt/intel/lib/intel64
>>>>> >> >
>>>>> >> >
>>>>> >> > MKL version 11.0 update 5 (latest)
>>>>> >> >
>>>>> >> >
>>>>> >> > Linux version CentOS 6.4
>>>>> >> > $ uname -a
>>>>> >> > Linux 2.6.32-358.6.2.el6.x86_64 #1 SMP Thu May 16 20:59:36 UTC
>>>>> >> > 2013
>>>>> >> > x86_64
>>>>> >> > x86_64 x86_64 GNU/Linux
>>>>> >> >
>>>>> >> >
>>>>> >> > $ gcc -v
>>>>> >> > Using built-in specs.
>>>>> >> > Target: x86_64-redhat-linux
>>>>> >> > Configured with: ../configure --prefix=/usr
>>>>> >> > --mandir=/usr/share/man
>>>>> >> > --infodir=/usr/share/info
>>>>> >> > --with-bugurl=http://bugzilla.redhat.com/bugzilla
>>>>> >> > --enable-bootstrap --enable-shared --enable-threads=posix
>>>>> >> > --enable-checking=release --with-system-zlib --enable-__cxa_atexit
>>>>> >> > --disable-libunwind-exceptions --enable-gnu-unique-object
>>>>> >> > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
>>>>> >> > --enable-java-awt=gtk
>>>>> >> > --disable-dssi
>>>>> >> > --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
>>>>> >> > --enable-libgcj-multifile --enable-java-maintainer-mode
>>>>> >> > --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
>>>>> >> > --disable-libjava-multilib
>>>>> >> > --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686
>>>>> >> > --build=x86_64-redhat-linux
>>>>> >> > Thread model: posix
>>>>> >> > gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC)
>>>>> >> >
>>>>> >> >
>>>>> >> > ~/kaldi/trunk/src/matrix
>>>>> >> > $ make test
>>>>> >> > Running matrix-lib-test .../bin/sh: line 1:  5561 Aborted
>>>>> >> > (core dumped) ./$x > /dev/null 2>&1
>>>>> >> > ... FAIL
>>>>> >> > Running kaldi-gpsr-test ...... SUCCESS
>>>>> >> > make: *** [test] Error 1
>>>>> >> >
>>>>> >> >
>>>>> >> > $ ./matrix-lib-test
>>>>> >> > ...
>>>>> >> > ...
>>>>> >> > VLOG[2] (UnitTestLbfgs():matrix-lib-test.cc:2734) L-BFGS finished
>>>>> >> > after
>>>>> >> > 59
>>>>> >> > function evaluations.
>>>>> >> > LOG (UnitTestCompressedMatrix():matrix-lib-test.cc:3623) M =  [ ]
>>>>> >> >
>>>>> >> > LOG (UnitTestCompressedMatrix():matrix-lib-test.cc:3624) M2 =  [ ]
>>>>> >> >
>>>>> >> > LOG (UnitTestCompressedMatrix():matrix-lib-test.cc:3626)
>>>>> >> > Compressed
>>>>> >> > matrix,
>>>>> >> > tot = 0, diff = 0
>>>>> >> >
>>>>> >> > MKL ERROR: Parameter 13 was incorrect on entry to DGESVD.
>>>>> >> > KALDI_ASSERT: at LapackGesvd:kaldi-matrix.cc:401, failed: result
>>>>> >> > >= 0 &&
>>>>> >> > "Call to CLAPACK dgesvd_ called with wrong arguments"
>>>>> >> > Stack trace is:
>>>>> >> > kaldi::KaldiGetStackTrace()
>>>>> >> > kaldi::KaldiAssertFailure_(char const*, char const*, int, char
>>>>> >> > const*)
>>>>> >> > kaldi::MatrixBase<double>::LapackGesvd(kaldi::VectorBase<double>*,
>>>>> >> > kaldi::MatrixBase<double>*, kaldi::MatrixBase<double>*)
>>>>> >> >
>>>>> >> > kaldi::MatrixBase<double>::DestructiveSvd(kaldi::VectorBase<double>*,
>>>>> >> > kaldi::MatrixBase<double>*, kaldi::MatrixBase<double>*)
>>>>> >> > kaldi::MatrixBase<double>::Svd(kaldi::VectorBase<double>*,
>>>>> >> > kaldi::MatrixBase<double>*, kaldi::MatrixBase<double>*) const
>>>>> >> > .
>>>>> >> > .
>>>>> >> > .
>>>>> >> > ./matrix-lib-test() [0x43055a]
>>>>> >> > ./matrix-lib-test() [0x42de8b]
>>>>> >> > ./matrix-lib-test(main+0x1d) [0x42dac7]
>>>>> >> > /lib64/libc.so.6(__libc_start_main+0xfd) [0x330901ecdd]
>>>>> >> > ./matrix-lib-test() [0x42ced9]
>>>>> >> > Aborted (core dumped)
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > ------------------------------------------------------------------------------
>>>>> >> > See everything from the browser to the database with AppDynamics
>>>>> >> > Get end-to-end visibility with application monitoring from
>>>>> >> > AppDynamics
>>>>> >> > Isolate bottlenecks and diagnose root cause in seconds.
>>>>> >> > Start your free trial of AppDynamics Pro today!
>>>>> >> >
>>>>> >> >
>>>>> >> > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>>> >> > _______________________________________________
>>>>> >> > Kaldi-users mailing list
>>>>> >> > Kal...@li...
>>>>> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>>>> >> >
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> ------------------------------------------------------------------------------
>>>>> >> See everything from the browser to the database with AppDynamics
>>>>> >> Get end-to-end visibility with application monitoring from
>>>>> >> AppDynamics
>>>>> >> Isolate bottlenecks and diagnose root cause in seconds.
>>>>> >> Start your free trial of AppDynamics Pro today!
>>>>> >>
>>>>> >>
>>>>> >> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>>> >> _______________________________________________
>>>>> >> Kaldi-users mailing list
>>>>> >> Kal...@li...
>>>>> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > ------------------------------------------------------------------------------
>>>>> > See everything from the browser to the database with AppDynamics
>>>>> > Get end-to-end visibility with application monitoring from
>>>>> > AppDynamics
>>>>> > Isolate bottlenecks and diagnose root cause in seconds.
>>>>> > Start your free trial of AppDynamics Pro today!
>>>>> >
>>>>> > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>>> > _______________________________________________
>>>>> > Kaldi-users mailing list
>>>>> > Kal...@li...
>>>>> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>>>> >
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> See everything from the browser to the database with AppDynamics
>>>>> Get end-to-end visibility with application monitoring from AppDynamics
>>>>> Isolate bottlenecks and diagnose root cause in seconds.
>>>>> Start your free trial of AppDynamics Pro today!
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>>> _______________________________________________
>>>>> Kaldi-users mailing list
>>>>> Kal...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> See everything from the browser to the database with AppDynamics
>>>> Get end-to-end visibility with application monitoring from AppDynamics
>>>> Isolate bottlenecks and diagnose root cause in seconds.
>>>> Start your free trial of AppDynamics Pro today!
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Kaldi-users mailing list
>>>> Kal...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>>>
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> See everything from the browser to the database with AppDynamics
>>> Get end-to-end visibility with application monitoring from AppDynamics
>>> Isolate bottlenecks and diagnose root cause in seconds.
>>> Start your free trial of AppDynamics Pro today!
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Kaldi-users mailing list
>>> Kal...@li...
>>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> See everything from the browser to the database with AppDynamics
>> Get end-to-end visibility with application monitoring from AppDynamics
>> Isolate bottlenecks and diagnose root cause in seconds.
>> Start your free trial of AppDynamics Pro today!
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Kaldi-users mailing list
>> Kal...@li...
>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>
>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>