Re: [Kaldi-users] matrix-lib-test test fail with MKL 11.0 update5

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Thanks

On 31 July 2013 09:01, Mailing list used for User Communication and Updates
<kal...@li...> wrote:

> Hi-- yes there is but it depends what you're doing.  I don't recall
> even the approximate percentages of improvement though.  You can get a
> temporary research license for MKL for free I think, so you could try
> it out.
> Dan
>
>
> On Tue, Jul 30, 2013 at 7:59 PM, Mailing list used for User
> Communication and Updates <kal...@li...> wrote:
> > Hi,
> > This is a bit off-topic, but is there a bit improvement in speed when
> using
> > MKL vs ATLAS?
> >
> > Thanks
> >
> > Paul
> >
> >
> > On 30 July 2013 23:29, Mailing list used for User Communication and
> Updates
> > <kal...@li...> wrote:
> >>
> >> That's right.  They are working on it.
> >>
> >> --Mike
> >>
> >>
> >> -----Original Message-----
> >> From: Mailing list used for User Communication and Updates
> >> [mailto:kal...@li...]
> >> Sent: Monday, July 29, 2013 9:58 AM
> >> To: kal...@li...
> >> Subject: Re: [Kaldi-users] matrix-lib-test test fail with MKL 11.0
> update5
> >>
> >> I previously spoke to Michael Deisher at Intel about this, he says he
> has
> >> asked the relevant team.  So I think they are aware of the problem.
> >> Dan
> >>
> >>
> >> On Mon, Jul 29, 2013 at 12:06 PM, Mailing list used for User
> Communication
> >> and Updates <kal...@li...> wrote:
> >> > It has been reported on intel dev forum.
> >> > http://software.intel.com/en-us/forums/topic/402436
> >> > http://software.intel.com/en-us/forums/topic/402473
> >> >
> >> > The problem is caused by  the incorrect query result of workspace size
> >> > given by dgesvd().
> >> >
> >> > 11.0.5 seems to be a very buggy version.
> >> >
> >> >
> >> >
> >> > On Mon, Jul 29, 2013 at 11:39 PM, Mailing list used for User
> >> > Communication and Updates <kal...@li...> wrote:
> >> >>
> >> >> I just checked different versions of MKL and indeed, this was
> >> >> introduced in the  update 5. Anyone with the commercial MKL support
> >> >> cares to submit a ticket to Intel?
> >> >> y.
> >> >>
> >> >>
> >> >> On Wed, Jul 24, 2013 at 12:42 PM, Mailing list used for User
> >> >> Communication and Updates <kal...@li...> wrote:
> >> >>>
> >> >>> You are right.... I just downgraded to MKL 11.0update1 and passed
> >> >>> the test.
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Thu, Jul 25, 2013 at 12:28 AM, Mailing list used for User
> >> >>> Communication and Updates <kal...@li...>
> wrote:
> >> >>>>
> >> >>>> I saw some changes regarding improving performance of *GESVD in the
> >> >>>> MKL
> >> >>>> 11.0 update5  changelog. I wonder if they messed something up? How
> >> >>>> about trying slightly "older" version of the MKL? If you are on the
> >> >>>> subscription of Intel Software Network, you can download older
> >> >>>> versions of the MKL at no charge...
> >> >>>>
> >> >>>> We are using Intel 11.0 and the tests pass without any problem...
> >> >>>>
> >> >>>> yenda
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Wed, Jul 24, 2013 at 12:14 PM, Mailing list used for User
> >> >>>> Communication and Updates <kal...@li...>
> wrote:
> >> >>>>>
> >> >>>>> I can't see any problems in the Kaldi code.  Perhaps this is an
> >> >>>>> older version of MKL?  It could be a bug in MKL itself.  MKL tests
> >> >>>>> OK for me.
> >> >>>>> Dan
> >> >>>>>
> >> >>>>>
> >> >>>>> On Wed, Jul 24, 2013 at 11:55 AM, Mailing list used for User
> >> >>>>> Communication and Updates <kal...@li...>
> wrote:
> >> >>>>> > here's the dbg info copied from eclipse.
> >> >>>>> > PS: I can pass the test with atlas provided by centos repo
> >> >>>>> > atlas-sse3.x86_64                          3.8.4-2.el6
> >> >>>>> > @base
> >> >>>>> > atlas-sse3-devel.x86_64                    3.8.4-2.el6
> >> >>>>> > @base
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > stack:
> >> >>>>> >
> >> >>>>> > matrix-lib-test [C/C++ Application] matrix-lib-test [11312]
> >> >>>>> > [cores: 1] Thread [1] 11312 [core: 1] (Suspended : Signal :
> >> >>>>> > SIGABRT:Aborted)
> >> >>>>> > raise() at 0x33090328a5
> >> >>>>> > abort() at 0x3309034085
> >> >>>>> > kaldi::KaldiAssertFailure_() at kaldi-error.cc:136 0x4cf401
> >> >>>>> > kaldi::MatrixBase<double>::LapackGesvd() at kaldi-matrix.cc:401
> >> >>>>> > 0x48fdd3
> >> >>>>> > kaldi::MatrixBase<double>::DestructiveSvd() at
> >> >>>>> > kaldi-matrix.cc:1,330
> >> >>>>> > 0x48b902
> >> >>>>> > kaldi::MatrixBase<double>::Svd() at kaldi-matrix.cc:1,352
> >> >>>>> > 0x48b9bb
> >> >>>>> > kaldi::MatrixBase<double>::Svd() at kaldi-matrix.h:315 0x468e50
> >> >>>>> > kaldi::MatrixBase<double>::Cond() at kaldi-matrix.cc:1,222
> >> >>>>> > 0x48bf47
> >> >>>>> > kaldi::InitRand<double>() at matrix-lib-test.cc:74 0x42dd7e
> >> >>>>> > kaldi::UnitTestCompressedMatrix<double>() at
> >> >>>>> > matrix-lib-test.cc:3,532 0x43055a
> >> >>>>> > kaldi::MatrixUnitTest<double>() at matrix-lib-test.cc:3,705
> >> >>>>> > 0x42de8b
> >> >>>>> > main() at matrix-lib-test.cc:3,829 0x42dac7
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > variable list after invoking the following    @line393
> >> >>>>> > kaldi-matrix.cc
> >> >>>>> >   clapack_Xgesvd(v_job, u_job,
> >> >>>>> >                  &M, &N, data_, &LDA,
> >> >>>>> >                  s->Data(),
> >> >>>>> >                  V->Data(), &V_stride,
> >> >>>>> >                  U->Data(), &U_stride,
> >> >>>>> >                  p_work, &l_work,
> >> >>>>> >                  &result);
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > __func__ char [12] 0x4d5418
> >> >>>>> > M KaldiBlasInt 1
> >> >>>>> > LDA KaldiBlasInt 2
> >> >>>>> > U kaldi::MatrixBase<double> * 0x7ffffffface0 data_ double *
> >> >>>>> > 0x7413b0 *data_ double 0 num_cols_ kaldi::MatrixIndexT 1
> >> >>>>> > num_rows_ kaldi::MatrixIndexT 10 stride_ kaldi::MatrixIndexT 2
> >> >>>>> > l_work KaldiBlasInt 5 p_work double * 0x741460 *p_work double
> >> >>>>> > 1.0829823019173015e-312 temp void * 0x741460 N KaldiBlasInt 10 V
> >> >>>>> > kaldi::MatrixBase<double> * 0x7fffffffacc0 data_ double *
> >> >>>>> > 0x7406a0 *data_ double 0 num_cols_ kaldi::MatrixIndexT 1
> >> >>>>> > num_rows_ kaldi::MatrixIndexT 1 stride_ kaldi::MatrixIndexT 2
> >> >>>>> > result KaldiBlasInt -13 tmpU kaldi::Matrix<double> {...}
> >> >>>>> > V_stride KaldiBlasInt 2 work_query double 5 u_job char *
> >> >>>>> > 0x4d4fea "N"
> >> >>>>> > tmpV kaldi::Matrix<double> {...} U_stride KaldiBlasInt 2 v_job
> >> >>>>> > char * 0x4d4fea "N"
> >> >>>>> > this kaldi::MatrixBase<double> * const 0x7fffffffaf80 data_
> >> >>>>> > double * 0x741300 *data_ double 0.78239572048187256 num_cols_
> >> >>>>> > kaldi::MatrixIndexT 1 num_rows_ kaldi::MatrixIndexT 10 stride_
> >> >>>>> > kaldi::MatrixIndexT 2 s kaldi::VectorBase<double> *
> >> >>>>> > 0x7fffffffb190 data_ double * 0x7404a0 *data_ double 0 dim_
> >> >>>>> > kaldi::MatrixIndexT 1 U_in kaldi::MatrixBase<double> * 0x0 V_in
> >> >>>>> > kaldi::MatrixBase<double> * 0x0
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > On Wed, Jul 24, 2013 at 11:04 PM, Mailing list used for User
> >> >>>>> > Communication and Updates <kal...@li...>
> >> >>>>> > wrote:
> >> >>>>> >>
> >> >>>>> >> Could you please run this in a debugger and show me the values
> >> >>>>> >> of p_work, l_work and result, e.g.
> >> >>>>> >>
> >> >>>>> >> gdb matrix-lib
> >> >>>>> >> (gdb) r
> >> >>>>> >> [wait till it crashes]
> >> >>>>> >> (gdb) up
> >> >>>>> >> (gdb) up    # not sure how many times
> >> >>>>> >> (gdb) up
> >> >>>>> >> (gdb) p l_work
> >> >>>>> >> (gdb) p p_work
> >> >>>>> >> (gdb) p result
> >> >>>>> >>
> >> >>>>> >> Dan
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >> On Wed, Jul 24, 2013 at 10:34 AM, Mailing list used for User
> >> >>>>> >> Communication and Updates <kal...@li...>
> >> >>>>> >> wrote:
> >> >>>>> >> > Hi,
> >> >>>>> >> >
> >> >>>>> >> > I tried latest trunk version with MKL. I have no problem when
> >> >>>>> >> > running `make all`, but get a FAIL when testing. Can anyone
> >> >>>>> >> > help? Thanks.
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> > svn revision 2720 trunk
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> > configuration line:
> >> >>>>> >> > $ ./configure --threaded-math=yes --mathlib=MKL
> >> >>>>> >> > --use-cuda=yes --mkl-root=$MKLROOT
> >> >>>>> >> > --omp-libdir=/opt/intel/lib/intel64
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> > MKL version 11.0 update 5 (latest)
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> > Linux version CentOS 6.4
> >> >>>>> >> > $ uname -a
> >> >>>>> >> > Linux 2.6.32-358.6.2.el6.x86_64 #1 SMP Thu May 16 20:59:36
> >> >>>>> >> > UTC
> >> >>>>> >> > 2013
> >> >>>>> >> > x86_64
> >> >>>>> >> > x86_64 x86_64 GNU/Linux
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> > $ gcc -v
> >> >>>>> >> > Using built-in specs.
> >> >>>>> >> > Target: x86_64-redhat-linux
> >> >>>>> >> > Configured with: ../configure --prefix=/usr
> >> >>>>> >> > --mandir=/usr/share/man --infodir=/usr/share/info
> >> >>>>> >> > --with-bugurl=http://bugzilla.redhat.com/bugzilla
> >> >>>>> >> > --enable-bootstrap --enable-shared --enable-threads=posix
> >> >>>>> >> > --enable-checking=release --with-system-zlib
> >> >>>>> >> > --enable-__cxa_atexit --disable-libunwind-exceptions
> >> >>>>> >> > --enable-gnu-unique-object
> >> >>>>> >> > --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
> >> >>>>> >> > --enable-java-awt=gtk
> >> >>>>> >> > --disable-dssi
> >> >>>>> >> > --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
> >> >>>>> >> > --enable-libgcj-multifile --enable-java-maintainer-mode
> >> >>>>> >> > --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
> >> >>>>> >> > --disable-libjava-multilib
> >> >>>>> >> > --with-ppl --with-cloog --with-tune=generic
> >> >>>>> >> > --with-arch_32=i686 --build=x86_64-redhat-linux Thread model:
> >> >>>>> >> > posix gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC)
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> > ~/kaldi/trunk/src/matrix
> >> >>>>> >> > $ make test
> >> >>>>> >> > Running matrix-lib-test .../bin/sh: line 1:  5561 Aborted
> >> >>>>> >> > (core dumped) ./$x > /dev/null 2>&1 ... FAIL Running
> >> >>>>> >> > kaldi-gpsr-test ...... SUCCESS
> >> >>>>> >> > make: *** [test] Error 1
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> > $ ./matrix-lib-test
> >> >>>>> >> > ...
> >> >>>>> >> > ...
> >> >>>>> >> > VLOG[2] (UnitTestLbfgs():matrix-lib-test.cc:2734) L-BFGS
> >> >>>>> >> > finished after
> >> >>>>> >> > 59
> >> >>>>> >> > function evaluations.
> >> >>>>> >> > LOG (UnitTestCompressedMatrix():matrix-lib-test.cc:3623) M =
> >> >>>>> >> > [ ]
> >> >>>>> >> >
> >> >>>>> >> > LOG (UnitTestCompressedMatrix():matrix-lib-test.cc:3624) M2 =
> >> >>>>> >> > [ ]
> >> >>>>> >> >
> >> >>>>> >> > LOG (UnitTestCompressedMatrix():matrix-lib-test.cc:3626)
> >> >>>>> >> > Compressed
> >> >>>>> >> > matrix,
> >> >>>>> >> > tot = 0, diff = 0
> >> >>>>> >> >
> >> >>>>> >> > MKL ERROR: Parameter 13 was incorrect on entry to DGESVD.
> >> >>>>> >> > KALDI_ASSERT: at LapackGesvd:kaldi-matrix.cc:401, failed:
> >> >>>>> >> > result
> >> >>>>> >> > >= 0 &&
> >> >>>>> >> > "Call to CLAPACK dgesvd_ called with wrong arguments"
> >> >>>>> >> > Stack trace is:
> >> >>>>> >> > kaldi::KaldiGetStackTrace()
> >> >>>>> >> > kaldi::KaldiAssertFailure_(char const*, char const*, int,
> >> >>>>> >> > char
> >> >>>>> >> > const*)
> >> >>>>> >> > kaldi::MatrixBase<double>::LapackGesvd(kaldi::VectorBase<doub
> >> >>>>> >> > le>*, kaldi::MatrixBase<double>*, kaldi::MatrixBase<double>*)
> >> >>>>> >> >
> >> >>>>> >> > kaldi::MatrixBase<double>::DestructiveSvd(kaldi::VectorBase<d
> >> >>>>> >> > ouble>*, kaldi::MatrixBase<double>*,
> >> >>>>> >> > kaldi::MatrixBase<double>*)
> >> >>>>> >> > kaldi::MatrixBase<double>::Svd(kaldi::VectorBase<double>*,
> >> >>>>> >> > kaldi::MatrixBase<double>*, kaldi::MatrixBase<double>*) const
> >> >>>>> >> > .
> >> >>>>> >> > .
> >> >>>>> >> > .
> >> >>>>> >> > ./matrix-lib-test() [0x43055a]
> >> >>>>> >> > ./matrix-lib-test() [0x42de8b]
> >> >>>>> >> > ./matrix-lib-test(main+0x1d) [0x42dac7]
> >> >>>>> >> > /lib64/libc.so.6(__libc_start_main+0xfd) [0x330901ecdd]
> >> >>>>> >> > ./matrix-lib-test() [0x42ced9] Aborted (core dumped)
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> > -------------------------------------------------------------
> >> >>>>> >> > ----------------- See everything from the browser to the
> >> >>>>> >> > database with AppDynamics Get end-to-end visibility with
> >> >>>>> >> > application monitoring from AppDynamics Isolate bottlenecks
> >> >>>>> >> > and diagnose root cause in seconds.
> >> >>>>> >> > Start your free trial of AppDynamics Pro today!
> >> >>>>> >> >
> >> >>>>> >> >
> >> >>>>> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/41
> >> >>>>> >> > 40/ostg.clktrk
> >> >>>>> >> > _______________________________________________
> >> >>>>> >> > Kaldi-users mailing list
> >> >>>>> >> > Kal...@li...
> >> >>>>> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >>>>> >> >
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >> ---------------------------------------------------------------
> >> >>>>> >> --------------- See everything from the browser to the database
> >> >>>>> >> with AppDynamics Get end-to-end visibility with application
> >> >>>>> >> monitoring from AppDynamics Isolate bottlenecks and diagnose
> >> >>>>> >> root cause in seconds.
> >> >>>>> >> Start your free trial of AppDynamics Pro today!
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140
> >> >>>>> >> /ostg.clktrk _______________________________________________
> >> >>>>> >> Kaldi-users mailing list
> >> >>>>> >> Kal...@li...
> >> >>>>> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > ----------------------------------------------------------------
> >> >>>>> > -------------- See everything from the browser to the database
> >> >>>>> > with AppDynamics Get end-to-end visibility with application
> >> >>>>> > monitoring from AppDynamics Isolate bottlenecks and diagnose
> >> >>>>> > root cause in seconds.
> >> >>>>> > Start your free trial of AppDynamics Pro today!
> >> >>>>> >
> >> >>>>> >
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/
> >> >>>>> > ostg.clktrk _______________________________________________
> >> >>>>> > Kaldi-users mailing list
> >> >>>>> > Kal...@li...
> >> >>>>> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >>>>> >
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------------------------------------------
> >> >>>>> ------------ See everything from the browser to the database with
> >> >>>>> AppDynamics Get end-to-end visibility with application monitoring
> >> >>>>> from AppDynamics Isolate bottlenecks and diagnose root cause in
> >> >>>>> seconds.
> >> >>>>> Start your free trial of AppDynamics Pro today!
> >> >>>>>
> >> >>>>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/os
> >> >>>>> tg.clktrk _______________________________________________
> >> >>>>> Kaldi-users mailing list
> >> >>>>> Kal...@li...
> >> >>>>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> -------------------------------------------------------------------
> >> >>>> ----------- See everything from the browser to the database with
> >> >>>> AppDynamics Get end-to-end visibility with application monitoring
> >> >>>> from AppDynamics Isolate bottlenecks and diagnose root cause in
> >> >>>> seconds.
> >> >>>> Start your free trial of AppDynamics Pro today!
> >> >>>>
> >> >>>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ost
> >> >>>> g.clktrk _______________________________________________
> >> >>>> Kaldi-users mailing list
> >> >>>> Kal...@li...
> >> >>>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >>>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --------------------------------------------------------------------
> >> >>> ---------- See everything from the browser to the database with
> >> >>> AppDynamics Get end-to-end visibility with application monitoring
> >> >>> from AppDynamics Isolate bottlenecks and diagnose root cause in
> >> >>> seconds.
> >> >>> Start your free trial of AppDynamics Pro today!
> >> >>>
> >> >>>
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg
> >> >>> .clktrk _______________________________________________
> >> >>> Kaldi-users mailing list
> >> >>> Kal...@li...
> >> >>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> --------- See everything from the browser to the database with
> >> >> AppDynamics Get end-to-end visibility with application monitoring
> >> >> from AppDynamics Isolate bottlenecks and diagnose root cause in
> >> >> seconds.
> >> >> Start your free trial of AppDynamics Pro today!
> >> >>
> >> >> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg
> .
> >> >> clktrk _______________________________________________
> >> >> Kaldi-users mailing list
> >> >> Kal...@li...
> >> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >>
> >> >
> >> >
> >> >
> >> > ----------------------------------------------------------------------
> >> > -------- See everything from the browser to the database with
> >> > AppDynamics Get end-to-end visibility with application monitoring from
> >> > AppDynamics Isolate bottlenecks and diagnose root cause in seconds.
> >> > Start your free trial of AppDynamics Pro today!
> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.c
> >> > lktrk _______________________________________________
> >> > Kaldi-users mailing list
> >> > Kal...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Get your SQL database under version control now!
> >> Version control is standard for application code, but databases havent
> >> caught up. So what steps can you take to put your SQL databases under
> >> version control? Why should you start doing it? Read more to find out.
> >>
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
> >> _______________________________________________
> >> Kaldi-users mailing list
> >> Kal...@li...
> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Get your SQL database under version control now!
> >> Version control is standard for application code, but databases havent
> >> caught up. So what steps can you take to put your SQL databases under
> >> version control? Why should you start doing it? Read more to find out.
> >>
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
> >> _______________________________________________
> >> Kaldi-users mailing list
> >> Kal...@li...
> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Get your SQL database under version control now!
> > Version control is standard for application code, but databases havent
> > caught up. So what steps can you take to put your SQL databases under
> > version control? Why should you start doing it? Read more to find out.
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Kaldi-users mailing list
> > Kal...@li...
> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >
>
>
> ------------------------------------------------------------------------------
> Get your SQL database under version control now!
> Version control is standard for application code, but databases havent
> caught up. So what steps can you take to put your SQL databases under
> version control? Why should you start doing it? Read more to find out.
> http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>