Re: [Kaldi-users] Kaldi DNN implemenation

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi All,

When running DNN training on GPUs, I am getting following error.

*Log File : exp/tri4b_pretrain-dbn/_pretrain_dbn.log*

*# PRE-TRAINING RBM LAYER 1
Initializing 'exp/tri4b_pretrain-dbn/1.rbm.init'
Traceback (most recent call last):
  File "utils/nnet/gen_rbm_init.py", line 40, in ?
    dimL.append(int(dimStrL[i]))
ValueError: invalid literal for int(): *

I am running this in a GPU cluster which assigns the job to a GPU
dynamically, So I cannot configure the *_gpu_id= # manually select GPU id
to run on, (-1 disables GPU)*.
Can this be the cause?

Thanks,
Lahiru

On Fri, Jun 28, 2013 at 11:06 PM, Mailing list used for User Communication
and Updates <kal...@li...> wrote:

> It's not the same as that.  Each machine does SGD separately and,
> periodically, the parameters are averaged across machines.
> Dan
>
>
> On Fri, Jun 28, 2013 at 11:03 AM, Mailing list used for User
> Communication and Updates <kal...@li...> wrote:
> > Wow, nice.
> > Does the implementation similar to the Jeff Dean's paper Large Scale
> > Distributed Deep Networks
> > (
> http://www.cs.toronto.edu/~ranzato/publications/DistBeliefNIPS2012_withAppendix.pdf
> )
> > ?
> > Does Kaldi use Asynchronous SGD?
> >
> > Please give me a brief description.
> >
> > Thanks,
> > Lahiru
> >
> >
> > On Fri, Jun 28, 2013 at 10:28 PM, Mailing list used for User
> Communication
> > and Updates <kal...@li...> wrote:
> >>
> >> It's on multiple machines and also multiple threads per machine.
> >> Dan
> >>
> >>
> >> On Fri, Jun 28, 2013 at 2:05 AM, Mailing list used for User
> >> Communication and Updates <kal...@li...> wrote:
> >> > Thanks guys :-)
> >> >
> >> > Dan, is your setup for distributed training? Or is it only parallelize
> >> > with
> >> > in a single machine?
> >> >
> >> > Thanks,
> >> > Lahiru
> >> >
> >> >
> >> >
> >> > On Fri, Jun 28, 2013 at 5:29 AM, Mailing list used for User
> >> > Communication
> >> > and Updates <kal...@li...> wrote:
> >> >>
> >> >> In my setup there is RBM pre-training:
> >> >> http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf
> >> >> <http://www.cs.toronto.edu/%7Ehinton/absps/guideTR.pdf>
> >> >> followed by per-frame cross entropy training and sMBR training:
> >> >> http://www.danielpovey.com/files/2013_interspeech_dnn.pdf
> >> >>
> >> >>
> >> >> Dne 27.6.2013 13:21, Mailing list used for User Communication and
> >> >> Updates napsal(a):
> >> >> > There are basically two setups there: Karel's setup, generally
> called
> >> >> > run_dnn.sh or run_nnet.sh, which is for GPUs, and my setup, called
> >> >> > run_nnet_cpu.sh, which is for CPUs in parallel.  Karel's setup may
> >> >> > have an ICASSP paper, Karel can tell you.  Mine is mostly
> >> >> > unpublished.
> >> >> >
> >> >> > Dan
> >> >> >
> >> >> >
> >> >> > On Thu, Jun 27, 2013 at 5:31 AM, Mailing list used for User
> >> >> > Communication and Updates <kal...@li...>
> wrote:
> >> >> >> Hi All,
> >> >> >>
> >> >> >> I am in the process of running the wsj/s5 recipe. Now I am about
> the
> >> >> >> run DNN
> >> >> >> experiments and specifically interested in the DNN training.  I am
> >> >> >> planning
> >> >> >> to look into the DNN code for more understanding. Since there are
> >> >> >> many
> >> >> >> DNN
> >> >> >> variants, could anyone tell me the papers Kalid DNN implementation
> >> >> >> represents?
> >> >> >>
> >> >> >> Thanks,
> >> >> >> Lahiru
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> ------------------------------------------------------------------------------
> >> >> >> This SF.net email is sponsored by Windows:
> >> >> >>
> >> >> >> Build for Windows Store.
> >> >> >>
> >> >> >> http://p.sf.net/sfu/windows-dev2dev
> >> >> >> _______________________________________________
> >> >> >> Kaldi-users mailing list
> >> >> >> Kal...@li...
> >> >> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> ------------------------------------------------------------------------------
> >> >> > This SF.net email is sponsored by Windows:
> >> >> >
> >> >> > Build for Windows Store.
> >> >> >
> >> >> > http://p.sf.net/sfu/windows-dev2dev
> >> >> > _______________________________________________
> >> >> > Kaldi-users mailing list
> >> >> > Kal...@li...
> >> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> ------------------------------------------------------------------------------
> >> >> This SF.net email is sponsored by Windows:
> >> >>
> >> >> Build for Windows Store.
> >> >>
> >> >> http://p.sf.net/sfu/windows-dev2dev
> >> >> _______________________________________________
> >> >> Kaldi-users mailing list
> >> >> Kal...@li...
> >> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >
> >> >
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------------
> >> > This SF.net email is sponsored by Windows:
> >> >
> >> > Build for Windows Store.
> >> >
> >> > http://p.sf.net/sfu/windows-dev2dev
> >> > _______________________________________________
> >> > Kaldi-users mailing list
> >> > Kal...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >> >
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> This SF.net email is sponsored by Windows:
> >>
> >> Build for Windows Store.
> >>
> >> http://p.sf.net/sfu/windows-dev2dev
> >> _______________________________________________
> >> Kaldi-users mailing list
> >> Kal...@li...
> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > This SF.net email is sponsored by Windows:
> >
> > Build for Windows Store.
> >
> > http://p.sf.net/sfu/windows-dev2dev
> > _______________________________________________
> > Kaldi-users mailing list
> > Kal...@li...
> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
> >
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
>
> Build for Windows Store.
>
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>