Re: [Kaldi-users] Kaldi DNN implemenation

[Mailing l. u. f. U. C. a. U. <kal...@li...>]

Sorry, I was wrong. It selects the GPU automatically,

I found the error in *exp/tri4b_pretrain-dbn/log/cmvn_glob_fwd.log *file.

ERROR (nnet-forward:PdfPrior():nnet-pdf-prior.cc:26) --class-frame-counts
is empty: Cannot initialize priors without the counts.
ERROR (nnet-forward:main():nnet-forward.cc:196) ERROR
(nnet-forward:PdfPrior():nnet-pdf-prior.cc:26) --class-frame-counts is
empty: Cannot initialize priors without the counts.


Thanks
Lahiru


On Tue, Jul 2, 2013 at 9:10 PM, Lahiru Samarakoon <lah...@gm...>wrote:

> Hi All,
>
> When running DNN training on GPUs, I am getting following error.
>
> *Log File : exp/tri4b_pretrain-dbn/_pretrain_dbn.log*
>
> *# PRE-TRAINING RBM LAYER 1
> Initializing 'exp/tri4b_pretrain-dbn/1.rbm.init'
> Traceback (most recent call last):
>   File "utils/nnet/gen_rbm_init.py", line 40, in ?
>     dimL.append(int(dimStrL[i]))
> ValueError: invalid literal for int(): *
>
>
> I am running this in a GPU cluster which assigns the job to a GPU
> dynamically, So I cannot configure the *_gpu_id= # manually select GPU id
> to run on, (-1 disables GPU)*.
> Can this be the cause?
>
> Thanks,
> Lahiru
>
>
> On Fri, Jun 28, 2013 at 11:06 PM, Mailing list used for User Communication
> and Updates <kal...@li...> wrote:
>
>> It's not the same as that.  Each machine does SGD separately and,
>> periodically, the parameters are averaged across machines.
>> Dan
>>
>>
>> On Fri, Jun 28, 2013 at 11:03 AM, Mailing list used for User
>> Communication and Updates <kal...@li...> wrote:
>> > Wow, nice.
>> > Does the implementation similar to the Jeff Dean's paper Large Scale
>> > Distributed Deep Networks
>> > (
>> http://www.cs.toronto.edu/~ranzato/publications/DistBeliefNIPS2012_withAppendix.pdf
>> )
>> > ?
>> > Does Kaldi use Asynchronous SGD?
>> >
>> > Please give me a brief description.
>> >
>> > Thanks,
>> > Lahiru
>> >
>> >
>> > On Fri, Jun 28, 2013 at 10:28 PM, Mailing list used for User
>> Communication
>> > and Updates <kal...@li...> wrote:
>> >>
>> >> It's on multiple machines and also multiple threads per machine.
>> >> Dan
>> >>
>> >>
>> >> On Fri, Jun 28, 2013 at 2:05 AM, Mailing list used for User
>> >> Communication and Updates <kal...@li...> wrote:
>> >> > Thanks guys :-)
>> >> >
>> >> > Dan, is your setup for distributed training? Or is it only
>> parallelize
>> >> > with
>> >> > in a single machine?
>> >> >
>> >> > Thanks,
>> >> > Lahiru
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Jun 28, 2013 at 5:29 AM, Mailing list used for User
>> >> > Communication
>> >> > and Updates <kal...@li...> wrote:
>> >> >>
>> >> >> In my setup there is RBM pre-training:
>> >> >> http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf
>> >> >> <http://www.cs.toronto.edu/%7Ehinton/absps/guideTR.pdf>
>> >> >> followed by per-frame cross entropy training and sMBR training:
>> >> >> http://www.danielpovey.com/files/2013_interspeech_dnn.pdf
>> >> >>
>> >> >>
>> >> >> Dne 27.6.2013 13:21, Mailing list used for User Communication and
>> >> >> Updates napsal(a):
>> >> >> > There are basically two setups there: Karel's setup, generally
>> called
>> >> >> > run_dnn.sh or run_nnet.sh, which is for GPUs, and my setup, called
>> >> >> > run_nnet_cpu.sh, which is for CPUs in parallel.  Karel's setup may
>> >> >> > have an ICASSP paper, Karel can tell you.  Mine is mostly
>> >> >> > unpublished.
>> >> >> >
>> >> >> > Dan
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Jun 27, 2013 at 5:31 AM, Mailing list used for User
>> >> >> > Communication and Updates <kal...@li...>
>> wrote:
>> >> >> >> Hi All,
>> >> >> >>
>> >> >> >> I am in the process of running the wsj/s5 recipe. Now I am about
>> the
>> >> >> >> run DNN
>> >> >> >> experiments and specifically interested in the DNN training.  I
>> am
>> >> >> >> planning
>> >> >> >> to look into the DNN code for more understanding. Since there are
>> >> >> >> many
>> >> >> >> DNN
>> >> >> >> variants, could anyone tell me the papers Kalid DNN
>> implementation
>> >> >> >> represents?
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> Lahiru
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> ------------------------------------------------------------------------------
>> >> >> >> This SF.net email is sponsored by Windows:
>> >> >> >>
>> >> >> >> Build for Windows Store.
>> >> >> >>
>> >> >> >> http://p.sf.net/sfu/windows-dev2dev
>> >> >> >> _______________________________________________
>> >> >> >> Kaldi-users mailing list
>> >> >> >> Kal...@li...
>> >> >> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> ------------------------------------------------------------------------------
>> >> >> > This SF.net email is sponsored by Windows:
>> >> >> >
>> >> >> > Build for Windows Store.
>> >> >> >
>> >> >> > http://p.sf.net/sfu/windows-dev2dev
>> >> >> > _______________________________________________
>> >> >> > Kaldi-users mailing list
>> >> >> > Kal...@li...
>> >> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> ------------------------------------------------------------------------------
>> >> >> This SF.net email is sponsored by Windows:
>> >> >>
>> >> >> Build for Windows Store.
>> >> >>
>> >> >> http://p.sf.net/sfu/windows-dev2dev
>> >> >> _______________________________________________
>> >> >> Kaldi-users mailing list
>> >> >> Kal...@li...
>> >> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> ------------------------------------------------------------------------------
>> >> > This SF.net email is sponsored by Windows:
>> >> >
>> >> > Build for Windows Store.
>> >> >
>> >> > http://p.sf.net/sfu/windows-dev2dev
>> >> > _______________________________________________
>> >> > Kaldi-users mailing list
>> >> > Kal...@li...
>> >> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
>> >> >
>> >>
>> >>
>> >>
>> ------------------------------------------------------------------------------
>> >> This SF.net email is sponsored by Windows:
>> >>
>> >> Build for Windows Store.
>> >>
>> >> http://p.sf.net/sfu/windows-dev2dev
>> >> _______________________________________________
>> >> Kaldi-users mailing list
>> >> Kal...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>> >
>> >
>> >
>> >
>> ------------------------------------------------------------------------------
>> > This SF.net email is sponsored by Windows:
>> >
>> > Build for Windows Store.
>> >
>> > http://p.sf.net/sfu/windows-dev2dev
>> > _______________________________________________
>> > Kaldi-users mailing list
>> > Kal...@li...
>> > https://lists.sourceforge.net/lists/listinfo/kaldi-users
>> >
>>
>>
>> ------------------------------------------------------------------------------
>> This SF.net email is sponsored by Windows:
>>
>> Build for Windows Store.
>>
>> http://p.sf.net/sfu/windows-dev2dev
>> _______________________________________________
>> Kaldi-users mailing list
>> Kal...@li...
>> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>>
>
>