Re: [Kaldi-developers] Training DNN with GPU is too slow

[Daniel P. <dp...@gm...>]

Also, make sure the different jobs are using different GPUs.  Setting
to exclusive mode (nvidia-smi -c 1) will help.
Dan


On Wed, Jul 15, 2015 at 9:04 AM, Jan Trmal <jt...@gm...> wrote:
> Are you sure you are using CUDA? You have to enable the support manually
> when running the kaldi configure script. As you have gaming cards,
> nvidia-smi won't probably display GPU utilization. You can have a look at
> the temperature of the cards to verify the GPUs are used.
> Expect ~ 4 times longer time for the ensemble training than for the "normal"
> training.
> y.
>
> On Wed, Jul 15, 2015 at 5:10 AM, Cemil Demir <cem...@gm...> wrote:
>>
>> Hi,
>>
>> I have a workstation which have 4 NVIDIA TITAN X GPU card.
>>
>> I want to train a DNN model using egs/babel/s5c recipe. I am using
>> "run-2a-nnet-ensemble-gpu.sh" script.
>>
>> When I use "run-2a-nnet-gpu.sh" script, the accuracy is below as compared
>> to sgmm-mmi case. Therefore I
>>
>> Total amount of training data is 13 hours.
>> want to use "ensemble" version.
>>
>> Training takes about 35 hours. I think, it is too slow.
>>
>> Could give me any suggestion to speed up training?
>>
>> İt is a little bit urgent for me.
>>
>> Thank you.
>>
>> --
>> Cemil Demir
>>
>> cem...@gm...
>>
>>
>> ------------------------------------------------------------------------------
>> Don't Limit Your Business. Reach for the Cloud.
>> GigeNET's Cloud Solutions provide you with the tools and support that
>> you need to offload your IT needs and focus on growing your business.
>> Configured For All Businesses. Start Your Cloud Today.
>> https://www.gigenetcloud.com/
>> _______________________________________________
>> Kaldi-developers mailing list
>> Kal...@li...
>> https://lists.sourceforge.net/lists/listinfo/kaldi-developers
>>
>
>
> ------------------------------------------------------------------------------
> Don't Limit Your Business. Reach for the Cloud.
> GigeNET's Cloud Solutions provide you with the tools and support that
> you need to offload your IT needs and focus on growing your business.
> Configured For All Businesses. Start Your Cloud Today.
> https://www.gigenetcloud.com/
> _______________________________________________
> Kaldi-developers mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-developers
>