Re: [Kaldi-developers] Training DNN with GPU is too slow

[Cemil D. <cem...@gm...>]

Yes, I am using CUDA. I am using nvidia-smi and GPU utilizaiton is seen.
Although, TITAN X has 12 GB memory, only 300 MB of it is used by nnet
trainer process.

On Wed, Jul 15, 2015 at 7:04 PM, Jan Trmal <jt...@gm...> wrote:

> Are you sure you are using CUDA? You have to enable the support
> manually when running the kaldi configure script. As you have gaming cards,
> nvidia-smi won't probably display GPU utilization. You can have a look at
> the temperature of the cards to verify the GPUs are used.
> Expect ~ 4 times longer time for the ensemble training than for the
> "normal" training.
> y.
>
> On Wed, Jul 15, 2015 at 5:10 AM, Cemil Demir <cem...@gm...>
> wrote:
>
>> Hi,
>>
>> I have a workstation which have 4 NVIDIA TITAN X GPU card.
>>
>> I want to train a DNN model using egs/babel/s5c recipe. I am using
>> "run-2a-nnet-ensemble-gpu.sh" script.
>>
>> When I use "run-2a-nnet-gpu.sh" script, the accuracy is below as compared
>> to sgmm-mmi case. Therefore I
>>
>> Total amount of training data is 13 hours.
>> want to use "ensemble" version.
>>
>> Training takes about 35 hours. I think, it is too slow.
>>
>> Could give me any suggestion to speed up training?
>>
>> İt is a little bit urgent for me.
>>
>> Thank you.
>>
>> --
>> Cemil Demir
>>
>> cem...@gm...
>>
>>
>> ------------------------------------------------------------------------------
>> Don't Limit Your Business. Reach for the Cloud.
>> GigeNET's Cloud Solutions provide you with the tools and support that
>> you need to offload your IT needs and focus on growing your business.
>> Configured For All Businesses. Start Your Cloud Today.
>> https://www.gigenetcloud.com/
>> _______________________________________________
>> Kaldi-developers mailing list
>> Kal...@li...
>> https://lists.sourceforge.net/lists/listinfo/kaldi-developers
>>
>>
>


-- 
Cemil Demir

cem...@gm...