From: Daniel P. <dp...@gm...> - 2015-07-15 17:47:19
|
Also, make sure the different jobs are using different GPUs. Setting to exclusive mode (nvidia-smi -c 1) will help. Dan On Wed, Jul 15, 2015 at 9:04 AM, Jan Trmal <jt...@gm...> wrote: > Are you sure you are using CUDA? You have to enable the support manually > when running the kaldi configure script. As you have gaming cards, > nvidia-smi won't probably display GPU utilization. You can have a look at > the temperature of the cards to verify the GPUs are used. > Expect ~ 4 times longer time for the ensemble training than for the "normal" > training. > y. > > On Wed, Jul 15, 2015 at 5:10 AM, Cemil Demir <cem...@gm...> wrote: >> >> Hi, >> >> I have a workstation which have 4 NVIDIA TITAN X GPU card. >> >> I want to train a DNN model using egs/babel/s5c recipe. I am using >> "run-2a-nnet-ensemble-gpu.sh" script. >> >> When I use "run-2a-nnet-gpu.sh" script, the accuracy is below as compared >> to sgmm-mmi case. Therefore I >> >> Total amount of training data is 13 hours. >> want to use "ensemble" version. >> >> Training takes about 35 hours. I think, it is too slow. >> >> Could give me any suggestion to speed up training? >> >> İt is a little bit urgent for me. >> >> Thank you. >> >> -- >> Cemil Demir >> >> cem...@gm... >> >> >> ------------------------------------------------------------------------------ >> Don't Limit Your Business. Reach for the Cloud. >> GigeNET's Cloud Solutions provide you with the tools and support that >> you need to offload your IT needs and focus on growing your business. >> Configured For All Businesses. Start Your Cloud Today. >> https://www.gigenetcloud.com/ >> _______________________________________________ >> Kaldi-developers mailing list >> Kal...@li... >> https://lists.sourceforge.net/lists/listinfo/kaldi-developers >> > > > ------------------------------------------------------------------------------ > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > _______________________________________________ > Kaldi-developers mailing list > Kal...@li... > https://lists.sourceforge.net/lists/listinfo/kaldi-developers > |