From: Cemil D. <cem...@gm...> - 2015-07-15 09:10:49
|
Hi, I have a workstation which have 4 NVIDIA TITAN X GPU card. I want to train a DNN model using egs/babel/s5c recipe. I am using "run-2a-nnet-ensemble-gpu.sh" script. When I use "run-2a-nnet-gpu.sh" script, the accuracy is below as compared to sgmm-mmi case. Therefore I Total amount of training data is 13 hours. want to use "ensemble" version. Training takes about 35 hours. I think, it is too slow. Could give me any suggestion to speed up training? İt is a little bit urgent for me. Thank you. -- Cemil Demir cem...@gm... |
From: Jan T. <jt...@gm...> - 2015-07-15 16:04:07
|
Are you sure you are using CUDA? You have to enable the support manually when running the kaldi configure script. As you have gaming cards, nvidia-smi won't probably display GPU utilization. You can have a look at the temperature of the cards to verify the GPUs are used. Expect ~ 4 times longer time for the ensemble training than for the "normal" training. y. On Wed, Jul 15, 2015 at 5:10 AM, Cemil Demir <cem...@gm...> wrote: > Hi, > > I have a workstation which have 4 NVIDIA TITAN X GPU card. > > I want to train a DNN model using egs/babel/s5c recipe. I am using > "run-2a-nnet-ensemble-gpu.sh" script. > > When I use "run-2a-nnet-gpu.sh" script, the accuracy is below as compared > to sgmm-mmi case. Therefore I > > Total amount of training data is 13 hours. > want to use "ensemble" version. > > Training takes about 35 hours. I think, it is too slow. > > Could give me any suggestion to speed up training? > > İt is a little bit urgent for me. > > Thank you. > > -- > Cemil Demir > > cem...@gm... > > > ------------------------------------------------------------------------------ > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > _______________________________________________ > Kaldi-developers mailing list > Kal...@li... > https://lists.sourceforge.net/lists/listinfo/kaldi-developers > > |
From: Daniel P. <dp...@gm...> - 2015-07-15 17:47:19
|
Also, make sure the different jobs are using different GPUs. Setting to exclusive mode (nvidia-smi -c 1) will help. Dan On Wed, Jul 15, 2015 at 9:04 AM, Jan Trmal <jt...@gm...> wrote: > Are you sure you are using CUDA? You have to enable the support manually > when running the kaldi configure script. As you have gaming cards, > nvidia-smi won't probably display GPU utilization. You can have a look at > the temperature of the cards to verify the GPUs are used. > Expect ~ 4 times longer time for the ensemble training than for the "normal" > training. > y. > > On Wed, Jul 15, 2015 at 5:10 AM, Cemil Demir <cem...@gm...> wrote: >> >> Hi, >> >> I have a workstation which have 4 NVIDIA TITAN X GPU card. >> >> I want to train a DNN model using egs/babel/s5c recipe. I am using >> "run-2a-nnet-ensemble-gpu.sh" script. >> >> When I use "run-2a-nnet-gpu.sh" script, the accuracy is below as compared >> to sgmm-mmi case. Therefore I >> >> Total amount of training data is 13 hours. >> want to use "ensemble" version. >> >> Training takes about 35 hours. I think, it is too slow. >> >> Could give me any suggestion to speed up training? >> >> İt is a little bit urgent for me. >> >> Thank you. >> >> -- >> Cemil Demir >> >> cem...@gm... >> >> >> ------------------------------------------------------------------------------ >> Don't Limit Your Business. Reach for the Cloud. >> GigeNET's Cloud Solutions provide you with the tools and support that >> you need to offload your IT needs and focus on growing your business. >> Configured For All Businesses. Start Your Cloud Today. >> https://www.gigenetcloud.com/ >> _______________________________________________ >> Kaldi-developers mailing list >> Kal...@li... >> https://lists.sourceforge.net/lists/listinfo/kaldi-developers >> > > > ------------------------------------------------------------------------------ > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > _______________________________________________ > Kaldi-developers mailing list > Kal...@li... > https://lists.sourceforge.net/lists/listinfo/kaldi-developers > |
From: Cemil D. <cem...@gm...> - 2015-07-15 20:17:09
|
Yes, I am using CUDA. I am using nvidia-smi and GPU utilizaiton is seen. Although, TITAN X has 12 GB memory, only 300 MB of it is used by nnet trainer process. On Wed, Jul 15, 2015 at 7:04 PM, Jan Trmal <jt...@gm...> wrote: > Are you sure you are using CUDA? You have to enable the support > manually when running the kaldi configure script. As you have gaming cards, > nvidia-smi won't probably display GPU utilization. You can have a look at > the temperature of the cards to verify the GPUs are used. > Expect ~ 4 times longer time for the ensemble training than for the > "normal" training. > y. > > On Wed, Jul 15, 2015 at 5:10 AM, Cemil Demir <cem...@gm...> > wrote: > >> Hi, >> >> I have a workstation which have 4 NVIDIA TITAN X GPU card. >> >> I want to train a DNN model using egs/babel/s5c recipe. I am using >> "run-2a-nnet-ensemble-gpu.sh" script. >> >> When I use "run-2a-nnet-gpu.sh" script, the accuracy is below as compared >> to sgmm-mmi case. Therefore I >> >> Total amount of training data is 13 hours. >> want to use "ensemble" version. >> >> Training takes about 35 hours. I think, it is too slow. >> >> Could give me any suggestion to speed up training? >> >> İt is a little bit urgent for me. >> >> Thank you. >> >> -- >> Cemil Demir >> >> cem...@gm... >> >> >> ------------------------------------------------------------------------------ >> Don't Limit Your Business. Reach for the Cloud. >> GigeNET's Cloud Solutions provide you with the tools and support that >> you need to offload your IT needs and focus on growing your business. >> Configured For All Businesses. Start Your Cloud Today. >> https://www.gigenetcloud.com/ >> _______________________________________________ >> Kaldi-developers mailing list >> Kal...@li... >> https://lists.sourceforge.net/lists/listinfo/kaldi-developers >> >> > -- Cemil Demir cem...@gm... |