Re: [Kaldi-users] non-cluster usage of Librispeech s5 recipe?

[Jan T. <jt...@gm...>]

No, these jobs will not be retried. It's the user's responsibility to set
the number of training jobs accordingly to the number of GPU's he/she have
available for training  (--num-jobs-initial and --num-jobs-final).

About your observation with the jobs running on background. I'm not
familiar with the librispeech recipe per se, so I just can tell you a
general experience with the recipes in kaldi.
I guess that _could_ happen -- in a script, when you spawn something to run
on  background (using &) and the parent script exits (no matter if with
success or with failure), the background tasks will still run -- you could
actually list them using "ps" issued on the terminal where the original
script was executed. My feeling is that some part of the script failed,
because if that happens, exit 1 is usually called. When the script will run
successfully, there is usually "wait" at the end of the script, so the
script will wait until all child tasks finish.

hth
y.


On Fri, Jun 12, 2015 at 2:06 AM, David Warde-Farley <
d.w...@gm...> wrote:

> Hi,
>
> Apologies if this has been answered in the archives, but I'm trying to
> use the s5 recipe for LibriSpeech on a single machine with a single
> GPU. I've modified cmd.sh to use run.pl.
>
> After about a day, I see a lot of background processes like
> gmm-latgen-faster, lattice-add-penalty, lattice-scale, etc. that have
> been launched in the background (the terminal is actually free, which
> suggests the run.sh script has terminated...). I'm not totally sure
> what's going on, or how to find out.
>
> Specifically, I'm trying to export the features used to train the
> final stage neural network as well as the aligned targets.
>
> One thing I noticed earlier is that the script was trying to spawn
> multiple GPU jobs, but this GPU is configured (by administrators) to
> permit at most one CUDA process, and so I saw "3 of 4 jobs failed"
> messages. Would these jobs have been retried?
>
> Thanks in advance,
>
> David
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users
>