Re: [Kaldi-users] RBM - issue with anti-weight explosion protection

[Mailing l. u. f. U. C. a. U. <kal...@li...>]

Hi Teo,
In speech recognition, we use gaussian units on the DNN input,
because it reflects the nature of the input features.

In your case, when you use the pixel values, it is better to use 
Bernoulli units
on the DNN front-end. This you can do by modifying the training script
in a way that you remove special handling of the 1st RBM, which will 
effectively
make it use Bernoulli visible units.

Alternatively you can use PCA on your data, which will make it more 
'Gaussian'.
With regards,
Karel


Dne 19.7.2013 12:33, Mailing list used for User Communication and 
Updates napsal(a):
> Hi,
>
> I ran into some issue regarding rbm training. My features are pixels values (I work on handwriting recognition), and some of them are often zero (so on a subset of data, the stdev might be zero..).
> I applied an adaptation of wsj/s5 recipe with Karel's method to train DNN and got this :
>
> LOG (rbm-train-cd1-frmshuff:main():rbm-train-cd1-frmshuff.cc:140) RBM TRAINING STARTED
> LOG (rbm-train-cd1-frmshuff:main():rbm-train-cd1-frmshuff.cc:143) Iteration 1/2
> VLOG[1] (rbm-train-cd1-frmshuff:main():rbm-train-cd1-frmshuff.cc:243) Setting momentum 0.5 and learning rate 0.005 after processing 0.000277778h
> ERROR (rbm-train-cd1-frmshuff:ApplyPow():kaldi-vector.cc:421) Cannot take square root of negative value -3.72529e-09
> ERROR (copy-feats:Write():kaldi-matrix.cc:962) Failed to write matrix to stream
>
> I reckon we can say   -3.72529e-09 is actually zero (?).
>
> Anyways, I added Gaussian noise to my features and that solved the problem, but maybe we don't need a precision of 1e-8 for this check in RbmUpdate. Moreover, for the neg_vis stdev, there is a check forcing it to zero if negative.
>
> Cheers,
>
> téo
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Kaldi-users mailing list
> Kal...@li...
> https://lists.sourceforge.net/lists/listinfo/kaldi-users