usually each neural net (NN) layer has a fully connecting weight matrix. However, I like to force some weights (just in the first NN-layer) to be zero. Is there any way in Kaldi to initialize specific weights with zeros and to avoid learning of them? Maybe the BlockAffineComponentPreconditioned can do the job if I arrange my input features to blocks that should be connected? (or maybe I have a wrong idea about the BlockAffineComponent!?)
Best,
niko
Last edit: Morino 2015-07-09
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, if you want a block structure then
BlockAffineComponentPreconditioned would probably work.
Personally, I have not had much success with methods of this type.
Dan
usually each neural net (NN) layer has fully connected weights. However, I
like to force some weights in the first NN-layer to be zero. I.e., is there
any way in Kaldi that specific weights are not learned and initialized with
zeros? Maybe the BlockAffineComponentPreconditioned can solve my problem by
building blocks in the features? (or maybe I have a wrong idea about the
BlockAffineComponent!?)
A better ASR score is not my primary goal here. I just like to investigate learned weights to get more insight on what information a NN extracts. Thereto I try to force the net to extract specific cues separately, so that the result will be more easy to interpret.
However, I can see two parameters in the nnet2/train_block.sh script that supposedly control the blocks, i.e., block_size and block_shift. For my understanding: Can these two parameters be understood similar to a block processing known from the STFT, for example (with the feature vector being the signal)? What if the last coefficients of a feature vector doesn't fit into an entire block_size anymore? (Does it repeat some features or reduces the size of the last block?)
Sorry for that many questions and thank you for your feedback. :)
Niko
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry I don't recall the exact meaning and don't have time to check,
you'll have to look at the code.
However if you think you'll be able to see something interesting in
the weights, I think you may be disappointed.
Dan
A better ASR score is not my primary goal here. I just like to investigate
learned weights to get more insight on what information a NN extracts.
Thereto I try to force the net to extract specific cues separately, so that
the result will be more easy to interpret.
However, I can see two parameters in the nnet2/train_block.sh script that
supposedly control the blocks, i.e., block_size and block_shift. For my
understanding: Can these two parameters be understood similar to a block
processing known from the STFT, for example (with the feature vector being
the signal)? What if the last coefficients of a feature vector doesn't fit
into an entire block_size anymore? (Does it repeat some features or reduces
the size of the last block?)
Sorry for that many questions and thank you for your feedback. :)
Niko
Ok, I looked to the code and found it. It is as I assumed and if the last block doesn't fit, it is just shifted so that it will.
Thanks again and best,
niko
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
usually each neural net (NN) layer has a fully connecting weight matrix. However, I like to force some weights (just in the first NN-layer) to be zero. Is there any way in Kaldi to initialize specific weights with zeros and to avoid learning of them? Maybe the BlockAffineComponentPreconditioned can do the job if I arrange my input features to blocks that should be connected? (or maybe I have a wrong idea about the BlockAffineComponent!?)
Best,
niko
Last edit: Morino 2015-07-09
Yes, if you want a block structure then
BlockAffineComponentPreconditioned would probably work.
Personally, I have not had much success with methods of this type.
Dan
On Thu, Jul 9, 2015 at 7:17 AM, Morino mozno@users.sf.net wrote:
A better ASR score is not my primary goal here. I just like to investigate learned weights to get more insight on what information a NN extracts. Thereto I try to force the net to extract specific cues separately, so that the result will be more easy to interpret.
However, I can see two parameters in the nnet2/train_block.sh script that supposedly control the blocks, i.e., block_size and block_shift. For my understanding: Can these two parameters be understood similar to a block processing known from the STFT, for example (with the feature vector being the signal)? What if the last coefficients of a feature vector doesn't fit into an entire block_size anymore? (Does it repeat some features or reduces the size of the last block?)
Sorry for that many questions and thank you for your feedback. :)
Niko
Sorry I don't recall the exact meaning and don't have time to check,
you'll have to look at the code.
However if you think you'll be able to see something interesting in
the weights, I think you may be disappointed.
Dan
On Thu, Jul 9, 2015 at 10:08 AM, Morino mozno@users.sf.net wrote:
Ok, I looked to the code and found it. It is as I assumed and if the last block doesn't fit, it is just shifted so that it will.
Thanks again and best,
niko
Hi Dan,
I found a small bug in the steps/nnet2/get_lda_block.sh script. Here is a bug fix:
The old version may miss the last block with the consequence that higher indexes of a feature vector will be unused.
Best,
niko
Last edit: Morino 2015-07-14
Thanks, committing the fix.
Dan
On Tue, Jul 14, 2015 at 5:35 AM, Morino mozno@users.sf.net wrote: