I've started work on implementing EM together with, or rather as an extension to the EKF FFN algorithm. If anybody has any good references or documentation on this, please let me know.
It does not look to difficult to do on top of EKF FFN, or even on top of EKF RNN, but seems it will require some fancy footwork. I am not really planning to do this in joone, but rather in july - the lightweight version of joone that is discussed here somewhere. Given the matrix math, I also plan to use the colt library which assists in the two EKF algorithms.
Any comments and references welcome.
just out of curiosity.
what's the intention? the only application i can think of is parameter estimation for a hidden markov model ...
but is there a special reason for doing that in the context of neural networks? i mean wouldn't it be simpler to use an existing stats-package (quite sure there are some r-packages) and to write a java interface. don't get me wrong. imho it's freaking awesome you're doing this in java and i'm looking forward to use it ;-).
Application is training and the way I understand this is that EM is an extension of EKF - so a natural next step is to bolt EM on top of EKF to optimise weights.
I've done the research and have the papers for EM, but am not sure I even want to try.
On a decent sized, simple FFN network, vanilla EKF takes forever and EM is going to slow it down even more. Seems a nice theoretical device or good for small networks where it at least can chew on a few thousand iterations, or I am in serious need of new hardware.....
Coming to think of it, I've seen more than one paper where the author uses something like backprop or Levenberg-Marquardt to train a net and then use EKF only to update it with new patterns. While it seems powerful, it seems too slow to do a complete network training with - and this again referring just to vanilla EKF!
Long time no see, I'm back
What is the status and where can I help with the development?
I'll do this at some stage, but it is low on the list. EKF FFN is more or less useless to me, since it takes forever to train a decent sized network. Since EM - or should I say EKF EM - will slow it down even further, I think its use is limited to really small networks.
But I've done the research and will get around to it some day.