Initialize neural nets not trained with dlib
Brought to you by:
davisking
Menu
▾
▴
Hi,
I'd like to perform the fine tuning of, or just use, a neural net in dlib that has been pretrained with a different library (for instance with Matlab). I know everything of this net (structure, weights, biases, ...). Is it possible to initialize a network in dlib with these values, or do I necessarly have to use a net which has been saved by means of the dlib serialization function?
Thank you!
As far as I know, dlib do not support weight initialization of pretrain models of another libraries yet. I think we could try to contribute back some pretrain models back to dlib. I would try to do that if I can trained some decent model(ex : deepID2, vsdr). The more pretrained models, the more easier to attract users, the more users increase the changes of getting more models contribute back to dlib.
I want to know how to finetune the network too, by now I only know how to extract the features from pretrain network of dlib(ex : resnet34).
Last edit: thamngapwei 2016-11-19
There is no tooling in dlib to load models from other software. There might
as well be an infinite number of other formats so I'm not going to spend
time trying to keep a working conversion tool in dlib. So it's up to you
to do the conversion.
Or maybe your question is, can you just edit the weights of a dlib network
however you want. The answer is yes, certainly. Write a bit of code that
accesses the parameters of whatever layer you want and set them to any
value you want. The two dnn introduction's show how to access any part of
a network.
Hi guys,
first of all thank you for your reply.
Editing biases and weights of a dlib network is what I actually need. I succeded in getting these values by means of the function:
resizable tensor t = layer<n>(net).layer_details().get_weights()/get_biases(),
as shown in the Introduction2 example, and then I acceded the tensor values by the host() function. Nevertheless, I was not able to find in the source code either a setter function to edit them or a way to access directly the layer values. Can you please suggest me how do to that or where to look in the library source code?
Thank you very much for your support.</n>
host() returns a pointer to the weights. That's all you need to modify
them. I would also recommend reading one of the C++ books listed here
http://dlib.net/books.html since you are going to have a really hard time
with this if you aren't familiar with C++ language constructs like what
host() returns.
I think I figured out why I wasn't able to modify the weights. I used to instantiate a resizable_tensor variable instead of an alias_tensor_instance variable. In that way I always worked on a copy of the weights values.
Thank you very much for your support and for the effort you're putting into improving and mantaining dlib: it's really a great development tool!
No problem :)
also trying to finetune (do transfer learning) by copying weights of resnet34. I just need to change last fc layer size to 13. I've tried several ways of accessing layer params and copying weights via template functions. it is messy with 143 layers to copy (yes some don't have any params so skipping those, e.g. tag layers). is there an easier way to do this? like recursive using subnet() method. (tried this too but it doesn't work :( I was able to initialize a network with the same weights of resnet34 by changing the input image dimension via the copy constructor. That was very cool. But there doesn't seem to be a cool and easy way like that to change the last layer only and still initialize with resnet34 weights. Am I missing something? Also a solid example of finetuning would be highly appreciated.
You can just create the network you want and then do something like
mynewnet.subnet() = oldresnet.subnet();
yes I was missing sth. I just needed to do mynewnet.subnet().subnet() = resnet.subnet().subnet() !! thanks! :) (and also for this awesome library!) I'll continue with training now..
No problem :)
sometimes the average loss starts out with value inf and then becomes a normal looking value, or starts out with some value but then becomes inf at some point during the initial stages of training and then turns into a normal value. What does it mean exactly when average loss is inf? Does it basically destroy the gradients and so if it happens anything after that point is pretty much meaningless? thanks.
It doesn't matter. It's just part of the display, the gradients are
stable. Don't worry about it.