Hi, I've been looking through the forum without being able to find a clear answer on my problem.
I'm using the shape predictor with the bundled shape_predictor_68_face_landmarks.dat file, and i experience that the landmarks seems to be very jittery on my webcam, and i was wondering if this is the expected output, or I'm missing something.
Looking at the sample video , it seems to be able to run with much less jitter.
Last edit: HalfdanJ 2016-05-27
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
That example video is from a high quality HD camera. The results will be
less good with lower quality cameras. My personal webcam is also pretty
crappy, you can clearly see the pixel noise and it's also not HD, so I get
similar jitter.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I tried so many things on the past fews days,
did a lot if reading / browsing,
I am stuck.
So now I allow myself write to you.
I was using dlib landmarks with recorded video (good quality images) for a while
and it was a little jittery but managable.
Mainly reducing the size for detect, and skipping it, I managed to have it runing at 100 fps on a i5.
Now I am using a Camera (not a webcam, let say a 300$ camera with a decent lense and some lighting)
and the jitter is pretty serious, more in low light where I have to push the camera of course.
So, I tried to preprocess
(blur -> less noise, remanence, histogram equalisation ...)
didn't really solve it (blur have certain influence).
Then, to gain speed, I managed to use the detect and landmark with grey image (seem to be the same result than rgb/bgr can you confirm please ?),
So now I can run the landmarking at different resolution and I get differents results for different resolutions, and some time it is huge (e.g. the left side can jump by more than the height of an eye).
Filtering using these differents data is not an easy task.
Do you have any idea from where the landmarking instability / jitter come,
and what can be done ?
Did somebody trained it with more stable result ?
Thanks
Mâa
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
All the things you mentioned are reasonable to try. If it still isn't
working well enough on your data you will need to train a new model on the
data you are trying to use, or on similar looking data. I would try to
artificially augment the 300W training data (http://dlib.net/files/data/)
used to train the default model in some way to look like your test data.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry I tried everything
filtering the landmarks is my only solution now.
-can using the current data
but expending it by resizing each frame to different sizes
stabilize the instability ?
on the same lines
-can adding fake glasses to make new frames
will make faces with glasses better landmarked ?
-can adding again frames but with noise could stabilize the jitter ?
Infact all if this could be done in code at training time.
What the best starting place for training code ?
Thanks again
Mâa
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I manage to compile the training in x64 so I can load now the big ibug_300W database,
and I start training...
Thanks : if was pretty easy
I didn't read all the code yet, and don't understand all yet
but perhaps you can comment on this.
all the process seem to rely in integer landmark position,
all the image are pretty big,
so we are not training well for small images which would infact require sub pixel position.
Could you comment on this ?
Also is the training done on luminance or use color ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Infact the more I read the code, the more I have the feeling that using pyr_down on point coor (long)
the more we accummulate errors, especially on small image that are built at load if I understand.
If I get the result is a noisy training set, and could be the origin of the jittering.
Is it difficult to move to float all the points coordonnates or to shift up the long coor to get sub pixel precision ?
Last edit: Maa 2016-12-23
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi, I've been looking through the forum without being able to find a clear answer on my problem.
I'm using the shape predictor with the bundled shape_predictor_68_face_landmarks.dat file, and i experience that the landmarks seems to be very jittery on my webcam, and i was wondering if this is the expected output, or I'm missing something.
Looking at the sample video , it seems to be able to run with much less jitter.
Last edit: HalfdanJ 2016-05-27
That example video is from a high quality HD camera. The results will be
less good with lower quality cameras. My personal webcam is also pretty
crappy, you can clearly see the pixel noise and it's also not HD, so I get
similar jitter.
Hello Davis
I tried so many things on the past fews days,
did a lot if reading / browsing,
I am stuck.
So now I allow myself write to you.
I was using dlib landmarks with recorded video (good quality images) for a while
and it was a little jittery but managable.
Mainly reducing the size for detect, and skipping it, I managed to have it runing at 100 fps on a i5.
Now I am using a Camera (not a webcam, let say a 300$ camera with a decent lense and some lighting)
and the jitter is pretty serious, more in low light where I have to push the camera of course.
So, I tried to preprocess
(blur -> less noise, remanence, histogram equalisation ...)
didn't really solve it (blur have certain influence).
Then, to gain speed, I managed to use the detect and landmark with grey image (seem to be the same result than rgb/bgr can you confirm please ?),
So now I can run the landmarking at different resolution and I get differents results for different resolutions, and some time it is huge (e.g. the left side can jump by more than the height of an eye).
Filtering using these differents data is not an easy task.
Do you have any idea from where the landmarking instability / jitter come,
and what can be done ?
Did somebody trained it with more stable result ?
Thanks
Mâa
All the things you mentioned are reasonable to try. If it still isn't
working well enough on your data you will need to train a new model on the
data you are trying to use, or on similar looking data. I would try to
artificially augment the 300W training data (http://dlib.net/files/data/)
used to train the default model in some way to look like your test data.
Sorry I tried everything
filtering the landmarks is my only solution now.
-can using the current data
but expending it by resizing each frame to different sizes
stabilize the instability ?
on the same lines
-can adding fake glasses to make new frames
will make faces with glasses better landmarked ?
-can adding again frames but with noise could stabilize the jitter ?
Infact all if this could be done in code at training time.
What the best starting place for training code ?
Thanks again
Mâa
Sorry it is not 100 percent clear.
My questions are about extending the training but using the same training database
There is an example program that show you how to train a shape predictor:
http://dlib.net/train_shape_predictor_ex.cpp.html
Thanks
Do you have an opinion on the others questions ?
Do some experiments and see what works :)
I manage to compile the training in x64 so I can load now the big ibug_300W database,
and I start training...
Thanks : if was pretty easy
I didn't read all the code yet, and don't understand all yet
but perhaps you can comment on this.
all the process seem to rely in integer landmark position,
all the image are pretty big,
so we are not training well for small images which would infact require sub pixel position.
Could you comment on this ?
Also is the training done on luminance or use color ?
Infact the more I read the code, the more I have the feeling that using pyr_down on point coor (long)
the more we accummulate errors, especially on small image that are built at load if I understand.
If I get the result is a noisy training set, and could be the origin of the jittering.
Is it difficult to move to float all the points coordonnates or to shift up the long coor to get sub pixel precision ?
Last edit: Maa 2016-12-23
I don't know what you are talking about. The shape_predictor doesn't even
use any image pyramid tools.
Ok so I got lost in another branch, wrong template sorry.
and sorry to insist
David Please please
You should read the CVPR paper the method is based on. It explains
everything in great detail.
Did you solve the problem?