I developed a plugin that introduces IR-tracking capabilities
similar to Freetrack to FaceTrackNoIR
and would be interested in your opinion.
The motivation was to combine the tracking accuracy of Freetrack
(I sometimes had problems with the face recognition of FaceAPI)
with the stability of FTNoIR
(I often experienced crashes when using Freetrack)
The sources are available in the SVN repository of FTNoIR.
For detailed build instructions and help on the plugin settings, see here: http://ftnoirpt.sourceforge.net/
For those of you who do not know how to compile the sources,
I made an installer which includes a current development version of FTNoIR stripped from FaceAPI
and the new PointTracker plugin, which can be found here: https://sourceforge.net/projects/ftnoirpt/files/
Don't hesitate to post found bugs and feature suggestions here.
Happy tracking…
Patrick
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think I can lend a hand if I can find my homemade 3-point IR device, I already have the batteries recharging for it. One question though, Can I use my current UNmodified webcam, or do I need to still use the old FreeTrack modified webcam I have stored around here somewhere? Once I get it all working, mind you it has been a few years since last use, I'll get back with you on whether or not I will be able to test. Also, I had just been to the FreeTrack forum a week or two ago, and I may be able to get one of those users to try if you'd like. I'll hopefully be back later this afternoon to say whether or not I got it working. THANKS for the work you've done so far Patrick!
Scott
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well, I think I answered my first question about whether or not to use the old webcam (that would be a yes), sometimes I need to think things through a little more out loud, Hehehe ( I think I'm getting old, just about hit my mid-40's this past month).
Scott
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Scott,
yes, the freetrack forum seems to be a good place, posted it there too.
And yes, the point extraction algorithm is as dumb as freetrack's (simple threshold) at the moment,
so you typically need a modified camera.
There is definitely some room for improvement (like only searching in a window around the old positions…)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello Patrick, tonight I was finally able to download newer drivers for the old webcam, which helped tremendously. I was finally able to get the 3-points tracking consistantly with threashold and also settings inside the webcam software tuned correctly. Now I will need time to get the movements nice and smooth and a quicker response than what I have them at at the moment.
Scott
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
@sthalik: I'm not too deep into image processing but I do not see how to use optical flow here. You mean calculate the optical flow around the last extracted points and then move the position accordingly? Wonder if that is more accurate and
if the points will not diverge over time?
@scott:
I recommend using the accela-filter.
The accela-filter included in the beta package is 10 times more sensitive than the standard accela filter,
(to get faster response) so for example the point (1,1) would correspond to (1,10) under standard accela.
I have the following points with the more sensitive accela:
(0.1, 0), (2, 0.8), (3,3), (3.5,6)
@flyforever: How did you install it? The plugin in the binaray package is not compatible with pre-built versions of FTNoIR because it uses different Qt versions. Either you have to install the full test version from https://sourceforge.net/projects/ftnoirpt/files/ or build it yourself. (sorry, thats because of the beta stage)
I also tested it with WinXP 32bit SP3, in fact thats the only configuration I can guarantee it should work. ;-)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yow, your Accela settings are insane! But it's different than being 10x more sensitive. It's like, Accela slows down normally when reaching the destination. By using a different magic value, it'll move… differently.
In the HT code (github repo above), there's much, much left drift than in SM code. But beware, it has shortcomings too, for instance the mesh can drift off the face after intense movement. I don't know how to prevent that. SM code merely jitters around on 'extreme' positions, mine, gets lost completely.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Got it working, but a few questions:
Camera: normal or without IR filter?
I am currently using a PS3 eye with filter removed and it allows only 30 fps mode. Also, it's more sensitive to light. Difficult to maintain 3 points at all times.
FSX: I've tried all protocols and yet no response in FSX. What am I missing?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm using a ps3eye with IR filter removed and with an old floppy disk as daylight filter.
Then adjust exposure and gain in CL-Eye Test.
As I said, the point extraction is exactly as picky as freetrack at the moment,
so if your setup works with freetrack, it should also work with the plugin.
For FSX you need the simconnect protocol, which is not yet included in the package, I will update that today.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
FSX SDK is not freely available? Does M$ even want money from people trying to make FSX better?
Ah, I hate companies that think they must always make their own 'standards' and people are dumb enough to pay for using them! Switch to x-plane! ;-)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello again Patrick, so does that mean you will or you won't include the simconnect protocol? Just wondering since FSX & Prepar3D are 2 other programs I can be able to test in. Right now I'm just testing in rFactor2 and then in rFactor once I get my settings more pleasing.
Scott
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I developed a plugin that introduces IR-tracking capabilities
similar to Freetrack to FaceTrackNoIR
and would be interested in your opinion.
The motivation was to combine the tracking accuracy of Freetrack
(I sometimes had problems with the face recognition of FaceAPI)
with the stability of FTNoIR
(I often experienced crashes when using Freetrack)
The sources are available in the SVN repository of FTNoIR.
For detailed build instructions and help on the plugin settings, see here:
http://ftnoirpt.sourceforge.net/
For those of you who do not know how to compile the sources,
I made an installer which includes a current development version of FTNoIR stripped from FaceAPI
and the new PointTracker plugin, which can be found here:
https://sourceforge.net/projects/ftnoirpt/files/
Don't hesitate to post found bugs and feature suggestions here.
Happy tracking…
Patrick
I think I can lend a hand if I can find my homemade 3-point IR device, I already have the batteries recharging for it. One question though, Can I use my current UNmodified webcam, or do I need to still use the old FreeTrack modified webcam I have stored around here somewhere? Once I get it all working, mind you it has been a few years since last use, I'll get back with you on whether or not I will be able to test. Also, I had just been to the FreeTrack forum a week or two ago, and I may be able to get one of those users to try if you'd like. I'll hopefully be back later this afternoon to say whether or not I got it working. THANKS for the work you've done so far Patrick!
Scott
Well, I think I answered my first question about whether or not to use the old webcam (that would be a yes), sometimes I need to think things through a little more out loud, Hehehe ( I think I'm getting old, just about hit my mid-40's this past month).
Scott
Hi Scott,
yes, the freetrack forum seems to be a good place, posted it there too.
And yes, the point extraction algorithm is as dumb as freetrack's (simple threshold) at the moment,
so you typically need a modified camera.
There is definitely some room for improvement (like only searching in a window around the old positions…)
c14,
How about instead of searching for points every frame, apply pyramid Lukas-Kanade on subsequent frames?
Hello Patrick, tonight I was finally able to download newer drivers for the old webcam, which helped tremendously. I was finally able to get the 3-points tracking consistantly with threashold and also settings inside the webcam software tuned correctly. Now I will need time to get the movements nice and smooth and a quicker response than what I have them at at the moment.
Scott
after installing the plug in , FTNOIR fails to open. Win XP 32bit. FTNOIR opens fine without plug in.
tc
Never mind. The plug in is complete with FTNOIR, so uninstall FTNOIR before installing the plug in.
@sthalik: I'm not too deep into image processing but I do not see how to use optical flow here. You mean calculate the optical flow around the last extracted points and then move the position accordingly? Wonder if that is more accurate and
if the points will not diverge over time?
@scott:
I recommend using the accela-filter.
The accela-filter included in the beta package is 10 times more sensitive than the standard accela filter,
(to get faster response) so for example the point (1,1) would correspond to (1,10) under standard accela.
I have the following points with the more sensitive accela:
(0.1, 0), (2, 0.8), (3,3), (3.5,6)
@flyforever: How did you install it? The plugin in the binaray package is not compatible with pre-built versions of FTNoIR because it uses different Qt versions. Either you have to install the full test version from https://sourceforge.net/projects/ftnoirpt/files/ or build it yourself. (sorry, thats because of the beta stage)
I also tested it with WinXP 32bit SP3, in fact thats the only configuration I can guarantee it should work. ;-)
@c14, you can calculate reprojection error. Also, grab many points and check their min eigenvals.
https://github.com/sthalik/headtracker/blob/master/headtracker-lib/features.cpp for face tracking code that you can adapt.
Yow, your Accela settings are insane! But it's different than being 10x more sensitive. It's like, Accela slows down normally when reaching the destination. By using a different magic value, it'll move… differently.
In the HT code (github repo above), there's much, much left drift than in SM code. But beware, it has shortcomings too, for instance the mesh can drift off the face after intense movement. I don't know how to prevent that. SM code merely jitters around on 'extreme' positions, mine, gets lost completely.
Double post time!
Did you equalize the image's histogram when grabbing points? Also if points are e.g. red you can extract one channel only for greater precision.
Got it working, but a few questions:
Camera: normal or without IR filter?
I am currently using a PS3 eye with filter removed and it allows only 30 fps mode. Also, it's more sensitive to light. Difficult to maintain 3 points at all times.
FSX: I've tried all protocols and yet no response in FSX. What am I missing?
I'm using a ps3eye with IR filter removed and with an old floppy disk as daylight filter.
Then adjust exposure and gain in CL-Eye Test.
As I said, the point extraction is exactly as picky as freetrack at the moment,
so if your setup works with freetrack, it should also work with the plugin.
For FSX you need the simconnect protocol, which is not yet included in the package, I will update that today.
I'll wait for the simconnect protocol, since my test rig will be FSX.
tc
FSX SDK is not freely available? Does M$ even want money from people trying to make FSX better?
Ah, I hate companies that think they must always make their own 'standards' and people are dumb enough to pay for using them! Switch to x-plane! ;-)
Hello again Patrick, so does that mean you will or you won't include the simconnect protocol? Just wondering since FSX & Prepar3D are 2 other programs I can be able to test in. Right now I'm just testing in rFactor2 and then in rFactor once I get my settings more pleasing.
Scott
Sorry, I can't include it right now since I don't have access to the FSX SDK.
I'll ask Wim to make a beta build for you.
I'll wait to test until the simconnect protocol is available….
Ok, since I found a copy of FSX SDK, the protocol is now included. However, I could not test it. (do not own a copy of FSX)
You can send it to me and I'll test it with FSX.
Hello flyforever, just wanted you to know it's available in the same download location as before **HERE**.
Works in Prepar3D, but haven't checked FSX yet.
Scott
Sorry Patrick, forgot to thank you for entering it into this build. THANKS!
I can confirm that the latest build works great in FSX and Win 7 64. Now let's do some testing….
Thanks for a great job…..
tc
I get the following bug, if it is a bug?! is this normal?
Scott
No, it's a bug in the FunctionConfigurator…
Thanks for reporting it :-)