Inertial Head Tracker Development

[dermorat]

Hi all.

I just came across a video of RC guys using a cheap arduino nano board with
9 axis inertial sensor to control a gimbal for a camera.
This looks promising for game head tracking.
If it could just be as easy as flashing the firmware and having it
connected to usb for power and data, the question just remains if it is
possible to connect it to Facetrack?

I am not a coder myself so I have no idea about the work involved.
But could Facetrack interface with the headtrack software already developed?

The video https://www.youtube.com/watch?v=Q9Av10UxBEI

Arduino Nano BLE https://store.arduino.cc/arduino-nano-33-ble

Headtrack software https://headtracker.gitbook.io/head-tracker/

[Wim Vriend]

Hello dermorat,

Thanks for your interesting links: it is good to see that creating an IMU tracker is becoming more easy :).

Indeed, FaceTrackNoIR can receive data from the Arduino (or any other IMU hardware). The HAT tracker was especially created for this purpose. It can, for example, be used to receive serial data from a Bluetooth dongle. This was tested by myself and the great thing is, that it is wireless.

If you would use a wireless radio module on the Arduino, then you could also send UDP messages and the OSC tracker would be a good choice.

There is a downside: you would have to program the Arduino to send the data to FaceTrackNoIR. I have not yet studied the software, but I did not yet see examples of sending data...

[Warlock SRB]

...well, I tried, maybe I messed something but for some reason, HeadTrackNoIR doesn't receive any data niether via USB nor via BT. Have same Nano 33 BLE board.

[Wim Vriend]

Hello Ranko,

Your Nano program should (of course) send the data. Can you share the code you use to do that?
Do you have a USB bluetooth device, to receive the data on your PC-side?

[Warlock SRB]

...here is last simple example that I've wrote just to get it working, this one is over USB (COM), also tried opentrack with same plugin (although there is v2.1.1).

[Wim Vriend]

And what kind of device are you using to receive the data? An USB serial device?

If you want to use the HAT tracker, the data should be sent in the format:

// Arduino frame structure
#pragma pack(push,2)
struct TArduinoData
{
    quint16 Begin;         // Header trame 0xAAAA;
    quint16 Code;         // 0-999 = TRACKING (+frame counter), 1000-1999 = CALIBRATING (+frame counter)
    float Rot[3];
    float Trans[3];
    quint16 End;            // End trame 0x5555;
} ;
#pragma pack(pop) 

But you probably know that already ;)

[Warlock SRB]

Yeah I know about that... I as said currently using USB, but have Asus USB-BT400

[Wim Vriend]

Maybe there is some kind of utility program that let's you monitor all incoming serial data?

[Warlock SRB]

...what if I wanna send all data, like yaw, pitch, roll and x, y and z. Does code change then?

For some reason, data is sent but headtracking doesn't work... not sure what am I doing wrong... bu as I said in About section, it reads data...

[Warlock SRB]

Code for Arduino looks like this:

at the begining of ino

struct TArduinoData
{
    uint16_t Begin;         // Header trame 0xAAAA;
    uint16_t Code;         // 0-999 = TRACKING (+frame counter), 1000-1999 = CALIBRATING (+frame counter)
    float Rot[3];
    float Trans[3];
    uint16_t End;            // End trame 0x5555;
};

and for sending in void loop()

// Store the data in the struct
  arduinoData.Begin = 0xAAAA;
  arduinoData.Code = 0;  // or 1000 if in calibrating mode
  arduinoData.Rot[0] = heading;
  arduinoData.Rot[1] = pitch;
  arduinoData.Rot[2] = roll;
  arduinoData.Trans[0] = 0.0;
  arduinoData.Trans[1] = 0.0;
  arduinoData.Trans[2] = 0.0;
  arduinoData.End = 0x5555;

  // Send the data as a string over USB
  char buffer[50];
  sprintf(buffer, "%f %f %f\n", heading, pitch, roll);
  Serial.print(buffer);

...maybe my port was interfearing with something, although there was update this morning for Windows10.

Currently I'm getting in About section readout of coordinates, but they don't affect tracking and under Raw Input there are only zeroes

[Bhargu]

Did you resolve it?

I am writing arduinoData, which is an instance of TArduinoData to the Serial as below. The frame is 30 byte sized, as shown in the documentation or if you add the bytes of the structure.

Serial.write((byte*)&arduinoData, 30);

Please refer https://facetracknoir.sourceforge.net/Trackers/HAT.htm where sample code is also given.

[Bhargu]

I am having decent accuracy with just the IMU onboard the Nano 33 BLE. It is easily influenced by metal around (even your chair) and other magnetic fields. You could use improved sensor modules.

As the board has only BLE support, using a Bluetooth serial port won't be possible. I was thinking of using a small program to read from the BLE packets and create a serial port and write to it in the correct "hat" data structure format. Does anyone think this would work, or is there a better way to do this?

I am not sure about the latency of BLE communication, but wouldn't it work for the use case of head tracking in games?

[Wim Vriend]

Hello Bhargu,

I'm not familiar with BLE. However, when searching the internet I do see USB adapters which claim to support BLE. Wouldn't that suffice?

I have tested the HAT communication with a 'normal Bluetooth' adapter, which creates a COM-interface FaceTrackNoIR can use. I suppose the BLE dongle can also do that?

[Bhargu]

The normal Bluetooth / Bluetooth Classic has an profile for serial communication interface called SPP (Serial Port Profile). It emulates the COM interface. I believe that is what you used.

From what I understood, Bluetooth Low Energy (BLE) doesn't have that. It sends formatted packets. There are emulations of SPP using BLE generally called Serial over BLE. There will be a software on the receiving end which will convert that data stream into a COM port. I think Windows 10 even has something builtin for that. BLE is supposed to have better latency/delay than Bluetooth Classic, so that has piqued my interest a bit.

Anyway, I will try out both approach and see what is best. I am interested in this for more use cases than just head tracking for games, though that was the first reason for me to test this.

[Wim Vriend]

Please let us know your findings ;)