Pocketsphinx Android French Phoneme Recognition

[Guillaume L]

Hi there !

I'm working on a project about Cued Speech (it's a visual system of communication used with and among deaf or hard-of-hearing people). It is a phonemic-based system which makes traditionally spoken languages accessible by using a small number of handshapes, known as cues, (representing consonants) in different locations near the mouth (representing vowels), as a supplement to speechreading.

I already followed the tutorial about phonemes recognition (pocketsphinx_continuous), pocketsphinx on android and it's working.
I'm working for now on Android but the final goal of this project is to make it work on Google glass.
The best case is to write on the device the speech (closer as real time) and an avatar in 3D which will reproduce the correct hand configuration and lips mouvement according to the phoneme identified. (That's why i need to work with phonemes and not the word it self)

I read http://cmusphinx.sourceforge.net/wiki/phonemerecognition but i'm quite confuse about how to create my own "files" for french phonemes to make it works on Android.

What are the steps ? What kind of file should i create ?

In other words, is it possible to have the same result of pocketsphinx_continuous with phonemes on Android with voice input ?

I hope you can guide me !

Guillaume

[Nickolay V. Shmyrev]

I answered you here:

http://stackoverflow.com/questions/27190486/how-to-add-phonemes-recognition-with-pocketsphinx-on-android/27208699#27208699

Basically you need to checkout latest demo and replace English models with French

[Guillaume L]

Hi,

I have some issues ='(
I did like you say :

    File modelsDir = new File(assetsDir, "models");
    recognizer = defaultSetup()
            .setAcousticModel(new File(modelsDir, "hmm/lium_french_f0"))
            .setBoolean("-allphone_ci", true)
            .getRecognizer();
    recognizer.addListener(this);   
  // Phonetic search
  File phoneticModel = new File(modelsDir, "phone/fr-phone.lm.dmp");
  recognizer.addAllphoneSearch(PHONE_SEARCH, phoneticModel);

But it always give me the same result and i'm not even saying something :
(In logs) 12-15 15:55:40.197: E/TEXT(15523): SIL ei +mus+ ei

I tried with the english version but ended in the same way =/
I don't know what i did wrong because it doesn't look hard to implement

Here is my github if you want to see all the src
https://github.com/lefevrG/SpeechToCode

Guillaume

[Nickolay V. Shmyrev]

Hello Guillaume

Please add setRawLogDir to log audio files to sdcard. Please share raw files.

Please share logcat output.

[Guillaume L]

Hello,

You'll find all you want at :

https://github.com/lefevrG/SpeechToCode/tree/master/raw

I've tried to understand how all of this work but i wasn't able to figure it out :/

[Guillaume L]

Any update ='( ?

It's been a week i'm trying whatever i can in the code (i think i'm today able to rewrite every class from memory) to make phoneme recognition work but in the end i have the same result :
- When i launch the demo, i get on phoneme recognition (that's what i want so it's OK)
- I say something like "Bonjour" (btw the result looks like a litte bit wrong)
- Then in onPartialResults, i always get the phonemes from "Bonjour" what ever i can say
after this (mostly have "+b+", what does this mean ? any doc somewhere ?)

I have fr-phone.lm.dmp in phone folder, french acoustic model in hmm folder.
I really have no idea what i am doing wrong and it's blowing my mind ><

Hope someone could figure this out,

Guillaume

[Nickolay V. Shmyrev]

I checked your raw files, they seem to be pretty noisy. French model is too big for mobile and too inaccurate for noisy recognition so it's not easy to expect something good. You might try parameters of recognizer setup to approach accuracy like this:

            .setBoolean("-allphone_ci", true)
            .setFloat("-lw", 2.0)
            .setFloat("-beam", 1e-20)
            .setFloat("-pbeam", 1e-20)
            .setFloat("-vad_threshold", 3.0)
            .setBoolean("-remove_noise", false)

but overall you need to perform accuracy testing in realistic condition on desktop first with pocketsphinx_batch and see what is the real phonetic decoding accuracy. If it is below expectation you might want to focus on the acoustic model first, it's unlikely current model would work well.

[Nickolay V. Shmyrev]

you need to perform accuracy testing in realistic condition on desktop first with pocketsphinx_batch and see what is the real phonetic decoding accuracy.

I.e. you need to record samples with android application (raw files), create reference for them, then decode them on desktop to see what is the accuracy of phonetic recognition.

[Rahul Sharma]

Hi,
I want to implement offline voice recognition on google glass 2.I had tried using PocketSphinx on android mobile and its working fine but when i run the same app on google glass it is not responding to voice.Any help would be appreciated

[Nickolay V. Shmyrev]

Google glass is probably not powerful enough. You can check logcat for details.

[ayda]

do u mind sharing the code its actualy not working for me