Change pocketsphinx from unity desktop to Android

[Danai Kor]

Hello everyone,

I have a Unity game that uses Pocketshinx and I want to export it for Android. I managed to export it but the speech recognition doesn't work as I assume there are different settings involved. Could anyone help please?

[Nickolay V. Shmyrev]

Sure, as soon as you provide additional details - pocketsphinx logs, configuration, model, audio recordings. Describe what you mean by "doesn't work" exactly.

[Danai Kor]

So do you need the .dll files and the libraries? What do you mean by models?

[Nickolay V. Shmyrev]

You can learn about models used for recognition from our tutorial

http://cmusphinx.sourceforge.net/wiki/tutorial

[Danai Kor]

Thank you!

[Danai Kor]

Great! Thnx for the quick reply! Right, so first of I'm applying reverse engineering on an app that uses Pocketsphinx with a custom plugin for Unity. When I export the project on Android it kind of skips the speech recognition part as if doesn't detect an input probably. Sorry if I don't make sence but I'm kind of new on this and i cannot detect the problem exactly. I hope those details help, Ill try to gather the rest as well. This is the error that i get on Unity when I run it on the Editor if it is relevant:

NullReferenceException: Object reference not set to an instance of an object
Master.StartSpeech (System.String questionGrammar, Int32 speechTimeout) (at assets/Scripts/AgentScripts/Master.cs:610)
Master+<WaitSpeechAnswer>c__IteratorD.MoveNext () (at assets/Scripts/AgentScripts/Master.cs:647)
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
<MainLoop>c__IteratorC:MoveNext() (at assets/Scripts/AgentScripts/Master.cs:265)

and it reffers to those parts of the code:

// -- START OF SHOP KEEPER SESSION -- //
        while (Scene.ItemNumber <= 4) {
            scoreAndInventory.SetPoints(1000);
            fader.StartFade(-1, 0.5f);
            yield return new WaitForSeconds(1.0f);
            messaging.AgentsSend(202, "", "shopkeeper"); // What can I get you? (variations)
            yield return StartCoroutine(messaging.AgentReceive()); 
            recAttempt = 0;
            bool success = false;
            bool roundAgain = false;
            do {
                if (recAttempt == 0) {
                    if(Scene.productGroupB) 
                        yield return StartCoroutine(WaitSpeechAnswer("HelpYouInitialB", 5));
                    else 
                        yield return StartCoroutine(WaitSpeechAnswer("HelpYouInitial", 5));             
                } else {
                    if(Scene.productGroupB) 
                        yield return StartCoroutine(WaitSpeechAnswer("HelpYouSubsB", 5));
                    else
                        yield return StartCoroutine(WaitSpeechAnswer("HelpYouSubs", 5));
                }





 // Enables background speech recognition for the specified grammar.
    // Also increments the recognition attempt counter.
    private void StartSpeech(string questionGrammar, int speechTimeout) {
        recAttempt++;
        Scene.Trace("SPEECH RECOGNITION: " + questionGrammar + ", ATTEMPT " + recAttempt + ", TIMEOUT " + speechTimeout + "s");
        if ((Scene.defaultUtterance) &&
            (questionGrammar.Contains("HelpYouInitial") || (questionGrammar.Contains("HelpYouSubs")))) {
            answerStatus = 2;
            currentUtterance = "EGGS BEEF WASHING CHOCOLATE CORNFLAKES BUTTER MUSTARD FLOOR CLEANER";
        } else if ((Scene.defaultUtterance) &&
                   ((questionGrammar == "YesNoItem") || (questionGrammar == "ReadyInitial") || (questionGrammar == "ReadyTakeThat") || (questionGrammar == "ReadyMoreTime"))) {
            answerStatus = 2;
            currentUtterance = "YES NO";
        } else if ((Scene.defaultUtterance) && (questionGrammar == "CoinRecog")) {
            answerStatus = 2;
            currentUtterance = "PENNY THRUPPENCE SIXPENCE SHILLING FLORIN";
        } else if ((Scene.defaultUtterance) && ((questionGrammar == "CoinValue") || (questionGrammar == "CoinValue1"))) {
            answerStatus = 2;
            currentUtterance = "THREE PENCE TWELVE PENCE TWENTY FOUR PENCE TWO SHILLINGS";
        } else {
            answerStatus = -1;
            currentUtterance = "";
            speechEngine.StartRecognition(questionGrammar, (float)speechTimeout);
        }
    }


// Enables speech recognition for the specified grammar.
    // Also increments the recognition attempt counter.
    private IEnumerator WaitSpeechAnswer(string questionGrammar, int speechTimeout) {
        StartSpeech(questionGrammar, speechTimeout);
        yield return new WaitForSeconds(0.2f);
        while (!GetSpeechAnswer()) {
            yield return new WaitForSeconds(0.2f);
        }
    }
}

[Danai Kor]

Hi again, with some help froma friend I found out what is going on.
So in the project I have a plugin CCIRPocketSphinx.DLL, which is custom code not written by me compiled for Windows as a bridge to PocketSphinx.

Also, in the root of the project, Pocketsphinx.DLL and Sphinxbase.DLL - these are the Windows DLLs for PocketSphinx and come with the SDK. For Android, they need to be .a or .so files, compiled for that platform.

and this piece of code

#if !UNITY_IOS && !UNITY_ANDROID
  try {
            speechEngine = new SpeechRecognition(new SpeechRecognition.LoggerDelegate(Scene.Warning));
            if (!speechEngine.OpenSpeechRec(Application.dataPath, SpeechPackage, Scene.participantID)) {
                speechEngine = null;
            }
        } catch (System.DllNotFoundException e) {
            Scene.Warning("Speech recognition disabled: " + e);
        }
#endif
        if (Scene.cutToKeeper) {
            Scene.showTutorial = false;
        }
    }

is in there, to stop the DLLs being loaded as they wouldn't be found for the Android platform.

What I need basically is some help with the files that need to be .a or .so files and compiled for Android. Is there anyone who can help with that?

[Nickolay V. Shmyrev]

Native code for android is usually compiled with NDK, you can find some details here:

http://developer.android.com/intl/ru/tools/sdk/ndk/index.html

pocketsphinx NDK setup is available in our sources in pocketsphinx-android package, see for details

http://github.com/cmusphinx/pocketsphinx-android

http://cmusphinx.sourceforge.net/wiki/tutorialandroid

Beside standard android code you might include some parts of the wrapper you've already constructed into the build.

[Nishant Anindya]

Hey Danai Kor,
Can you provide the source code for the module where you used speech recognition part.
looking forward to hear from you soon.
Thanks,
Nishant