[Android] Bluetooth headset asi voice input for pocketsphinx

[tsn]

Hello, I am trying (for some time) to get working a bluetooth headset as an input of voice for pockesphinx (on Android) instead of internal mic.

My current state is:

in onCreate method I do this:

        if(isBluetoothHeadsetConnected()){
            setAudioManagerToHeadset();
            Log.i(TAG,"Headset is connected");
        }
        else {
            setAudioManagerDefault();
            Log.i(TAG, "Headset is disconnected");
        }

setAudioManagerToHeadset() and setAudioManagerDefault() are implemented like this:

    public void setAudioManagerDefault(){
        audioManager = (AudioManager) getSystemService(AUDIO_SERVICE);
        audioManager.stopBluetoothSco();
        audioManager.setMode(AudioManager.MODE_NORMAL);
        audioManager.setBluetoothScoOn(false);
        audioManager.setMicrophoneMute(false);
    }

    public void setAudioManagerToHeadset(){
        audioManager = (AudioManager) getSystemService(AUDIO_SERVICE);
        audioManager.startBluetoothSco();
        audioManager.setMode(AudioManager.MODE_IN_COMMUNICATION);
        audioManager.setBluetoothScoOn(true);
        audioManager.setMicrophoneMute(true);
    }

But it doesn't work.

I also tried aswer in this thread https://sourceforge.net/p/cmusphinx/discussion/help/thread/5be47108/ but it also didn't work.

I am getting this error from pocketsphinx:

E/AudioTrackShared: obtainBuffer() timeout [110]

In attachment is that java file if you needed.

Do you know where problem is? Thank you very much.

EDIT:
To be more accurate: Pocketsphinx works awesome with built-in mic, but problem is with external BT headset - when I open the app, BT headset beeps that it is prepared - SCO is on (I checked), but it seems like BT headset is muted and I am getting error E/AudioTrackShared.

[Nickolay V. Shmyrev]

But it doesn't work.

Sure, this code is not the recommended one

I also tried aswer in this thread https://sourceforge.net/p/cmusphinx/discussion/help/thread/5be47108/ but it also didn't work.

This code should work

To be more accurate: Pocketsphinx works awesome with built-in mic, but problem is with external BT headset - when I open the app, BT headset beeps that it is prepared - SCO is on (I checked), but it seems like BT headset is muted and I am getting error E/AudioTrackShared.

Android Bluetooth stack is not very stable, if you have some exotic phone I recommend you to try on some other more common model. Samsung Galaxy S3 should be working ok.

[tsn]

Thank you for your quick answer. I tried that code once more. My code looks like this:

public class PocketSphinxActivity extends Activity implements
        RecognitionListener {

    private static final String TAG = "PocketSphinxActivity";
    private static final int HEADSET_ENABLE_TIMEOUT = 5000;

    /* Named searches allow to quickly reconfigure the decoder */
    private static final String KWS_SEARCH = "wakeup";
    private static final String FORECAST_SEARCH = "forecast";
    private static final String DIGITS_SEARCH = "digits";
    private static final String PHONE_SEARCH = "phones";
    private static final String MENU_SEARCH = "menu";

    /* Keyword we are looking for to activate menu */
    private static final String KEYPHRASE = "oh mighty computer";

    /* Used to handle permission request */
    private static final int PERMISSIONS_REQUEST_RECORD_AUDIO = 1;

    private SpeechRecognizer recognizer;
    private HashMap<String, Integer> captions;

    @Override
    public void onCreate(Bundle state) {
        super.onCreate(state);

        // Prepare the data for UI
        captions = new HashMap<String, Integer>();
        captions.put(KWS_SEARCH, R.string.kws_caption);
        captions.put(MENU_SEARCH, R.string.menu_caption);
        captions.put(DIGITS_SEARCH, R.string.digits_caption);
        captions.put(PHONE_SEARCH, R.string.phone_caption);
        captions.put(FORECAST_SEARCH, R.string.forecast_caption);
        setContentView(R.layout.main);
        ((TextView) findViewById(R.id.caption_text))
                .setText("Preparing the recognizer");

        // Check if user has given permission to record audio
        int permissionCheck = ContextCompat.checkSelfPermission(getApplicationContext(), Manifest.permission.RECORD_AUDIO);
        if (permissionCheck == PackageManager.PERMISSION_DENIED) {
            ActivityCompat.requestPermissions(this, new String[]{Manifest.permission.RECORD_AUDIO}, PERMISSIONS_REQUEST_RECORD_AUDIO);
            return;
        }

        //registerReceiver(connectionReceiver, new IntentFilter(BluetoothAdapter.ACTION_CONNECTION_STATE_CHANGED));

        registerReceiver(connectionReceiver, new IntentFilter(BluetoothAdapter.ACTION_CONNECTION_STATE_CHANGED));
        runRecognizerSetup();
    }

    BroadcastReceiver connectionReceiver = new BroadcastReceiver() {
        @Override
        public void onReceive(Context context, Intent intent) {
            int state = intent.getIntExtra(BluetoothAdapter.EXTRA_CONNECTION_STATE, -1);
            Log.i(TAG, "Bluetooth connection state " + state);
            checkHeadset();
        }
    };

    @SuppressWarnings("deprecation")
    private void checkHeadset() {
        BluetoothAdapter adapter = BluetoothAdapter.getDefaultAdapter();
        if (adapter.getProfileConnectionState(BluetoothProfile.HEADSET) == BluetoothProfile.STATE_CONNECTED) {

            new Handler(Looper.getMainLooper()).postDelayed(new Runnable() {
                public void run() {
                    AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE);
                    am.startBluetoothSco();
                    am.setMicrophoneMute(true);
                }
            }, HEADSET_ENABLE_TIMEOUT);
        }
    }

    private void runRecognizerSetup() {
        // Recognizer initialization is a time-consuming and it involves IO,
        // so we execute it in async task

        new AsyncTask<Void, Void, Exception>() {
            @Override
            protected Exception doInBackground(Void... params) {
                try {
                    Assets assets = new Assets(PocketSphinxActivity.this);
                    File assetDir = assets.syncAssets();
                    setupRecognizer(assetDir);
                } catch (IOException e) {
                    return e;
                }

                return null;
            }

            @Override
            protected void onPostExecute(Exception result) {
                if (result != null) {
                    ((TextView) findViewById(R.id.caption_text))
                            .setText("Failed to init recognizer " + result);
                } else {
                    switchSearch(PHONE_SEARCH);
                }
            }
        }.execute();
    }

    @Override
    public void onRequestPermissionsResult(int requestCode,
                                           String[] permissions, int[] grantResults) {
        super.onRequestPermissionsResult(requestCode, permissions, grantResults);

        if (requestCode == PERMISSIONS_REQUEST_RECORD_AUDIO) {
            if (grantResults.length > 0 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                runRecognizerSetup();
            } else {
                finish();
            }
        }
    }

    @Override
    public void onDestroy() {
        super.onDestroy();

        AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE);
        am.stopBluetoothSco();
        am.setMicrophoneMute(false);

       // setAudioManagerDefault();

        if (recognizer != null) {
            recognizer.cancel();
            recognizer.shutdown();
        }
    }

    /**
     * In partial result we get quick updates about current hypothesis. In
     * keyword spotting mode we can react here, in other modes we need to wait
     * for final result in onResult.
     */
    @Override
    public void onPartialResult(Hypothesis hypothesis) {
        if (hypothesis == null)
            return;

        String text = hypothesis.getHypstr();
        switchSearch(PHONE_SEARCH);

            ((TextView) findViewById(R.id.result_text)).setText(text);
    }

    /**
     * This callback is called when we stop the recognizer.
     */
    @Override
    public void onResult(Hypothesis hypothesis) {
        ((TextView) findViewById(R.id.result_text)).setText("");
        if (hypothesis != null) {
            String text = hypothesis.getHypstr();
            makeText(getApplicationContext(), text, Toast.LENGTH_SHORT).show();
        }
    }

    @Override
    public void onBeginningOfSpeech() {
    }

    /**
     * We stop recognizer here to get a final result
     */
    @Override
    public void onEndOfSpeech() {
       /* if (!recognizer.getSearchName().equals(KWS_SEARCH))
            switchSearch(PHONE_SEARCH); */
    }

    private void switchSearch(String searchName) {
        recognizer.stop();

        // If we are not spotting, start listening with timeout (10000 ms or 10 seconds).
        if (searchName.equals(KWS_SEARCH))
            recognizer.startListening(searchName);
        else
            recognizer.startListening(searchName, 10000);

        String caption = getResources().getString(captions.get(searchName));
        ((TextView) findViewById(R.id.caption_text)).setText(caption);
    }

    private void setupRecognizer(File assetsDir) throws IOException {
        // The recognizer can be configured to perform multiple searches
        // of different kind and switch between them

        recognizer = SpeechRecognizerSetup.defaultSetup()
                .setAcousticModel(new File(assetsDir, "en-us-ptm"))
                .setDictionary(new File(assetsDir, "cmudict-en-us.dict"))

                .setRawLogDir(assetsDir) // To disable logging of raw audio comment out this call (takes a lot of space on the device)

                .getRecognizer();
        recognizer.addListener(this);

        /** In your application you might not need to add all those searches.
         * They are added here for demonstration. You can leave just one.
         */

        // Create keyword-activation search.
        recognizer.addKeyphraseSearch(KWS_SEARCH, KEYPHRASE);

        // Create grammar-based search for selection between demos
        File menuGrammar = new File(assetsDir, "menu.gram");
        recognizer.addGrammarSearch(MENU_SEARCH, menuGrammar);

        // Create grammar-based search for digit recognition
        File digitsGrammar = new File(assetsDir, "digits.gram");
        recognizer.addGrammarSearch(DIGITS_SEARCH, digitsGrammar);

        // Create language model search
        File languageModel = new File(assetsDir, "weather.dmp");
        recognizer.addNgramSearch(FORECAST_SEARCH, languageModel);

        // Phonetic search
        File phoneticModel = new File(assetsDir, "en-phone.dmp");
        recognizer.addAllphoneSearch(PHONE_SEARCH, phoneticModel);
    }

    @Override
    public void onError(Exception error) {
        ((TextView) findViewById(R.id.caption_text)).setText(error.getMessage());
    }

    @Override
    public void onTimeout() {
        switchSearch(PHONE_SEARCH);
    }

    @Override
    public void onStart() {
        super.onStart();
    }

    @Override
    public void onStop() {
        super.onStop();
    }
}

On my Huawei P8 Lite I get that error

E/AudioTrackShared: obtainBuffer() timeout [110]

So I tried that on my Nexus 7 (tablet), where I don't getting that error, but just warning

W/AudioRecord: dead IAudioRecord, creating a new one from obtainBuffer()

Which I think is not the cause of the problem, but still.. No phoneme recognition even with Nexus 7.

Am I doing something wrong in my code?

Thank you for your help.

[Nickolay V. Shmyrev]

Did you try to cancel recognizer before starting sco and restarting it again? This is what nexus message tells you probalby. You can also check recorded audio in rawlogdir if it actually contains sounds.

[tsn]

I tried to cancel recognizer and start it again like this:

    @SuppressWarnings("deprecation")
    private void checkHeadset() {
        BluetoothAdapter adapter = BluetoothAdapter.getDefaultAdapter();
        if (adapter.getProfileConnectionState(BluetoothProfile.HEADSET) == BluetoothProfile.STATE_CONNECTED) {

            new Handler(Looper.getMainLooper()).postDelayed(new Runnable() {
                public void run() {

                    recognizer.cancel();

                    am = (AudioManager) getSystemService(Context.AUDIO_SERVICE);
                    am.startBluetoothSco();
                    am.setMicrophoneMute(true);
                    Log.i(TAG, "Mic is muted " + am.isMicrophoneMute());

                    recognizer.startListening(PHONE_SEARCH);

                }
            }, HEADSET_ENABLE_TIMEOUT);
        }
    }

But nothing chagned. The warning is still there.

Raw file is over 1MB (attachment), but I don't know what to look for in it.

Also, there is some weird beeps in headset (two beeps after turning on). I don't know what that means, some warning maybe? I have to look into documentation.

Thank you.

[tsn]

Those two beeps in headset are actually after call of startBluetoothSco method. Not after turning on.

[Nickolay V. Shmyrev]

Ok, what if you call recognizer.shutdown()? it will properly release the recorder then. You will have to create another recognizer after that.

[tsn]

Now I have it like this:

    @SuppressWarnings("deprecation")
    private void checkHeadset() {
        BluetoothAdapter adapter = BluetoothAdapter.getDefaultAdapter();
        if (adapter.getProfileConnectionState(BluetoothProfile.HEADSET) == BluetoothProfile.STATE_CONNECTED) {

            recognizer.shutdown();

            new Handler(Looper.getMainLooper()).postDelayed(new Runnable() {
                public void run() {
                    am = (AudioManager) getSystemService(Context.AUDIO_SERVICE);
                  //  am.setMode(AudioManager.MODE_IN_CALL);
                  //  am.setBluetoothScoOn(true);
                    am.startBluetoothSco();
                    am.setMicrophoneMute(true);
                    Log.i(TAG, "Mic is muted " + am.isMicrophoneMute());

                    runRecognizerSetup();
                }
            }, HEADSET_ENABLE_TIMEOUT);
        }
    }

The warning is still there and it is actualy before postDelayed method is called

I/PocketSphinxActivity: Bluetooth connection state 2
W/AudioRecord: dead IAudioRecord, creating a new one from obtainBuffer()

//after this it waits for 12 seconds (HEADSET_ENABLE_TIMEOUT = 12000)

Beebps from headset are gone! But still no phoneme recognition.

I tried to play with am.setBluetoothScoOn(true) and setMode, but it didn't help. I also tried to call recognizer.shutdown() before checkHeadset(), but it didn't help.

[Nickolay V. Shmyrev]

I'm sorry, and what about cancel first and then shutdown? Did you try with a different headset?

[tsn]

Sorry for late answer. I did try cancel recognizer first, then shutdown and after startBluetoothSco() I call runRecognizerSetup(). Still same problem. I also tried different headset.

Logcat log after SCO is on:

I/cmusphinx: INFO: pocketsphinx.c(986): Writing raw audio file: /storage/sdcard0/Android/data/edu.cmu.sphinx.pocketsphinx/files/sync/000000009.raw
D/AudioRecord: stop
D/AudioRecord: -stop
I/cmusphinx: INFO: allphone_search.c(849): 0 frames, 0 HMMs (0/fr), 0 senones (0/fr), 0 history entries (0/fr)
I/SpeechRecognizer: Stop recognition
I/SpeechRecognizer: Start recognition "phones"
D/AudioRecord: return status 0
D/SpeechRecognizer: Starting decoding
I/cmusphinx: INFO: pocketsphinx.c(986): Writing raw audio file: /storage/sdcard0/Android/data/edu.cmu.sphinx.pocketsphinx/files/sync/000000010.raw
D/AudioRecord: stop
D/AudioRecord: -stop
I/cmusphinx: INFO: allphone_search.c(849): 0 frames, 0 HMMs (0/fr), 0 senones (0/fr), 0 history entries (0/fr)
I/SpeechRecognizer: Stop recognition
I/SpeechRecognizer: Start recognition "phones"
D/AudioRecord: return status 0
D/SpeechRecognizer: Starting decoding
I/cmusphinx: INFO: pocketsphinx.c(986): Writing raw audio file: /storage/sdcard0/Android/data/edu.cmu.sphinx.pocketsphinx/files/sync/000000011.raw
Application terminated.

[tsn]

I think that it's working now!

I deleted line am.setMicrophoneMute(true)

And now it's working. I think that setMicrophoneMute trigger mute on my BT headset. When I press mute on my BT headset then the phoneme recognition stops and after pressing mute again, it start to recognize phonemes. So I think it is getting input from BT headset and not from internal mic.

At least with Nexus 7 it works.

Thank you Mr. Shmyrev

[Nickolay V. Shmyrev]

Great, congratulations. I think it is an interesting change in new android.

You need to check that the embedded microphone of the phone is properly muted in this case. It might still record sound at least it was recording the sound before. You can take phone away and try only with headset.

[Riccardo Casero]

Hi Nickolay and tns,
I tried to do the same with a Lenovo phone (Android 5.1) but even though I am not getting error the app only gets audio from mic, and not from the BT headset.

Here it is claimed AudioRecord must be initialized with AudioSource.VOICE_COMMUNICATION whereas the SpeechRecognizer class uses AudioSource.VOICE_RECOGNITION (value=6).

 protected SpeechRecognizer(Config config) throws IOException {
        this.decoder = new Decoder(config);
        this.sampleRate = (int)this.decoder.getConfig().getFloat("-samprate");
        this.bufferSize = Math.round((float)this.sampleRate * 0.4F);
        this.recorder = new AudioRecord(6, this.sampleRate, 16, 2, this.bufferSize * 2);
        if(this.recorder.getState() == 0) {
            this.recorder.release();
            throw new IOException("Failed to initialize recorder. Microphone might be already in use.");
        }
    }

tns: were you able to doublecheck you are actually getting audio from BT?

Nickolay: do you know anyone ever getting pocketsphinx working through BT? or any suggestion in general?

Thanks, Rick

[Nickolay V. Shmyrev]

Here it is claimed AudioRecord must be initialized with AudioSource.VOICE_COMMUNICATION whereas the SpeechRecognizer class uses AudioSource.VOICE_RECOGNITION (value=6).

Ok, this is easy to change in our sources.

Nickolay: do you know anyone ever getting pocketsphinx working through BT? or any suggestion in general?

tns above did, I did it too.

[Riccardo Casero]

Investigating a bit more, I was actually already getting the audio from the BT headset. I listened to the raw files and voice is there! So apparantely it is fine to initialize AudioRecord with VOICE_RECOGNITION.

What happens though (and made me think audio was not comng from BT) is that when using BT no recognition happens, while when using phone microphone pocketsphinx does recognize voice. In both cases I am forcing sample rate to 8000KHz, and I am using an 8KHz acoustic model. Would you have an explanation for that?

Thanks, Rick

[Nickolay V. Shmyrev]

It is hard to give you an advise on the accuracy without logcat output and raw recordings.

When switch to bluetooth happens volume becomes low and CMN needs update, so it might be quiet for couple of utterances.