General questions on android implemention

[Simon Tomlinson]

Started a new thread to discuss the android port with Oboe I'm working on.

Whilst I pretty much the Obo implementation working in terms of being able to open both the speaker device plus the recording devices on the android device, I'm struggling a little in understanding the method of transferring the data back and forther between the main application.

I've obviously modified the android/sound.cpp/h files to use the Oboe libraries and I see that CSound is a derirvative of CSoundBase which has a protected vector (vecsTmpAudioSndCrdStereo) for the transferring of data between the sound buffers and the main program, what I can't figure out is the other CSound implementations for mac, linux and windows appear to use that same vector for both incoming sound (from device to be sent up to jamulus server) and outgoing sound (mixed audio data to be delivered to audio device speakers/headphones). The way oboe works is with a single callback method for low latency audio delivery this is called for both incoming and outgoing audio streams and in the callback is a buffer which can be cast to whatever structure you need, mostly it's just an array. But if I was to copy this data (for incoming) onto the CSoundBase->vecsTmpAudioSndCrdStereo vector surely this would overrite any outgoing data that jamulus client had placed on this vector for delivery to speakers? What is the method of protecting this structure so that data is not overwritten? I had a late night working on this so probably my brain is not working well enough this morning!

Thanks

[Volker Fischer]

If you just have one callback function which delivers you with the sound card input audio and expects you to write the sound card output signal its easy: You simply take the data from the vecsTmpAudioSndCrdStereo and write that in the sound card output signal vector of your callback and then you use the sound card input audio signal and write that in the vecsTmpAudioSndCrdStereo. That way you make sure that nothing is overwritten.

[Simon Tomlinson]

Thanks, I'm now getting some data back and sending it up to the server, but the playback stream is sounding odd: sounds like a static in a bathroom. I think it could be due to type casting on the playback callback. Here's a very simiplified versoin of the code i'm using:

The numFrames is being set to 256, channelCount is 2.

oboe::DataCallbackResult CSound::onAudioReady(oboe::AudioStream *oboeStream, void *audioData, int32_t numFrames)
{
   float *floatData = static_cast<float *>(audioData);
   for (int frmNum = 0; frmNum < numFrames; ++frmNum)
   {
        for (int channelNum = 0; channelNum < oboeStream->getChannelCount(); channelNum++)
        {
            floatData[frmNum * oboeStream->getChannelCount() + channelNum] =
                    (float) pSound->vecsTmpAudioSndCrdStereo [frmNum * oboeStream->getChannelCount() + channelNum] / _MAXSHORT;
           }
    }
}

[Volker Fischer]

The code looks ok to me.
"sounds like a static in a bathroom" could also be caused by too small network jitter buffers or other network problem. 256 samples is quite large. Is it possible to go down to 128?

[Simon Tomlinson]

That got set from the client, when i choose middle option (128 samples) CSound.Init(preferredBufferMono) is called, but I guess because it's stereo it creates a buffer that is 256 samples in size. I've tried reducing it down to 64 (which makes it 128) but that makes it sound worse. I'm running it within the android emulator on my PC, so perhaps it could be something to do with that. I'm about to compile one of the samples Oboe provides to see if streaming a wav file from my filesystem provides the same effect... will keep trying.

[Simon Tomlinson]

hi Volker. Still struggling with getting this sound quality to sound right in Oboe. I've taken a video of my phone connected to a jamulus room where music is being played so you can hear. I didn't have headphones plugged in so it does cause a feedback loop fairly quickly, but hopefully initially you can hear what it sounds like.

https://drive.google.com/drive/folders/1nmZcpgsN6bRP-KFdf5nJ_HH6nnmaFxEv?usp=sharing

Here's the code I'm using for sound.cpp

/******************************************************************************\
 * Copyright (c) 2004-2020
 *
 * Author(s):
 *  Volker Fischer
 *
 ******************************************************************************
 *
 * This program is free software; you can redistribute it and/or modify it under
 * the terms of the GNU General Public License as published by the Free Software
 * Foundation; either version 2 of the License, or (at your option) any later
 * version.
 *
 * This program is distributed in the hope that it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
 * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
 * details.
 *
 * You should have received a copy of the GNU General Public License along with
 * this program; if not, write to the Free Software Foundation, Inc.,
 * 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 *
\******************************************************************************/

#include "sound.h"
#include "androiddebug.cpp"

/* Implementation *************************************************************/


CSound::CSound ( void       (*fpNewProcessCallback) ( CVector<short>& psData, void* arg ),
                 void*      arg,
                 const int  iCtrlMIDIChannel,
                 const bool bNoAutoJackConnect ) :
    CSoundBase ( "OpenSL", true, fpNewProcessCallback, arg, iCtrlMIDIChannel, bNoAutoJackConnect )
{
  //  createCallback();
     qInfo() << "Exeucting CSound::CSound";
     pSound = this;

#ifdef ANDROIDDEBUG
  qInstallMessageHandler(myMessageHandler);
#endif
}

void CSound::setupRecordingStreamParams(oboe::AudioStreamBuilder *builder, oboe::ManagedStream *mOutStream)
{
    // check what properites we have on the playBack stream first;

    builder->setDirection(oboe::Direction::Input);
        return setupCommonStreamParams(builder);
}

void CSound::setupPlaybackStreamParams(oboe::AudioStreamBuilder *builder)
{
    builder->setDirection(oboe::Direction::Output);
       return setupCommonStreamParams(builder);
}

void CSound::setupCommonStreamParams(oboe::AudioStreamBuilder *builder)
{
    // We request EXCLUSIVE mode since this will give us the lowest possible
    // latency.
    // If EXCLUSIVE mode isn't available the builder will fall back to SHARED
    // mode.
    builder->setCallback(this)
            ->setFormat(oboe::AudioFormat::Float)
            ->setSharingMode(oboe::SharingMode::Shared)
            ->setFramesPerCallback(pSound->iOpenSLBufferSizeStereo)
            ->setChannelCount(oboe::ChannelCount::Stereo)
            ->setSampleRate(48000)
            ->setSampleRateConversionQuality(oboe::SampleRateConversionQuality::Medium)
            ->setPerformanceMode(oboe::PerformanceMode::None);
    return;
}

void CSound::openStreams()
{
    // Create callback
    qDebug() << "Setting up streams";
    mCallback = this;

     oboe::AudioStreamBuilder inBuilder, outBuilder;
     setupPlaybackStreamParams(&outBuilder);
     oboe::Result result = outBuilder.openManagedStream(mPlayStream);
     if (result != oboe::Result::OK) {
         return;
     }
    // TODO Figure out low latency
    warnIfNotLowLatency(mPlayStream, "PlayStream");

     setupRecordingStreamParams(&inBuilder, &mPlayStream);
     result = inBuilder.openManagedStream(mRecordingStream);
     if (result != oboe::Result::OK) {
         qInfo() << "Error, stream closing";
         closeStream(mPlayStream);
         return;
     }

     warnIfNotLowLatency(mRecordingStream, "RecordStream");
    qDebug() << "Recording/Playback streams opened";

}

void CSound::warnIfNotLowLatency(oboe::ManagedStream &stream, QString streamName) {
    if (stream->getPerformanceMode() != oboe::PerformanceMode::LowLatency) {
        QString latencyMode = (stream->getPerformanceMode()==oboe::PerformanceMode::None ? "None" : "Power Saving");
        qInfo() << "Stream '" << streamName << "' is not lowLatency:" << latencyMode;
       // throw CGenErr ( tr ( "Stream is NOT low latency."
                         //    "Check your requested format, sample rate and channel count." ) );
    }
}

void CSound::closeStream(oboe::ManagedStream &stream)
{
    if (stream) {
        oboe::Result requestStopRes = stream->requestStop();
        qDebug() << "Requeststop on stream, res:";
        oboe::Result result = stream->close();
        if (result != oboe::Result::OK) {
            throw CGenErr ( tr ( "Error closing stream: $s",
                                 oboe::convertToText(result) ) );
        }
        stream.reset();
    }
}

void CSound::closeStreams()
{
    // clean up
    closeStream(mRecordingStream);
    closeStream(mPlayStream);
}

void CSound::Start()
{
    qInfo() << "CSound::Start - entering";
    openStreams();

    // call base class
    CSoundBase::Start();
    qInfo() << "CSound::Start - exiting";

    // finally start the streams
    mRecordingStream->requestStart();
    mPlayStream->requestStart();

}

void CSound::Stop()
{
    closeStreams();

    // call base class
    CSoundBase::Stop();
}

int CSound::Init ( const int iNewPrefMonoBufferSize )
{
     qDebug() << "Executing CSound::Init, buffersize:" << iNewPrefMonoBufferSize;

    // store buffer size
    iOpenSLBufferSizeMono = iNewPrefMonoBufferSize;

    // init base class
    CSoundBase::Init ( iOpenSLBufferSizeMono );

    // set internal buffer size value and calculate stereo buffer size
    iOpenSLBufferSizeStereo = 2 * iOpenSLBufferSizeMono;

    // create memory for intermediate audio buffer
    qDebug() << "Executing CSound::Init, setting buffer size initial to:" << iOpenSLBufferSizeStereo;
    vecsTmpAudioSndCrdStereo.Init ( iOpenSLBufferSizeStereo );


// TEST
#if ( SYSTEM_SAMPLE_RATE_HZ != 48000 )
# error "Only a system sample rate of 48 kHz is supported by this module"
#endif
// audio interface number of channels is 1 and the sample rate
// is 16 kHz -> just copy samples and perform no filtering as a
// first simple solution
// 48 kHz / 16 kHz = factor 3 (note that the buffer size mono might
// be divisible by three, therefore we will get a lot of drop outs)
iModifiedInBufSize = iOpenSLBufferSizeMono / 3;
vecsTmpAudioInSndCrd.Init ( iModifiedInBufSize );

    qInfo() << "Exiting CSound::Init exiting, have set stereo buffer size to:" << iOpenSLBufferSizeStereo ;
    return iOpenSLBufferSizeMono;
}

// This is the main callback method for when an audio stream is ready to publish data to an output stream
// or has received data on an input stream.  As per manual much be very careful not to do anything in this back that
// can cause delays such as sleeping, file processing, allocate memory, etc
oboe::DataCallbackResult CSound::onAudioReady(oboe::AudioStream *oboeStream, void *audioData, int32_t numFrames)
{
     //qDebug("CSound::onAudioReady:BEGIN");

     // only process if we are running
     if ( ! pSound->bRun )
     {
         return oboe::DataCallbackResult::Continue;
     }

     // Need to modify the size of the buffer based on the numFrames requested in this callback.
     // Buffer size can change regularly by android devices
    int& iBufferSizeMono = pSound->iOpenSLBufferSizeMono;

    // perform the processing for input and output
    QMutexLocker locker ( &pSound->Mutex );
    locker.mutex();

    //This can be called from both input and output at different times
    if (oboeStream == pSound->mPlayStream.get() && audioData)
    {
        float *floatData = static_cast<float *>(audioData);

        // Zero out the incoming container array
        memset(audioData, 0, sizeof(float) * numFrames * oboeStream->getChannelCount());

        // Only copy data if we have data to copy, otherwise fill with silence
        if (!pSound->vecsTmpAudioSndCrdStereo.empty())
        {
            for (int frmNum = 0; frmNum < numFrames; ++frmNum)
            {
                for (int channelNum = 0; channelNum < oboeStream->getChannelCount(); channelNum++)
                {
                    // copy sample received from server into output buffer
                    floatData[frmNum * oboeStream->getChannelCount() + channelNum] =
                            (float) pSound->vecsTmpAudioSndCrdStereo [frmNum * oboeStream->getChannelCount() + channelNum] / 32768 /* Convert from 16bit signed to 24bit (float) */;
               }
            }
        }
        else
        {
            // prime output stream buffer with silence
            memset(static_cast<float*>(audioData) + numFrames * oboeStream->getChannelCount(), 0,
                       (numFrames) * oboeStream->getBytesPerFrame());
        }
    }
    else if (oboeStream == pSound->mRecordingStream.get() && audioData)
    {
        // First things first, we need to discard the input queue a little for 500ms or so
        if (pSound->mCountCallbacksToDrain > 0)
        {
            // discard the input buffer
            int32_t numBytes = numFrames * oboeStream->getBytesPerFrame();
            memset(audioData, 0 /* value */, numBytes);
            pSound->mCountCallbacksToDrain--;
        }

        // We're good to start recording now
        // Take the data from the recording device ouput buffer and move
        // it to the vector ready to send up to the server

        float *floatData = static_cast<float *>(audioData);

        // Copy recording data to internal vector
        for (int frmNum = 0; frmNum < numFrames; ++frmNum)
        {
            for (int channelNum = 0; channelNum < oboeStream->getChannelCount(); channelNum++)
            {
               pSound->vecsTmpAudioSndCrdStereo [frmNum * oboeStream->getChannelCount() + channelNum] =
                       (short) floatData[frmNum * oboeStream->getChannelCount() + channelNum] * _MAXSHORT;
            }
        }

        // Tell parent class that we've put some data ready to send to the server
        pSound->ProcessCallback ( pSound->vecsTmpAudioSndCrdStereo  );
    }
    locker.unlock();
    return oboe::DataCallbackResult::Continue;
}

//TODO better handling of stream closing errors
void CSound::onErrorAfterClose(oboe::AudioStream *oboeStream, oboe::Result result)
{
    qDebug() << "CSound::onErrorAfterClose";
}

//TODO better handling of stream closing errors
void CSound::onErrorBeforeClose(oboe::AudioStream *oboeStream, oboe::Result result)
{
     qDebug() << "CSound::onErrorBeforeClose";
}

[Volker Fischer]

I would be good if you would apply all the code you have so far to the Jamulus Git repo and create a Pull Request. I saw in the "Network" view on Github that you have created a fork quite a while ago but that seems not to be very up to date. If we have your code in the official Jamulus repo, maybe some other developers can give us some help. What do you think?

[Simon Tomlinson]

Hi. I've committed all the code I've done so far to my forked branch: https://github.com/sthenos/jamulus. Still wasn't able to get the audio sounding clean and free of distortion. I had to add some additional code to Jamulus.pro, main.cpp (to add the permission for recording which now on android requires explicit request in code, not just in the android manifest), global.h (types for android for int64_t, etc are different from win32).

So I should do a new pull request from the main repo and then add the code in there and commit it back?

[Simon Tomlinson]

Sorry, I understand what you mean. I've initiated pull request from main repo and merged all my changes in. Hopefully this works.

[Volker Fischer]

It is hard to say what goes wrong. Maybe the buffer is too short which leads to the strange audio artifacts. You could try to hard code a large size for a test:

Instead of:

// store buffer size
iOpenSLBufferSizeMono = iNewPrefMonoBufferSize;

try out:

// store buffer size
iOpenSLBufferSizeMono = 1024; // TEST!!!!!!!!!!!!!!!!

[Wolfgan]

@simon good job! Do you by any chance have a way to share the resulting apk for others to beta test/try?

(and of course thx to @volker for jamulus :-)

[Andreas]

Thanks a lot for the audio implementation on Android. I succeeded in compiling an deploying to my phone after installing Oboe from https://github.com/google/oboe.git. Sound doesn't really work yet for me (it doesn't pick up anything from the microphone and I hear only noise from other clients), I guess that's the same behaviour described above.
To make it look nicer, I have scaled the icon to the required resolutions, put it in the place Android expects and referenced it in the AndoirdManifest.xml. Shall I make a pull request for that?

[Volker Fischer]

Shall I make a pull request for that?

Yes please (if it does not cause a lot of work for you). That way I can easily see what you have changed and if I want these changes to apply to the official sources.

[Volker Fischer]

BTW: Have you seen in the Git repo that I have merged the changes from Simon already. He has used Oboe in a Git submodule. So if you write that you had to install oboe, I guess you do not use the latest git code, right?

[Andreas]

I have made a pull request.
The libs/oboe directory is still empty in my clone even though git pull tells me I'm up to date.

[Simon Tomlinson]

Make sure you do a git update/pull submodules and it should pull the oboe libraries in

[Andreas]

Thanks for the hint, I was not aware of the submodule. After running git submodule init and git submodule update libs/oboe is filled.

[Andrew S]

There don't appear to be any references to an Android client for jamulus since May - is anyone still working on this? I'm happily using Jamulus every week with friends, but they all have PCs or notebooks. I have another band that mostly has people who only have phones or tablets. I can't find ANY app for them - and the work on a Jamulus client port to Android using the Oboe library looks the most likely to produce something.

Can anyone please comment on progress? Is there anything I might do to help? Thanks.

[Gilgongo]

Hi Andrew - there's been some activity on this ticket recently:

https://github.com/corrados/jamulus/issues/83

[Julian Santander]

The current status is:
- Audio in and out seems to work in the limited testing (a few different terminals, three people).
- Having tested enough to tell if audio quality is good or bad: You can hear other people talking and they can hear you without apparent distortions.
- GUI is exactly the same as the desktop version, as rendered by QT for Android. This might eventually require changes.

Building will required
- init the oboe git submodule.
- build the project for the android target in Qt Creator.

[Andrew S]

Thanks Julian - I've never tried compliling for an Android target before so I'll have to make the time to get my environment set up. If there's a test apk about I'd be more than happy to test it with actual musicians, rather than just speech.

[DonC]

Is there a solution for getting wired internet into an Android device ?

[Gilgongo]

I was going to ask that too. I assume any effective Android implementation of Jamulus implies wired Ethernet unless perhaps 5G is implied (which it might?)

[Julian Santander]

There are adapters of RJ-45 to USB. Then you need a mobile terminal with support for USB OTG (which basically means that the USB port can be configured to be "host") and typically an adaptor from USB-C to USB-A.

Having said that, the testing I'm aware has only used WiFi. In my experience, provided you have good WiFi signal (close to the AP) the result is acceptable. Network delay have a lot of variability and that's the main problem.

[Mats Wessling]

my experiences with WiFi is the opposite: numerous cases of bad sound/connections has been solved by switching to a non-radio based communication.