Making More Waves - Sampler

1. Introduction

In this codelab we'll build an audio sampler. The app records audio from the phone's built in microphone and plays it back.

The app records up to 10 seconds of audio while the Record button is pressed. When you press Play the recorded audio plays back once (while you hold down the button). Alternatively, you can turn on Loop which replays the recording over and over until you release the Play button. Each time you press Record the previous audio recording is overwritten.

What you'll learn

Basic concepts for creating a low latency recording stream
How to store and playback audio data recorded from a microphone

Prerequisites

Before starting this codelab you should consider completing the WaveMaker Part 1 codelab. It covers some basic concepts for creating audio streams which are not discussed here.

What you'll need

Android Studio 3.0.0 or above
Android 8.0 (API level 26) SDK
NDK and Build Tools installed
A simulator or Android device running Android 8.0 (API level 26) or above for testing
Some knowledge of C++ is helpful but not required

2. Architecture overview

Our sampler app has four components:

UI - Written in Java, the MainActivity class is responsible for receiving touch events and forwarding them to the JNI bridge
JNI bridge - This C++ file uses JNI to provide a communication mechanism between the UI and C++ objects. It forwards events from the UI to the Audio Engine.
Audio engine - This C++ class creates the recording and playback audio streams.
Sound recording - This C++ class stores the audio data in memory.

Here's the architecture:

3. Getting started

Clone project

Clone the codelab repository on github.

git clone https://github.com/googlecodelabs/android-wavemaker2

Import project into Android Studio

Open Android Studio and import the project:

File -> New -> Import project...
Choose the "android-wavemaker2" folder

Run the project

Choose the base run configuration.

Then press CTRL+R to build and run the template app - it should compile and run but is non-functional. You will add functionality to it during this codelab.

Open the base module

The files you'll work on for this codelab are stored in the base module. Expand this module in the Project window, making sure that the Android view is selected.

Note: The finished source code for the WaveMaker2 app can be found in the final module.

4. Build the SoundRecording class

The SoundRecording object represents recorded audio data in memory. It lets the app write data from the microphone into memory and read data out for playback.

Let's start by figuring out how to store this audio data.

Choose an audio format

First we need to choose an audio format for our samples. AAudio supports two formats:

float: Single precision floating point (4 bytes per sample)
int16_t: 16-bit integers (2 bytes per sample)

For good sound quality at low volume and other reasons we use float samples. If memory capacity is an issue, you can sacrifice fidelity and gain space by using 16-bit integers.

Choose how much storage we need

Let's assume we want to store 10 seconds of audio data. At a sample rate of 48,000 samples per second, which is the most common sample rate on modern Android devices, we need to allocate memory for 480,000 samples.

Open base/cpp/SoundRecording.h and define this constant at the top of the file.

constexpr int kMaxSamples = 480000; // 10s of audio data @ 48kHz

Define the storage array

Now we have all the information we need to define an array of floats. Add the following declaration to SoundRecording.h:

private:
    std::array<float,kMaxSamples> mData { 0 };

The { 0 } uses aggregate initialization to set all values in the array to 0.

Implement `write`

The write method has this signature:

int32_t write(const float *sourceData, int32_t numFrames);

This method receives an array of audio samples in sourceData. The size of the array is specified by numFrames. The method should return the number of samples that it actually writes out.

This can be implemented by storing the next available write index. Initially it is 0:

After each sample is written the next write index moves along by one:

This can be easily implemented as a for loop. Add the following code to the write method in SoundRecording.cpp

for (int i = 0; i < numSamples; ++i) {
    mData[mWriteIndex++] = sourceData[i];
}
return numSamples;

But wait, what if we try to write more samples than we have space for? Bad things will happen! We'll get a segmentation fault caused by attempting to access an out-of-bounds array index.

Let's add a check which changes numSamples if mData doesn't have enough space left. Add the following above the existing code.

if (mWriteIndex + numSamples > kMaxSamples) {
    numSamples = kMaxSamples - mWriteIndex;
}

Implement `read`

The read method is similar to write. We store the next read index.

And increment it after a sample is read.

We repeat this until we've read the requested number of samples. What happens when we reach the end of the available data? We have two behaviors:

If looping is enabled: Set the read index to zero - the start of the data array
If looping is disabled: Do nothing - don't increment the read index

For both these behaviors we need to know when we have reached the end of the available data. Conveniently, mWriteIndex tells us this. It contains the total number of samples which have been written to the data array.

With this in mind we can now implement the read method in SoundRecording.cpp

int32_t framesRead = 0;
while (framesRead < numSamples && mReadIndex < mWriteIndex){
    targetData[framesRead++] = mData[mReadIndex++];
    if (mIsLooping && mReadIndex == mWriteIndex) mReadIndex = 0;
}
return framesRead;

5. Build the AudioEngine class

AudioEngine performs these main tasks:

Create an instance of SoundRecording
Create a recording stream to record data from the microphone
Write the recorded data into the SoundRecording instance in the recording stream callback
Create a playback stream to play back the recorded data
Read the recorded data from the SoundRecording instance in the playback stream callback
Respond to UI events for recording, playback and looping

Let's get started with creating the SoundRecording instance.

6. Create the SoundRecording instance

Start with something easy. Create an instance of SoundRecording in AudioEngine.h:

private:
    SoundRecording mSoundRecording;

7. Create the streams

We have two streams to create: playback and recording. Which one should we create first?

We should create the playback stream first because it has only one sample rate that provides the lowest latency. Once we've created it, we can then supply its sample rate to the recording stream builder. This ensures that the playback and recording streams have the same sample rate, which means we don't have to do any extra work resampling between the streams.

Playback stream properties

Use these properties to populate the StreamBuilder which will create the playback stream:

Direction: not specified, defaults to output
Sample rate: not specified, defaults to the rate with the lowest latency
Format: float
Channel count: 2 (stereo)
Performance mode: Low latency
Sharing mode: Exclusive

Create the playback stream

We have everything we need now to create and open the playback stream. Add the following code to the top of the start method in AudioEngine.cpp.

// Create the playback stream.
StreamBuilder playbackBuilder = makeStreamBuilder();
AAudioStreamBuilder_setFormat(playbackBuilder.get(), AAUDIO_FORMAT_PCM_FLOAT);
AAudioStreamBuilder_setChannelCount(playbackBuilder.get(), kChannelCountStereo);
AAudioStreamBuilder_setPerformanceMode(playbackBuilder.get(), AAUDIO_PERFORMANCE_MODE_LOW_LATENCY);
AAudioStreamBuilder_setSharingMode(playbackBuilder.get(), AAUDIO_SHARING_MODE_EXCLUSIVE);
AAudioStreamBuilder_setDataCallback(playbackBuilder.get(), ::playbackDataCallback, this);
AAudioStreamBuilder_setErrorCallback(playbackBuilder.get(), ::errorCallback, this);

aaudio_result_t result = AAudioStreamBuilder_openStream(playbackBuilder.get(), &mPlaybackStream);

if (result != AAUDIO_OK){
   __android_log_print(ANDROID_LOG_DEBUG, __func__,
                       "Error opening playback stream %s",
                       AAudio_convertResultToText(result));
   return;
}

// Obtain the sample rate from the playback stream so we can request the same sample rate from
// the recording stream.
int32_t sampleRate = AAudioStream_getSampleRate(mPlaybackStream);

result = AAudioStream_requestStart(mPlaybackStream);
if (result != AAUDIO_OK){
   __android_log_print(ANDROID_LOG_DEBUG, __func__,
                       "Error starting playback stream %s",
                       AAudio_convertResultToText(result));
   closeStream(&mPlaybackStream);
   return;
}

Note that template methods for the data and error callbacks have been created for you. If you need to recap on how these work please refer back to the first codelab.

Recording stream properties

Use these properties to create the recording stream:

Direction: input (recording is an input, whereas playback is an output)
Sample rate: same as output stream
Format: float
Channel count: 1 (mono)
Performance mode: Low latency
Sharing mode: Exclusive

Create the recording stream

Now add the following code underneath the previously added code in start.

// Create the recording stream.
StreamBuilder recordingBuilder = makeStreamBuilder();
AAudioStreamBuilder_setDirection(recordingBuilder.get(), AAUDIO_DIRECTION_INPUT);
AAudioStreamBuilder_setPerformanceMode(recordingBuilder.get(), AAUDIO_PERFORMANCE_MODE_LOW_LATENCY);
AAudioStreamBuilder_setSharingMode(recordingBuilder.get(), AAUDIO_SHARING_MODE_EXCLUSIVE);
AAudioStreamBuilder_setFormat(recordingBuilder.get(), AAUDIO_FORMAT_PCM_FLOAT);
AAudioStreamBuilder_setSampleRate(recordingBuilder.get(), sampleRate);
AAudioStreamBuilder_setChannelCount(recordingBuilder.get(), kChannelCountMono);
AAudioStreamBuilder_setDataCallback(recordingBuilder.get(), ::recordingDataCallback, this);
AAudioStreamBuilder_setErrorCallback(recordingBuilder.get(), ::errorCallback, this);

result = AAudioStreamBuilder_openStream(recordingBuilder.get(), &mRecordingStream);

if (result != AAUDIO_OK){
   __android_log_print(ANDROID_LOG_DEBUG, __func__,
                       "Error opening recording stream %s",
                       AAudio_convertResultToText(result));
   closeStream(&mRecordingStream);
   return;
}

result = AAudioStream_requestStart(mRecordingStream);
if (result != AAUDIO_OK){
   __android_log_print(ANDROID_LOG_DEBUG, __func__,
                       "Error starting recording stream %s",
                       AAudio_convertResultToText(result));
   return;
}

8. Store the recorded data

Now we're getting into the fun part: actually storing the recorded data from the microphone into the SoundRecording object.

When we built the recording stream we specified the data callback as ::recordingDataCallback. This method calls AudioEngine::recordingCallback which has the following signature:

aaudio_data_callback_result_t AudioEngine::recordingCallback(float *audioData,
                                                            int32_t numFrames)

The audio data is supplied in audioData. Its size (in samples) is numFrames because there is only one sample per frame, since we're recording in mono.

All we need to do is:

Check mIsRecording to see whether we should be recording
If not, ignore the incoming data
If we are recording:
Supply audioData to SoundRecording using its write method
Check the return value of write, if it's zero it means that SoundRecording is full and we should stop recording
Return AAUDIO_CALLBACK_RESULT_CONTINUE so that the callbacks continue

Add the following code to recordingCallback:

if (mIsRecording) {
    int32_t framesWritten = mSoundRecording.write(audioData, numFrames);
    if (framesWritten == 0) mIsRecording = false;
}
return AAUDIO_CALLBACK_RESULT_CONTINUE;

9. Play the recorded data

Similar to the recording stream, the playback stream calls playbackDataCallback when it needs new data. This method calls AudioEngine::playbackCallback, which has the following signature:

aaudio_data_callback_result_t AudioEngine::playbackCallback(float *audioData, int32_t numFrames)

Inside this method we need to:

Fill the array with zeros using fillArrayWithZeros
If we are playing the recorded data, indicated by mIsPlaying then:
Read numFrames of data from mSoundRecording
Convert audioData from mono to stereo using convertArrayMonoToStereo
If the number of frames read was less than the number of frames requested we have reached the end of the recorded data. Stop playback by setting mIsPlaying to false

Add the following code to playbackCallback:

fillArrayWithZeros(audioData, numFrames * kChannelCountStereo);

if (mIsPlaying) {
   int32_t framesRead = mSoundRecording.read(audioData, numFrames);
   convertArrayMonoToStereo(audioData, framesRead);
   if (framesRead < numFrames) mIsPlaying = false;
}
return AAUDIO_CALLBACK_RESULT_CONTINUE;

10. Respond to UI events

Start and stop recording

The method setRecording is used to start and stop recording. Here's its signature:

void setRecording(bool isRecording)

When the record button is pressed isRecording is true; when the button is released isRecording is false.

A member variable mIsRecording is used to toggle data storage inside recordingCallback. All we need to do is set it to the value of isRecording.

There's one more behaviour we need to add. When recording starts any existing data in mSoundRecording should be overwritten. This can be achieved using clear which will reset mSoundRecording's write index to zero.

Here's the code for setRecording:

if (isRecording) mSoundRecording.clear();
mIsRecording = isRecording;

Start and stop playback

Playback control is similar recording. setPlaying is called when the play button is pressed or released. The member variable mIsPlaying toggles playback inside the playbackCallback.

When the play button is pressed we want playback to start at the beginning of the recorded audio data. This can be achieved using setReadPositionToStart which reset's mSoundRecording's read head to zero.

Here's the code for setPlaying:

if (isPlaying) mSoundRecording.setReadPositionToStart();
mIsPlaying = isPlaying;

Toggle looped playback

Lastly, when the LOOP switch is toggled setLooping is called. Handling this is simple, we just pass the isOn argument to mSoundRecording.setLooping:

Here's the code for setLooping:

mSoundRecording.setLooping(isOn);

11. Add the recording permission

Apps which record audio must request the RECORD_AUDIO permission from the user. Much of the permission handling code is already written, however, we still need to declare that our app uses this permission.

Add the following line to manifests/AndroidManifest.xml inside the <manifest> section:

<uses-permission android:name="android.permission.RECORD_AUDIO" />

12. Build and run

Time to see whether all your hard work has paid off. Build and run the app, you should see the following UI.

Tap the red button to start recording. Recording continues while you keep the button pressed, up to a maximum of 10 seconds. Tap the green button to play the recorded data. Playback continues while you keep the button pressed until the end of the audio data. If LOOP is enabled the audio data will loop forever.