Google is committed to advancing racial equity for Black communities. See how.

Build a musical game using Oboe

In this codelab, you build a simple musical game using the Oboe library, a C++ library that uses the high-performance audio APIs in the Android NDK. The objective of the game is to copy the clapping pattern you hear by tapping the screen.

Prerequisites

  • Basic knowledge of C++, including how to use header and implementation files.

What you'll do

  • Play sounds using the Oboe library.
  • Create low-latency audio streams.
  • Mix sounds together.
  • Trigger sounds precisely on a timeline.
  • Synchronize audio with the on-screen UI.

What you'll need

The game plays a funky four-beat backing track that continually loops. When the game starts, it also plays a clapping sound on the first three beats of the bar.

The user must try to repeat the three claps with the same timing by tapping the screen when the second bar begins.

Each time the user taps, the game plays a clap sound. If the tap happens at the right time, the screen flashes green. If the tap is too early or too late, the screen flashes orange or purple, respectively.

Clone project

Clone the Oboe repository on GitHub and switch to the game-codelab branch.

git clone https://github.com/google/oboe 
cd oboe
git checkout game-codelab

Open the project in Android Studio

Load Android Studio and open the codelab project:

  1. File > Open...
  2. Select the oboe/samples folder

Run the project

  1. Choose the RhythmGame run configuration.

7b4f35798850bf56.png

  1. Press Control+R to build and run the template app. It should compile and run, but it doesn't do anything except turn the screen yellow. You add functionality to the game during this codelab.

b765df05ad65059a.png

Open the RhythmGame module

The files you work on for this codelab are stored in the RhythmGame module. Expand this module in the Project window, making sure that the Android view is selected.

Now expand the cpp/native-lib folder. During this codelab, you edit Game.h and Game.cpp.

3852ca925b510220.png

Compare with the final version

During the codelab, it can be useful to refer to the final version of the code, which is stored in the game-codelab-final branch. Android Studio makes it easy to compare changes across branches.

First, enable version control integration:

  1. VCS > Enable Version Control Integration....
  2. Select git as the version control system.

Now you can compare your code with code in the game-codelab-final branch.

  1. Click the current branch in the bottom-right corner.
  2. Go to game-codelab-final > Compare with Current.

98e51186465f19de.png

This opens a new window.

  1. Choose the Files tab. A list of files with differences appears.

7de3482fbd28726c.png

  1. Click any file to view the differences.

Here's the game architecture:

fb908048f894be35.png

UI

The left side of the diagram shows objects associated with the UI.

The OpenGL Surface calls tick each time the screen needs to be updated, typically 60 times per second. Game then instructs any UI-rendering objects to render pixels to the OpenGL surface and the screen is updated.

The UI for the game is very simple: the single method SetGLScreenColor updates the color of the screen. The following colours are used to show what's happening in the game:

  • Yellow—game is loading.
  • Red—game failed to load.
  • Grey—game is running.
  • Orange—user tapped too early.
  • Green—user tapped on time.
  • Purple—user tapped too late.

Tap events

Each time the user taps the screen, the tap method is called, passing the time the event occurred.

Audio

The right side of the diagram shows objects associated with audio. Oboe provides the AudioStream class and associated objects to allow Game to send audio data to the audio output (a speaker or headphones).

Each time the AudioStream needs more data it calls AudioDataCallback::onAudioReady. This passes an array named audioData to Game, which must then fill the array with numFrames of audio frames.

tart by making a sound! You're going to load an MP3 file into memory and play it whenever the user taps the screen.

Load the sound file

The project includes a file in the assets folder named CLAP.mp3, which contains MP3 audio data. You're going to decode that MP3 file and store it as audio data in memory.

  1. Open Game.h, and declare a std::unique_ptr<Player> called mClap and a method called bool setupAudioSources.
private:
    // ...existing code... 
    std::unique_ptr<Player> mClap;
   bool setupAudioSources();
  1. Open Game.cpp and add the following code:
bool Game::setupAudioSources() {

   // Create a data source and player for the clap sound.
   std::shared_ptr<AAssetDataSource> mClapSource {
           AAssetDataSource::newFromCompressedAsset(mAssetManager, "CLAP.mp3")
   };
   if (mClapSource == nullptr){
       LOGE("Could not load source data for clap sound");
       return false;
   }
   mClap = std::make_unique<Player>(mClapSource);
   return true;
}

This decodes CLAP.mp3 into PCM data and stores it in the Player object.

Build an AudioStream

An AudioStream allows you to communicate with an audio device, such as speakers or headphones. To create one, you use an AudioStreamBuilder. This allows you to specify the properties that you would like your stream to have once you open it.

Create a new private method called bool openStream in the Game class with the following code. Remember to put the method declaration in the Game.h header file as well.

bool Game::openStream() {
   AudioStreamBuilder builder;
   builder.setFormat(AudioFormat::Float);
   builder.setPerformanceMode(PerformanceMode::LowLatency);
   builder.setSharingMode(SharingMode::Exclusive);
   builder.setSampleRate(48000);
   builder.setSampleRateConversionQuality(
      SampleRateConversionQuality::Medium);
   builder.setChannelCount(2);
}

There's quite a bit going on here, so break it down.

You created the stream builder and requested the following properties:

  • setFormat requests the sample format to be float.
  • setPerformanceMode requests a low-latency stream. You want to minimize the delay between the user tapping the screen and hearing the clap sound.
  • setSharingMode requests exclusive access to the audio device.This reduces latency further on audio devices that support exclusive access.
  • setSampleRate sets the stream's sample rate to 48000 samples per second. This matches the sample rate of your source MP3 files.
  • setSampleRateConversionQuality sets the quality of the resampling algorithm that's used if the underlying audio device does not natively support a sample rate of 48000. In this case, a medium-quality algorithm is used. This provides a good tradeoff between resampling quality and computational load.
  • setChannelCount sets the stream's channel count to 2, a stereo stream. Again, this matches the channel count of your MP3 files.

Open the stream

Now that the stream has been set up using the builder, you can go ahead and open it. There's two methods you can use to do this. Each method works by accepting an AudioStream object as its parameter. The builder then takes care of construction. The methods are the following:

  • openStream, which takes an AudioStream* as its parameter. Using this method means you must take responsibility for closing and deleting the AudioStream.
  • openManagedStream, which takes a ManagedStream as its parameter. The ManagedStream is deleted automatically when it goes out of scope.

You use openManagedStream because it's less work for you and follows the RAII idiom.

  1. Declare a member variable of type ManagedStream inside Game.h.
private:
    ...
    ManagedStream mAudioStream { nullptr };
}
  1. Add the following to the end of openStream in Game.cpp.
bool Game::openStream() {

    [...]
    Result result = builder.openManagedStream(mAudioStream);
    if (result != Result::OK){
        LOGE("Failed to open stream. Error: %s", convertToText(result));
        return false;
    }
    return true;
}

This code attempts to open the stream and returns false if there is an error.

Set up the callback

So far, so good! You have methods for opening an audio stream and loading your MP3 file into memory. Now, you need to get the audio data from memory into the audio stream.

To do this, you use an AudioDataCallback because this approach provides the best performance. Update your Game class to implement the AudioDataCallback interface.

  1. Open Game.h and locate the following line:
class Game {
  1. Change it to the following:
class Game : public AudioStreamCallback {
  1. Override the AudioStreamCallback::onAudioReady method:
public:
    // ...existing code... 
     
    // Inherited from oboe::AudioStreamCallback
    DataCallbackResult
    onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) override;

The audioData parameter for onAudioReady is an array into which you render the audio data using mClap->renderAudio.

  1. Add the implementation of onAudioReady to Game.cpp.
// ...existing code... 

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {
    mClap->renderAudio(static_cast<float *>(audioData), numFrames);
    return DataCallbackResult::Continue;
}

The return value DataCallbackResult::Continue tells the stream that you intend to keep sending audio data so callbacks should continue. If you return DataCallbackResult::Stop, the callbacks would stop and no more audio would be played through the stream.

To complete the callback setup, you must tell the audio-stream builder where to find the callback object using setCallback in openStream. Do this before the stream is opened.

bool Game::openStream() {
    ...
    builder.setCallback(this);
    Result result = builder.openManagedStream(mAudioStream);

Loading

Before the game can be played, there's a couple of things that must happen:

  • The audio stream must be opened using openStream.
  • Any MP3 files used by the game need to be decoded and loaded into memory using setupAudioSources.

These operations are blocking and, depending on the size of the MP3 files and the speed of the decoder, they might take several seconds to complete. You should avoid performing these operations on the main thread. Otherwise, you might get a dreaded ANR.

Another thing which must happen before the game can be played is starting the audio stream. It makes sense to do this after the other loading operations have completed.

Add the following code to the existing load method. You call this method on a separate thread.

void Game::load() {

   if (!openStream()) {
       mGameState = GameState::FailedToLoad;
       return;
   }

   if (!setupAudioSources()) {
       mGameState = GameState::FailedToLoad;
       return;
   }

   Result result = mAudioStream->requestStart();
   if (result != Result::OK){
       LOGE("Failed to start stream. Error: %s", convertToText(result));
       mGameState = GameState::FailedToLoad;
       return;
   }

   mGameState = GameState::Playing;
}

Here's what's going on. You're using a member variable mGameState to keep track of the game state. This is initially Loading, changing to FailedToLoad or Playing. You update the tick method shortly to check mGameState and update the screen background color accordingly.

You call openStream to open your audio stream, then setupAudioSources to load the MP3 file from memory.

Lastly, you start your audio stream by calling requestStart. This is a non-blocking method, which starts the audio callbacks as soon as possible.

Start asynchronously

All you need to do now is call your load method asynchronously. For this, you can use the C++ async function, which calls a function on a separate thread asynchronously. Update your game's start method:

void Game::start() {
   mLoadingResult = std::async(&Game::load, this);
}

This simply calls your load method asynchronously and stores the result in mLoadingResult.

Update the background color

This step is simple. Depending on the game state, you update the background color.

Update the tick method:

void Game::tick(){
   switch (mGameState){
       case GameState::Playing:
           SetGLScreenColor(kPlayingColor);
           break;
       case GameState::Loading:
           SetGLScreenColor(kLoadingColor);
           break;
       case GameState::FailedToLoad:
           SetGLScreenColor(kLoadingFailedColor);
           break;
   }
}

Handle the tap event

You're almost there, just one more thing to do. The tap method is called each time the user taps the screen. Start the clap sound by calling setPlaying.

  1. Add the following to tap:
void Game::tap(int64_t eventTimeAsUptime) {
    mClap->setPlaying(true);
}
  1. Build and run the app. You should hear a clap sound when you tap the screen.
  2. Give yourself a round of applause!

Playing a single clap sound is going to get boring pretty quickly. It would be nice to also play a backing track with a beat you can clap along to.

Until now, the game places only clap sounds into the audio stream.

Using a mixer

To play multiple sounds simultaneously, you must mix them together. Conventienly, a Mixer object has been provided, which does this as part of this codelab.

b63f396874540947.png

Create the backing track and mixer

  1. Open Game.h and declare another std::unique_ptr<Player> for the backing track and a Mixer:
private:
    ..
    std::unique_ptr<Player> mBackingTrack; 
    Mixer mMixer;
  1. In Game.cpp, add the following code after the clap sound has been loaded in setupAudioSources.
bool Game::setupAudioSources() {
   ...
   // Create a data source and player for your backing track.
   std::shared_ptr<AAssetDataSource> backingTrackSource {
           AAssetDataSource::newFromCompressedAsset(mAssetManager, "FUNKY_HOUSE.mp3")
   };
   if (backingTrackSource == nullptr){
       LOGE("Could not load source data for backing track");
       return false;
   }
   mBackingTrack = std::make_unique<Player>(backingTrackSource);
   mBackingTrack->setPlaying(true);
   mBackingTrack->setLooping(true);

   // Add both players to a mixer.
   mMixer.addTrack(mClap.get());
   mMixer.addTrack(mBackingTrack.get());
   mMixer.setChannelCount(mAudioStream->getChannelCount());
   return true;
}

This loads the contents of the FUNKY_HOUSE.mp3 asset (which contains MP3 data in the same format as the clap sound asset) into a Player object. Playback starts when the game starts and loops indefinitely.

Both the clap sound and backing track are added to the mixer, and the mixer's channel count is set to match that of your audio stream.

Update the audio callback

You now need to tell the audio callback to use the mixer rather than the clap sound for rendering.

  1. Update onAudioReady to the following:
DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {

    mMixer.renderAudio(static_cast<float*>(audioData), numFrames);
    return DataCallbackResult::Continue;
}
  1. Build and run the game. You should hear the backing track and the clap sound when you tap the screen. Feel free to jam for a minute!

Now things start to get interesting. You're going to start adding the gameplay mechanics. The game plays a series of claps at specific times. This is called the clap pattern.

For this simple game, the clap pattern is just three claps that start on the first beat of the first bar of the backing track. The user must repeat the same pattern starting on the first beat of the second bar.

When should the game play the claps?

The backing track has a tempo of 120 beats per minute or 1 beat every 0.5 seconds. So the game must play a clap sound at the following times in the backing track:

Beat

Time (milliseconds)

1

0

2

500

3

1000

Synchronizing clap events with the backing track

Each time onAudioReady is called audio frames from the backing track (via the mixer) are rendered into the audio stream - this is what the user actually hears. By counting the number of frames which have been written we know the exact playback time, and therefore when to play a clap.

With this in mind, here's how we can play the clap events at exactly the right time:

  • Convert the current audio frame into a song position in milliseconds
  • Check whether a clap needs to be played at this song position
  • If it does, play it

Key point: By counting the number of frames which are written inside onAudioReady you know the exact playback position and can ensure perfect synchronization with the backing track.

Cross thread communication

The game has three threads: an OpenGL thread, a UI thread (main thread) and a real-time audio thread.

9cd3945342b3a7d9.png

Clap events are pushed onto the scheduling queue from the UI thread and popped off the queue from the audio thread.

The queue is accessed from multiple threads so it must be thread-safe. It must also be lock-free so it does not block the audio thread. This requirement is true for any object shared with the audio thread. Why? Because blocking the audio thread can cause audio glitches, and no-one wants to hear that!

Add the code

The game already includes a LockFreeQueue class template which is thread-safe when used with a single reader thread (in this case the audio thread) and a single writer thread (the UI thread).

To declare a LockFreeQueue you must supply two template parameters:

  • Data type of each element. Use int64_t because it allows a maximum duration in milliseconds in excess of any audio track length you would conceivably create.
  • Queue capacity (must be a power of 2). There are three clap events so set the capacity to 4.

Open Game.h and add the following declarations:

private:
    // ...existing code...  
    Mixer mMixer;
    
    LockFreeQueue<int64_t, 4> mClapEvents;
    std::atomic<int64_t> mCurrentFrame { 0 };
    std::atomic<int64_t> mSongPositionMs { 0 };

Note that mCurrentFrame and mSongPositionMs are std::atomic because they are accessed from the UI thread.

Now in Game.cpp create a new method called scheduleSongEvents. Inside this method enqueue the clap events.

void Game::scheduleSongEvents() {

   // schedule the claps
   mClapEvents.push(0);
   mClapEvents.push(500);
   mClapEvents.push(1000);
}

We need to call scheduleSongEvents from our load method before the stream is started so that all events are enqueued before the backing track starts playing.

void Game::load() {
   ...
   scheduleSongEvents();
   Result result = mAudioStream->requestStart();
   ...
}

Now update onAudioReady to the following:

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {

   float *outputBuffer = static_cast<float *>(audioData);
   int64_t nextClapEventMs;

   for (int i = 0; i < numFrames; ++i) {

       mSongPositionMs = convertFramesToMillis(
               mCurrentFrame,
               mAudioStream->getSampleRate());

       if (mClapEvents.peek(nextClapEventMs) && mSongPositionMs >= nextClapEventMs){
           mClap->setPlaying(true);
           mClapEvents.pop(nextClapEventMs);
       }mMixer.renderAudio(outputBuffer+(oboeStream->getChannelCount()*i), 1);
       mCurrentFrame++;
   }

   return DataCallbackResult::Continue;
}

The for loop iterates for numFrames. On each iteration it does the following:

  • converts the current frame into a time in milliseconds using convertFramesToMillis
  • uses that time to check whether a clap event is due, if it is:
  • the event is popped off the queue
  • the clap sound is set to playing
  • renders a single audio frame from mMixer. Pointer arithmetic is used to tell mMixer where in audioData to render the frame.

Build and run the game. Three claps should be played exactly on the beat when the game starts. Tapping on the screen still plays the clap sounds.

Feel free to experiment with different clap patterns by changing the frame values for the clap events. You can also add more clap events, remember to increase the capacity of the mClapEvents queue.

The game plays a clap pattern and expects the user to imitate the pattern. Finally, complete the game by scoring the user's taps.

Did the user tap at the right time?

After the game claps three times in the first bar, the user should tap three times starting on the first beat of the second bar.

You shouldn't expect the user to tap at exactly the right time - that would be virtually impossible! Instead, we'll allow some tolerance before and after the expected time. This defines a time range which we'll call the tap window.

If the user taps during the tap window, make the screen flash green, too early: orange and too late: purple.

35fbf0fb442b5eb7.png

Storing the tap windows is easy: store the song position at the center of each window in a queue, the same way we did for clap events. We can then pop each window off the queue when the user taps on the screen.

The song positions at the center of the tap window are as follows:

Beat

Time (milliseconds)

5

2000

6

2500

7

3000

Declare a new member variable to store these clap windows.

private: 
    LockFreeQueue<int64_t, kMaxQueueItems> mClapWindows;

Now add the song positions for the clap windows.

void Game::scheduleSongEvents() {

    ...
    // schedule the clap windows
    mClapWindows.push(2000);
    mClapWindows.push(2500);
    mClapWindows.push(3000);
}

Comparing tap events with the tap window

When the user taps on the screen we need to know whether the tap fell within the current tap window. The tap event is delivered as system uptime (milliseconds since boot), so we need to convert this to a position within the song.

Luckily this is simple. Store the uptime at the current song position each time onAudioReady is called. Declare a member variable in the header to store the song position:

private: 
    int64_t mLastUpdateTime { 0 };

Now add the following code to the end of onAudioReady.

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {
   ...
   mLastUpdateTime = nowUptimeMillis();

   return DataCallbackResult::Continue;
}

Now update the tap method to the following.

void Game::tap(int64_t eventTimeAsUptime) {

   if (mGameState != GameState::Playing){
       LOGW("Game not in playing state, ignoring tap event");
   } else {
       mClap->setPlaying(true);

       int64_t nextClapWindowTimeMs;
       if (mClapWindows.pop(nextClapWindowTimeMs)){

           // Convert the tap time to a song position
           int64_t tapTimeInSongMs = mSongPositionMs + (eventTimeAsUptime - mLastUpdateTime);
           TapResult result = getTapResult(tapTimeInSongMs, nextClapWindowTimeMs);
           mUiEvents.push(result);
       }
   }
}

We use the provided getTapResult method to determine the result of the user's tap. We then push the result onto a queue of UI events. This is explained in the next section.

Updating the screen

Once we know the accuracy of a user's tap (early, late, right-on), we need to update the screen to provide visual feedback.

To do this we'll use another instance of LockFreeQueue class with TapResult objects to create a queue of UI events. Declare a new member variable to store these.

private: 
    LockFreeQueue<TapResult, kMaxQueueItems> mUiEvents;

Then in the tick method we'll pop any pending UI events and update the screen color accordingly. Update the code for the Playing state in tick.

case GameState::Playing:
   TapResult r;
   if (mUiEvents.pop(r)) {
       renderEvent(r);
   } else {
       SetGLScreenColor(kPlayingColor);
   }
   break;

That's it! Build and run the game.

You should hear three claps when the game starts. If you tap exactly on the beat three times in the second bar you should see the screen flash green on each tap. If you're early it'll flash orange, if you're late it'll flash purple. Good luck!

Additional resources

OLD CONTENT - DELETE BEFORE PUBLICATION

Did we get what we asked for?

Just because you ask for something doesn't mean you're going to get it!

The stream builder will do its best to give you a stream which matches your requested properties, but sometimes it can't give you everything you asked for. Perhaps the current audio device doesn't allow low latency streams, or maybe it doesn't support exclusive sharing mode.

In our case, it is essential that the stream's sample format is float because that's the format we'll get from our audio assets. Let's add a check for that. Add the following at the end of openStream.

bool Game::openStream() {

    [...]
    if (mAudioStream->getFormat() != AudioFormat::Float){
        LOGE("The codelab version of this app only supports floating point output."
             "Please see the master branch for a version which includes sample format conversion");
    }

    return true;
}

If the check passes we can return true to indicate that the stream opened successfully.