The notes in this section supplement the OpenSL ES 1.0.1 specification.
Objects and Interface Initialization
Two aspects of the OpenSL ES programming model that may be unfamiliar to new developers are the distinction between objects and interfaces, and the initialization sequence.
Briefly, an OpenSL ES object is similar to the object concept in
programming languages such as Java
and C++, except an OpenSL ES object is only visible via its associated interfaces.
the initial interface for all objects, called
There is no handle for an object
itself, only a handle to the
SLObjectItf interface of the object.
An OpenSL ES object is first created, which returns an
realized. This is similar to the common programming pattern of first constructing an
object (which should never fail other than for lack of memory or invalid parameters), and then
completing initialization (which may fail due to lack of resources). The realize step gives the
implementation a logical place to allocate additional resources if needed.
As part of the API to create an object, an application specifies an array of desired interfaces that it plans to acquire later. Note that this array does not automatically acquire the interfaces; it merely indicates a future intention to acquire them. Interfaces are distinguished as implicit or explicit. An explicit interface must be listed in the array if it will be acquired later. An implicit interface need not be listed in the object create array, but there is no harm in listing it there. OpenSL ES has one more kind of interface called dynamic, which does not need to be specified in the object create array and can be added later after the object is created. The Android implementation provides a convenience feature to avoid this complexity, which is described in Dynamic interfaces at object creation.
After the object is created and realized, the application should acquire interfaces for each
feature it needs, using
GetInterface on the initial
Finally, the object is available for use via its interfaces, though note that some objects require further setup. In particular, an audio player with URI data source needs a bit more preparation in order to detect connection errors. See the Audio player prefetch section for details.
After your application is done with the object, you should explicitly destroy it; see the Destroy section below.
Audio Player Prefetch
For an audio player with URI data source,
resources but does not
connect to the data source (prepare) or begin pre-fetching data. These occur once the
player state is set to either
Some information may still be unknown until relatively late in this sequence. In
MuteSolo::GetChannelCount either returns successfully with channel count zero or the
SL_RESULT_PRECONDITIONS_VIOLATED. These APIs return the proper values
once they are known.
Other properties that are initially unknown include the sample rate and actual media content type based on examining the content's header (as opposed to the application-specified MIME type and container type). These are also determined later during prepare/prefetch, but there are no APIs to retrieve them.
The prefetch status interface is useful for detecting when all information is available, or your application can poll periodically. Note that some information, such as the duration of a streaming MP3, may never be known.
The prefetch status interface is also useful for detecting errors. Register a callback
at least the
events. If both of these events are delivered simultaneously, and
PrefetchStatus::GetFillLevel reports a zero level, and
indicates a non-recoverable error in the data source. This includes the inability to
connect to the
data source because the local filename does not exist or the network URI is invalid.
The next version of OpenSL ES is expected to add more explicit support for handling errors in the data source. However, for future binary compatibility, we intend to continue to support the current method for reporting a non-recoverable error.
In summary, a recommended code sequence is:
Note: Preparation and prefetching occur here; during this time your callback is called with periodic status updates.
Be sure to destroy all objects when exiting from your application. Objects should be destroyed in reverse order of their creation, as it is not safe to destroy an object that has any dependent objects. For example, destroy in this order: audio players and recorders, output mix, and then finally the engine.
OpenSL ES does not support automatic garbage collection or
counting of interfaces. After you call
Object::Destroy, all extant
interfaces that are
derived from the associated object become undefined.
The Android OpenSL ES implementation does not detect the incorrect use of such interfaces. Continuing to use such interfaces after the object is destroyed can cause your application to crash or behave in unpredictable ways.
We recommend that you explicitly set both the primary object interface and all associated
NULL as part of your object destruction sequence, which prevents the accidental
misuse of a stale interface handle.
Volume::EnableStereoPosition is used to enable stereo panning of a mono source,
there is a 3-dB reduction in total
level. This is needed to permit the total sound power level to remain constant as
the source is
panned from one channel to the other. Therefore, only enable stereo positioning if you need
it. For more information see the Wikipedia article on
Callbacks and Threads
Callback handlers are generally called synchronously when the implementation detects an event. This point is asynchronous with respect to the application, so you should use a non-blocking synchronization mechanism to control access to any variables shared between the application and the callback handler. In the example code, such as for buffer queues, we have either omitted this synchronization or used blocking synchronization in the interest of simplicity. However, proper non- blocking synchronization is critical for any production code.
Callback handlers are called from internal non-application threads that are not attached to the Android runtime, so they are ineligible to use JNI. Because these internal threads are critical to the integrity of the OpenSL ES implementation, a callback handler should also not block or perform excessive work.
If your callback handler needs to use JNI or execute work that is not proportional to the callback, the handler should instead post an event for another thread to process. Examples of acceptable callback workload include rendering and enqueuing the next output buffer (for an AudioPlayer), processing the just-filled input buffer and enqueueing the next empty buffer (for an AudioRecorder), or simple APIs such as most of the Get family. See the Performance section below regarding the workload.
Note that the converse is safe: an Android application thread that has entered JNI is allowed to directly call OpenSL ES APIs, including those that block. However, blocking calls are not recommended from the main thread, as they may result in Application Not Responding (ANR).
The determination regarding the thread that calls a callback handler is largely left up to the implementation. The reason for this flexibility is to permit future optimizations, especially on multi-core devices.
The thread on which the callback handler runs is not guaranteed to have the same identity across
different calls. Therefore, do not rely on the
pthread_t returned by
pthread_self() or the
pid_t returned by
gettid() to be
consistent across calls. For the same reason, do not use the thread local storage (TLS) APIs such as
pthread_getspecific() from a callback.
The implementation guarantees that concurrent callbacks of the same kind, for the same object, does not occur. However, concurrent callbacks of different kinds for the same object are possible on different threads.
As OpenSL ES is a native C API, non-runtime application threads that call OpenSL ES have no runtime-related overhead such as garbage collection pauses. With one exception described below, there is no additional performance benefit to the use of OpenSL ES other than this. In particular, the use of OpenSL ES does not guarantee enhancements such as lower audio latency and higher scheduling priority over that which the platform generally provides. On the other hand, as the Android platform and specific device implementations continue to evolve, an OpenSL ES application can expect to benefit from any future system performance improvements.
One such evolution is support for reduced
audio output latency.
The underpinnings for reduced
output latency were first included in Android 4.1 (API level 16), and then
continued progress occurred in Android 4.2 (API level 17). These improvements are available via
OpenSL ES for device implementations that
If the device doesn't claim this feature but supports Android 2.3 (API level 9)
or later, then you can still use the OpenSL ES APIs but the output latency may be higher.
output latency path is used only if the application requests a buffer size and sample rate
compatible with the device's native output configuration. These parameters are
should be obtained as described below.
Beginning with Android 4.2 (API level 17), an application can query for the platform native or optimal output sample rate and buffer size for the device's primary output stream. When combined with the feature test just mentioned, an app can now configure itself appropriately for lower latency output on devices that claim support.
For Android 4.2 (API level 17) and earlier, a buffer count of two or more is required for lower latency. Beginning with Android 4.3 (API level 18), a buffer count of one is sufficient for lower latency.
All OpenSL ES interfaces for output effects preclude the lower latency path.
The recommended sequence is as follows:
- Check for API level 9 or higher to confirm the use of OpenSL ES.
- Check for the
android.hardware.audio.low_latencyfeature using code such as this:
import android.content.pm.PackageManager ... val pm: PackageManager = context.packageManager val claimsFeature: Boolean = pm.hasSystemFeature(PackageManager.FEATURE_AUDIO_LOW_LATENCY)
import android.content.pm.PackageManager; ... PackageManager pm = getContext().getPackageManager(); boolean claimsFeature = pm.hasSystemFeature(PackageManager.FEATURE_AUDIO_LOW_LATENCY);
- Check for API level 17 or higher to confirm the use of
- Get the native or optimal output sample rate and buffer size for this device's
stream using code such as this:
import android.media.AudioManager ... val am = getSystemService(Context.AUDIO_SERVICE) as AudioManager val sampleRate: String = am.getProperty(AudioManager.PROPERTY_OUTPUT_SAMPLE_RATE) val framesPerBuffer: String = am.getProperty(AudioManager.PROPERTY_OUTPUT_FRAMES_PER_BUFFER)
import android.media.AudioManager; ... AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE); String sampleRate = am.getProperty(AudioManager.PROPERTY_OUTPUT_SAMPLE_RATE)); String framesPerBuffer = am.getProperty(AudioManager.PROPERTY_OUTPUT_FRAMES_PER_BUFFER));
framesPerBufferare strings. First check for null and then convert to int using
- Now use OpenSL ES to create an AudioPlayer with PCM buffer queue data locator.
Note: You can use the Audio Buffer Size test app to determine the native buffer size and sample rate for OpenSL ES audio applications on your audio device. You can also visit GitHub to view audio-buffer-size samples.
The number of lower latency audio players is limited. If your application requires more than a few audio sources, consider mixing your audio at the application level. Be sure to destroy your audio players when your activity is paused, as they are a global resource shared with other apps.
To avoid audible glitches, the buffer queue callback handler must execute within a small and predictable time window. This typically implies no unbounded blocking on mutexes, conditions, or I/O operations. Instead consider try locks, locks and waits with timeouts, and non-blocking algorithms.
The computation required to render the next buffer (for AudioPlayer) or consume the previous buffer (for AudioRecord) should take approximately the same amount of time for each callback. Avoid algorithms that execute in a non-deterministic amount of time or are bursty in their computations. A callback computation is bursty if the CPU time spent in any given callback is significantly larger than the average. In summary, the ideal is for the CPU execution time of the handler to have variance near zero, and for the handler to not block for unbounded times.
Lower latency audio is possible for these outputs only:
- On-device speakers.
- Wired headphones.
- Wired headsets.
- Line out.
- USB digital audio.
On some devices, speaker latency is higher than other paths due to digital signal processing for speaker correction and protection.
As of API level 21, lower latency
audio input is supported on select devices. To take advantage of this feature, first confirm
that lower latency output is available as described above. The capability for lower latency output
is a prerequisite for the lower latency input feature. Then, create an AudioRecorder with the same
sample rate and buffer size as would be used for output. OpenSL ES interfaces for input effects
preclude the lower latency path. The record preset
SL_ANDROID_RECORDING_PRESET_VOICE_RECOGNITION must be used for lower latency; this
preset disables device-specific digital signal processing that may add latency to the input path.
For more information on record presets, see the
interface section above.
For simultaneous input and output, separate buffer queue completion handlers are used for each side. There is no guarantee of the relative order of these callbacks, or the synchronization of the audio clocks, even when both sides use the same sample rate. Your application should buffer the data with proper buffer synchronization.
One consequence of potentially independent audio clocks is the need for asynchronous sample rate conversion. A simple (though not ideal for audio quality) technique for asynchronous sample rate conversion is to duplicate or drop samples as needed near a zero-crossing point. More sophisticated conversions are possible.
Security and Permissions
As far as who can do what, security in Android is done at the process level. Java programming language code cannot do anything more than native code, nor can native code do anything more than Java programming language code. The only differences between them are the available APIs.
Applications using OpenSL ES must request the permissions that they would need for similar
non-native APIs. For example, if your application records audio, then it needs the
android.permission.RECORD_AUDIO permission. Applications that use audio effects need
android.permission.MODIFY_AUDIO_SETTINGS. Applications that play network URI resources
android.permission.NETWORK. For more information see
Working with System
Depending on the platform version and implementation, media content parsers and software codecs may run within the context of the Android application that calls OpenSL ES (hardware codecs are abstracted but are device-dependent). Malformed content designed to exploit parser and codec vulnerabilities is a known attack vector. We recommend that you play media only from trustworthy sources or that you partition your application such that code that handles media from untrustworthy sources runs in a relatively sandboxed environment. For example, you could process media from untrustworthy sources in a separate process. Though both processes would still run under the same UID, this separation does make an attack more difficult.