mediapipe_audio library

Google's MediaPipe audio tasks: Audio Classifier, with one API on Android, iOS, macOS, Linux, Windows and the web, and Audio Transcriber and Text to Speech on 64-bit Android, iOS, macOS, Linux, Windows and the web.

Each task has static create(options), Google's verbs (classify, transcribe, synthesize), a delegate getter and dispose(). What a platform's runtime cannot do throws RuntimeUnavailableException, and the capability query reports it in advance.

Classes

AudioClassifier
Google's Audio Classifier (for example YAMNet): the categories of each window of audio the model reads (0.975 s for YAMNet), from clips or from a continuous stream.
AudioClassifierOptions
Options for Google's Audio Classifier.
AudioClassifierResult
The categories of one window of audio (0.975 s for YAMNet), per model head.
AudioData
Samples to classify: interleaved frames at sampleRate, in -1 to 1.
AudioModels
Verified official models accepted by audio task options.
AudioTranscriber
Google's Audio Transcriber (for example Parakeet TDT 0.6B v3): speech to text, from whole utterances or a stream fed in blocks.
AudioTranscriberOptions
Options for Google's Audio Transcriber.
AudioTranscriberResult
What one call to the Audio Transcriber recognized.
BoundingBox
A box in pixels of the input image, as Google's C API reports it, not clipped to the image.
CategoryMask
The winning category index per pixel, in row-major input-image coordinates, owned by the result and valid after the task is disposed.
Classifications
The categories one classifier head produced, best first.
ConfidenceMask
Per-pixel confidences in row-major input-image coordinates, from 0 to 1, owned by the result and valid after the task is disposed.
Detection
One detected object or face: its box, categories and, for models that report them, keypoints.
DownloadAsset
A reviewed download. Every location must serve the same pinned bytes.
DownloadFailure
One unsuccessful source.
Embedding
One embedding head's vector: floats, or bytes when quantized.
Landmark
A landmark in world space, in meters from the subject's center.
Matrix
A matrix in column-major order, as Google's C API lays it out.
MediaPipeCategory
One classification: Google's Category, under the package's name because Flutter's foundation library already exports a Category.
MediaPipeWebRuntime
The browser runtime's location, shared by the vision, text and audio packages. Exists on every platform; only browsers read it.
ModelSource
Where a verified model is: a file path on native platforms, bytes in browsers. What ModelStore returns and a task reads.
ModelStore
Verified models shared by every task family: the app's bundled copies, cached in the application's persistent support directory, and downloads when asked for. The same API in browsers, where models live in Cache Storage and a ModelSource holds bytes instead of a path.
NormalizedKeypoint
A point normalized to the input image: 0 to 1 across its width and height.
NormalizedLandmark
A landmark in image space: x and y relative to the image's width and height, z scaled like x with smaller values closer to the camera.
TaskCapabilities
Declared package support for one task on a platform: which Delegates run there and why the others do not. This is not an inference self-test.
TaskOptions
Model and delegate shared by every task's options, in every family.
TaskPlatform
The current process platform, including its ABI (not the host CPU's ABI).
TextToSpeech
Google's Text to Speech (for example Phonon): speech from text, as one utterance or chunk by chunk, with when each word is spoken.
TextToSpeechOptions
Options for Google's Text to Speech.
TextToSpeechResult
Synthesized audio: a whole utterance from synthesize, or one chunk of synthesizeStream.
TextToSpeechVoice
A voice the loaded model can speak in.
TranscriptionSegment
One stretch of recognized speech.
WordTimestamp
When one word of the input is spoken.

Enums

AudioRunningMode
How the Audio Classifier is fed, chosen when it is created; Google's names.
BrowserEngine
A browser's engine, for a task whose browser runtime fails in one of them.
Delegate
The processor a task runs on, chosen when the task is created and fixed until it is disposed. One enum serves vision, text and audio.
TranscriberModelArchitecture
Which family of speech recognition model Google's Audio Transcriber routes the model to; Google's names.

Constants

speechRuntimeTargets → const Map<String, String?>
Where Google's speech tasks run: core's 64-bit targets, whose library has Audio Transcriber and Text to Speech since 1.2.2.
textToSpeechGpuTargets → const Map<String, String?>
Where Text to Speech's native GPU delegate is offered: Metal on macOS 15 and later on arm64, which the package's tests prove with Kokoro on macOS 15, the only version CI runs. Kokoro has never run on macOS 14, so it gets the CPU there. A model with no GPU section, such as Phonon, is refused the GPU when the task is created.
transcriberGpuTargets → const Map<String, String?>
Where Audio Transcriber's native GPU delegate is offered: Metal on macOS arm64, which the package's tests prove on macOS 15, the only version CI runs (they passed once on macOS 14, before GitHub retired those runners), and Metal on iPhones and iPads (not the simulator), for apps that ask for it. On one iPhone 15 Pro (iOS 27) the GPU did the work: the same transcript as the CPU, about 1.7x faster per clip, about 578 MB held on Metal, and about 4.6 s to create (0.6 s on the CPU). No other iPhone, iPad or iOS version has been tested.

Functions

audioClassifierCapabilitiesForPlatform(TaskPlatform platform) → TaskCapabilities
Audio Classifier support on platform, without loading code.
audioTranscriberCapabilitiesForPlatform(TaskPlatform platform) → TaskCapabilities
Audio Transcriber support on platform, without loading code.
decodeWav(Uint8List bytes) → AudioData
Reads a RIFF WAVE file of 16-bit PCM or 32-bit float samples, as recorded by most tools and as MediaPipe's own sample clips are.
queryAudioClassifierCapabilities() → Future<TaskCapabilities>
Where Audio Classifier runs: CPU on Google's audio library on Android, iOS 15+, macOS 14+ arm64, Linux x64 and Windows x64, and through Google's browser runtime where the web plugin installs it.
queryAudioTranscriberCapabilities() → Future<TaskCapabilities>
Where Audio Transcriber runs: Google's audio library on 64-bit Android, iOS 15+, macOS 14+ arm64, Linux x64 and Windows x64, and through Google's browser runtime where the web plugin installs it, on the CPU everywhere and on the GPU on transcriberGpuTargets and in Chromium and WebKit on a Mac's Apple WebGPU adapter.
queryTextToSpeechCapabilities() → Future<TaskCapabilities>
Where Text to Speech runs: Google's audio library on 64-bit Android, iOS 15+, macOS 14+ arm64, Linux x64 and Windows x64, and through Google's browser runtime where the web plugin installs it, on the CPU everywhere and on the GPU on textToSpeechGpuTargets for a model with a GPU section.
textToSpeechCapabilitiesForPlatform(TaskPlatform platform) → TaskCapabilities
Text to Speech support on platform, without loading code.

Typedefs

RuntimeTargets = Map<String, String?>
Minimum product version for one process target, e.g. macos/arm64: 14.0.

Exceptions / Errors

MediaPipeException
A failure from MediaPipe or its supporting runtime.
ModelDownloadException
A model could not be downloaded from any configured source.
RuntimeUnavailableException
The platform cannot load a task runtime required by this operation.
TaskException
A failure Google's runtime reported while creating or running a task, on any platform and in any family.