mediapipe_audio library
Google's MediaPipe audio tasks: Audio Classifier, with one API on Android, iOS, macOS, Linux, Windows and the web, and Audio Transcriber and Text to Speech on 64-bit Android, iOS, macOS, Linux, Windows and the web.
Each task has static create(options), Google's verbs (classify,
transcribe, synthesize), a delegate getter and dispose(). What a
platform's runtime cannot do throws RuntimeUnavailableException, and
the capability query reports it in advance.
Classes
- AudioClassifier
- Google's Audio Classifier (for example YAMNet): the categories of each window of audio the model reads (0.975 s for YAMNet), from clips or from a continuous stream.
- AudioClassifierOptions
- Options for Google's Audio Classifier.
- AudioClassifierResult
- The categories of one window of audio (0.975 s for YAMNet), per model head.
- AudioData
- Samples to classify: interleaved frames at sampleRate, in -1 to 1.
- AudioModels
- Verified official models accepted by audio task options.
- AudioTranscriber
- Google's Audio Transcriber (for example Parakeet TDT 0.6B v3): speech to text, from whole utterances or a stream fed in blocks.
- AudioTranscriberOptions
- Options for Google's Audio Transcriber.
- AudioTranscriberResult
- What one call to the Audio Transcriber recognized.
- BoundingBox
- A box in pixels of the input image, as Google's C API reports it, not clipped to the image.
- CategoryMask
- The winning category index per pixel, in row-major input-image coordinates, owned by the result and valid after the task is disposed.
- Classifications
- The categories one classifier head produced, best first.
- ConfidenceMask
- Per-pixel confidences in row-major input-image coordinates, from 0 to 1, owned by the result and valid after the task is disposed.
- Detection
- One detected object or face: its box, categories and, for models that report them, keypoints.
- DownloadAsset
- A reviewed download. Every location must serve the same pinned bytes.
- DownloadFailure
- One unsuccessful source.
- Embedding
- One embedding head's vector: floats, or bytes when quantized.
- Landmark
- A landmark in world space, in meters from the subject's center.
- Matrix
- A matrix in column-major order, as Google's C API lays it out.
- MediaPipeCategory
-
One classification: Google's
Category, under the package's name because Flutter's foundation library already exports aCategory. - MediaPipeWebRuntime
- The browser runtime's location, shared by the vision, text and audio packages. Exists on every platform; only browsers read it.
- ModelSource
-
Where a verified model is: a file path on native platforms, bytes in
browsers. What
ModelStorereturns and a task reads. - ModelStore
- Verified models shared by every task family: the app's bundled copies, cached in the application's persistent support directory, and downloads when asked for. The same API in browsers, where models live in Cache Storage and a ModelSource holds bytes instead of a path.
- NormalizedKeypoint
- A point normalized to the input image: 0 to 1 across its width and height.
- NormalizedLandmark
- A landmark in image space: x and y relative to the image's width and height, z scaled like x with smaller values closer to the camera.
- TaskCapabilities
- Declared package support for one task on a platform: which Delegates run there and why the others do not. This is not an inference self-test.
- TaskOptions
- Model and delegate shared by every task's options, in every family.
- TaskPlatform
- The current process platform, including its ABI (not the host CPU's ABI).
- TextToSpeech
- Google's Text to Speech (for example Phonon): speech from text, as one utterance or chunk by chunk, with when each word is spoken.
- TextToSpeechOptions
- Options for Google's Text to Speech.
- TextToSpeechResult
-
Synthesized audio: a whole utterance from
synthesize, or one chunk ofsynthesizeStream. - TextToSpeechVoice
- A voice the loaded model can speak in.
- TranscriptionSegment
- One stretch of recognized speech.
- WordTimestamp
- When one word of the input is spoken.
Enums
- AudioRunningMode
- How the Audio Classifier is fed, chosen when it is created; Google's names.
- BrowserEngine
- A browser's engine, for a task whose browser runtime fails in one of them.
- Delegate
- The processor a task runs on, chosen when the task is created and fixed until it is disposed. One enum serves vision, text and audio.
- TranscriberModelArchitecture
- Which family of speech recognition model Google's Audio Transcriber routes the model to; Google's names.
Constants
-
speechRuntimeTargets
→ const Map<
String, String?> - Where Google's speech tasks run: core's 64-bit targets, whose library has Audio Transcriber and Text to Speech since 1.2.2.
-
textToSpeechGpuTargets
→ const Map<
String, String?> - Where Text to Speech's native GPU delegate is offered: Metal on macOS 15 and later on arm64, which the package's tests prove with Kokoro on macOS 15, the only version CI runs. Kokoro has never run on macOS 14, so it gets the CPU there. A model with no GPU section, such as Phonon, is refused the GPU when the task is created.
-
transcriberGpuTargets
→ const Map<
String, String?> - Where Audio Transcriber's native GPU delegate is offered: Metal on macOS arm64, which the package's tests prove on macOS 15, the only version CI runs (they passed once on macOS 14, before GitHub retired those runners), and Metal on iPhones and iPads (not the simulator), for apps that ask for it. On one iPhone 15 Pro (iOS 27) the GPU did the work: the same transcript as the CPU, about 1.7x faster per clip, about 578 MB held on Metal, and about 4.6 s to create (0.6 s on the CPU). No other iPhone, iPad or iOS version has been tested.
Functions
-
audioClassifierCapabilitiesForPlatform(
TaskPlatform platform) → TaskCapabilities -
Audio Classifier support on
platform, without loading code. -
audioTranscriberCapabilitiesForPlatform(
TaskPlatform platform) → TaskCapabilities -
Audio Transcriber support on
platform, without loading code. -
decodeWav(
Uint8List bytes) → AudioData - Reads a RIFF WAVE file of 16-bit PCM or 32-bit float samples, as recorded by most tools and as MediaPipe's own sample clips are.
-
queryAudioClassifierCapabilities(
) → Future< TaskCapabilities> - Where Audio Classifier runs: CPU on Google's audio library on Android, iOS 15+, macOS 14+ arm64, Linux x64 and Windows x64, and through Google's browser runtime where the web plugin installs it.
-
queryAudioTranscriberCapabilities(
) → Future< TaskCapabilities> - Where Audio Transcriber runs: Google's audio library on 64-bit Android, iOS 15+, macOS 14+ arm64, Linux x64 and Windows x64, and through Google's browser runtime where the web plugin installs it, on the CPU everywhere and on the GPU on transcriberGpuTargets and in Chromium and WebKit on a Mac's Apple WebGPU adapter.
-
queryTextToSpeechCapabilities(
) → Future< TaskCapabilities> - Where Text to Speech runs: Google's audio library on 64-bit Android, iOS 15+, macOS 14+ arm64, Linux x64 and Windows x64, and through Google's browser runtime where the web plugin installs it, on the CPU everywhere and on the GPU on textToSpeechGpuTargets for a model with a GPU section.
-
textToSpeechCapabilitiesForPlatform(
TaskPlatform platform) → TaskCapabilities -
Text to Speech support on
platform, without loading code.
Typedefs
-
RuntimeTargets
= Map<
String, String?> -
Minimum product version for one process target, e.g.
macos/arm64: 14.0.
Exceptions / Errors
- MediaPipeException
- A failure from MediaPipe or its supporting runtime.
- ModelDownloadException
- A model could not be downloaded from any configured source.
- The platform cannot load a task runtime required by this operation.
- TaskException
- A failure Google's runtime reported while creating or running a task, on any platform and in any family.