avatar_lipsync 0.2.0
avatar_lipsync: ^0.2.0 copied to clipboard
Real-time lip sync for Flutter. Turns streamed speech into mouth shapes and animates a drawn face or a single photo (web, via MediaPipe).
avatar_lipsync #
Real-time lip sync for Flutter: a talking avatar driven by streamed speech. Show a drawn face on any platform, or bring a single frontal photo to life on the web.
- Works ahead of the sound. Each audio chunk is analysed when it is scheduled, before it is heard, so the mouth is never late.
- A real mouth model. The jaw follows loudness and the lips follow articulation (JALI): vowel shapes, rounding, lip closures for m/b/p, and springs for natural motion. It adapts to each speaker's voice.
- One photo is enough.
PhotoLipSyncAvatarfinds 478 face points with MediaPipe (loaded in the browser at runtime) and animates the photo as a mesh, with teeth and tongue painted in colors taken from the photo. - Alive. It blinks, sways, nods, raises its brows and shifts its gaze, and respects the reduced-motion setting.
- Any audio stack. You connect your player through one small interface. The package itself does not play audio.
Install #
flutter pub add avatar_lipsync
Quick start #
1. Tell the avatar what is being heard #
Implement LipSyncSource over your audio player. This is the smallest one:
it keeps time with a Stopwatch and is enough to try the package.
import 'dart:async';
import 'dart:typed_data';
import 'package:avatar_lipsync/avatar_lipsync.dart';
class ClockSpeechSource implements LipSyncSource {
final _clock = Stopwatch()..start();
final _chunks = StreamController<SpeechChunk>.broadcast();
final _flushes = StreamController<double>.broadcast();
@override
Stream<SpeechChunk> get chunks => _chunks.stream;
@override
Stream<double> get flushes => _flushes.stream;
@override
double? get audibleTime => _clock.elapsedMicroseconds / 1e6;
/// Mono samples in -1..1, starting now.
void speak(Float32List samples, int sampleRate) => _chunks.add(
SpeechChunk(
samples: samples,
startTime: audibleTime! + 0.05,
sampleRate: sampleRate,
),
);
/// The user interrupted: close the mouth.
void stop() => _flushes.add(audibleTime!);
}
With a real player, the rules are:
- Emit each chunk on
chunkswhen you schedule it, before it sounds, withstartTimeon the player's clock. audibleTimeis that same clock right now, minus the output latency, ornullwhile nothing plays.- Emit the cut time on
flusheswhen playback is interrupted.
example/lib/sources/web_audio_speech_source.dart
does this over Web Audio in about 150 lines.
Realtime voice models (OpenAI Realtime, Gemini Live) #
I built this to give a face to voice bots running on realtime models. Both
APIs stream the model's voice as base64 PCM16, mono, 24 kHz: hand each chunk
to your source the moment it arrives, and stop on barge-in.
pcm16ToSamples converts the bytes; the example's
WebAudioSpeechSource
wraps it as playPcm16.
Gemini Live (every server message from your WebSocket or SDK):
void onGeminiMessage(WebAudioSpeechSource source, Map<String, Object?> msg) {
if (msg case {'serverContent': {'interrupted': true}}) {
return source.stop(); // barge-in
}
if (msg case {
'serverContent': {'modelTurn': {'parts': final List<Object?> parts}},
}) {
for (final part in parts) {
if (part case {'inlineData': {'data': final String audio}}) {
unawaited(source.playPcm16(base64Decode(audio), 24000));
}
}
}
}
OpenAI Realtime (WebSocket transport; over WebRTC the audio arrives as a media track and its samples are not available):
void onOpenAIEvent(WebAudioSpeechSource source, Map<String, Object?> event) {
switch (event) {
// 'response.audio.delta' is the beta API's name for the same event.
case {
'type': 'response.output_audio.delta' || 'response.audio.delta',
'delta': final String audio,
}:
unawaited(source.playPcm16(base64Decode(audio), 24000));
case {'type': 'input_audio_buffer.speech_started'}: // the user talks
source.stop();
}
}
Both are in example/lib/realtime_recipes.dart.
Tips:
- Schedule each chunk the moment it arrives, not when it plays.
- On barge-in, also tell the model how much was heard (OpenAI:
conversation.item.truncate), so its transcript matches what was played. - Mount the avatar before the first chunk arrives.
2. Show a face #
A drawn face, on every platform:
LipSyncAvatar(
source: source,
colors: const LipSyncAvatarColors(
face: Color(0xFFF2D3B3),
ink: Color(0xFF3B2A20),
mouth: Color(0xFF6B1F2A),
),
size: 200,
)
A photo, on the web. Use a frontal face with the mouth closed, declared as an
asset in your pubspec.yaml:
PhotoLipSyncAvatar(
source: source,
imageAsset: 'assets/face.jpg',
// Shown off the web, or if no face is found.
fallback: LipSyncAvatar(source: source, colors: myColors),
)
3. Drive your own character #
Both widgets call onFrame with a LipSyncFrame on every animation frame:
jaw, gap, width, round, press, level, speaking and the dominant
viseme. Use them to drive a Rive state machine, sprites or a 3D rig:
LipSyncAvatar(
source: source,
colors: myColors,
onFrame: (frame) {
visemeInput.value = frame.viseme.index.toDouble();
openInput.value = frame.mouthOpen;
},
)
To skip the widgets entirely, run LipSyncDriver(source) and call
tick(dt) from your own Ticker.
Example #
example/ is a Flutter web app with synthetic speech, audio file
playback, and both faces:
cd example && flutter run -d chrome
Platforms #
| Feature | Web | Android, iOS, desktop |
|---|---|---|
Speech analysis, LipSyncFrame, onFrame |
✅ | ✅ |
LipSyncAvatar (drawn face) |
✅ | ✅ |
PhotoLipSyncAvatar (photo rig) |
✅ | Shows fallback |
The photo rig loads MediaPipe Tasks Vision 0.10.14 and its face model from
public CDNs the first time a photo is rigged, so the page needs network
access to cdn.jsdelivr.net and storage.googleapis.com.
How it works #
LipSyncSource ──chunks──▶ SpeechTimeline ──▶ LipSyncAnalyzer ──▶ LipSyncFrame
(your player) FFT per 10 ms: jaw, lips, viseme │
level, formants, springs, 30 ms lead ▼
hiss, nasal, flux LipSyncAvatar /
PhotoLipSyncAvatar
(+ AvatarMotion, FaceRig)
SpeechTimelineanalyses each scheduled chunk (25 ms Hann window, 10 ms hop, FFT): level, first and second formant estimates, hiss, nasal energy and spectral flux.LipSyncAnalyzerreads the timeline 30 ms ahead of what is heard. Vowel shapes are soft blends normalised to the speaker, shapes coarticulate, stop closures are anticipated, and the jaw and lip gap move as springs.FaceRigbuilds a Delaunay mesh over the photo with weight fields for the jaw, lips, corners, eyelids, brows, irises and head, andPhotoLipSyncAvatardraws it withCanvas.drawVertices.
Contributing #
flutter pub get
flutter analyze
flutter test
Tuning tools (instructions in each file header):
test/src/viseme_calibration_test.dartruns a WAV file or a labelled consonant corpus through the pipeline and prints a report.test/src/photo_preview_test.dartrenders mouth shapes on the photo to PNG.
Working with an AI coding agent? See AGENTS.md.
Credits #
test/fixtures/face.jpgandexample/assets/face.jpg: "Portrait de Blanche Angers, actrice" (Bacard fils, Paris, c. 1880), Paris Musées / Musée Carnavalet, CC0, via Wikimedia Commons.- MediaPipe Tasks Vision (Apache 2.0) is loaded from a CDN at runtime, not bundled.
License #
MIT. See LICENSE.