avatar_lipsync 0.1.0
avatar_lipsync: ^0.1.0 copied to clipboard
Real-time lip sync for Flutter. Turns streamed speech into mouth shapes and animates a drawn face or a single photo (web, via MediaPipe).
avatar_lipsync #
Real-time lip sync for Flutter: a talking avatar driven by streamed speech. Show a drawn face on any platform, or bring a single frontal photo to life on the web.
- Works ahead of the sound. Each audio chunk is analysed when it is scheduled, before it is heard, so the mouth is never late.
- A real mouth model. The jaw follows loudness and the lips follow articulation (JALI): vowel shapes, rounding, lip closures for m/b/p, and springs for natural motion. It adapts to each speaker's voice.
- One photo is enough.
PhotoLipSyncAvatarfinds 478 face points with MediaPipe (loaded in the browser at runtime) and animates the photo as a mesh, with teeth and tongue painted in colors taken from the photo. - Alive. It blinks, sways, nods, raises its brows and shifts its gaze, and respects the reduced-motion setting.
- Any audio stack. You connect your player through one small interface. The package itself does not play audio.
Install #
flutter pub add avatar_lipsync
Quick start #
1. Tell the avatar what is being heard #
Implement LipSyncSource over your audio player. This is the smallest one:
it keeps time with a Stopwatch and is enough to try the package.
import 'dart:async';
import 'dart:typed_data';
import 'package:avatar_lipsync/avatar_lipsync.dart';
class ClockSpeechSource implements LipSyncSource {
final _clock = Stopwatch()..start();
final _chunks = StreamController<SpeechChunk>.broadcast();
final _flushes = StreamController<double>.broadcast();
@override
Stream<SpeechChunk> get chunks => _chunks.stream;
@override
Stream<double> get flushes => _flushes.stream;
@override
double? get audibleTime => _clock.elapsedMicroseconds / 1e6;
/// Mono samples in -1..1, starting now.
void speak(Float32List samples, int sampleRate) => _chunks.add(
SpeechChunk(
samples: samples,
startTime: audibleTime! + 0.05,
sampleRate: sampleRate,
),
);
/// The user interrupted: close the mouth.
void stop() => _flushes.add(audibleTime!);
}
With a real player, the rules are:
- Emit each chunk on
chunkswhen you schedule it, before it sounds, withstartTimeon the player's clock. audibleTimeis that same clock right now, minus the output latency, ornullwhile nothing plays.- Emit the cut time on
flusheswhen playback is interrupted.
example/lib/sources/web_audio_speech_source.dart
does this over Web Audio in about 100 lines.
2. Show a face #
A drawn face, on every platform:
LipSyncAvatar(
source: source,
colors: const LipSyncAvatarColors(
face: Color(0xFFF2D3B3),
ink: Color(0xFF3B2A20),
mouth: Color(0xFF6B1F2A),
),
size: 200,
)
A photo, on the web. Use a frontal face with the mouth closed, declared as an
asset in your pubspec.yaml:
PhotoLipSyncAvatar(
source: source,
imageAsset: 'assets/face.jpg',
// Shown off the web, or if no face is found.
fallback: LipSyncAvatar(source: source, colors: myColors),
)
3. Drive your own character #
Both widgets call onFrame with a LipSyncFrame on every animation frame:
jaw, gap, width, round, press, level, speaking and the dominant
viseme. Use them to drive a Rive state machine, sprites or a 3D rig:
LipSyncAvatar(
source: source,
colors: myColors,
onFrame: (frame) {
visemeInput.value = frame.viseme.index.toDouble();
openInput.value = frame.mouthOpen;
},
)
To skip the widgets entirely, run LipSyncDriver(source) and call
tick(dt) from your own Ticker.
Example #
example/ is a Flutter web app with synthetic speech, audio file
playback, and both faces:
cd example && flutter run -d chrome
Platforms #
| Feature | Web | Android, iOS, desktop |
|---|---|---|
Speech analysis, LipSyncFrame, onFrame |
✅ | ✅ |
LipSyncAvatar (drawn face) |
✅ | ✅ |
PhotoLipSyncAvatar (photo rig) |
✅ | Shows fallback |
The photo rig loads MediaPipe Tasks Vision 0.10.14 and its face model from
public CDNs the first time a photo is rigged, so the page needs network
access to cdn.jsdelivr.net and storage.googleapis.com.
How it works #
LipSyncSource ──chunks──▶ SpeechTimeline ──▶ LipSyncAnalyzer ──▶ LipSyncFrame
(your player) FFT per 10 ms: jaw, lips, viseme │
level, formants, springs, 30 ms lead ▼
hiss, nasal, flux LipSyncAvatar /
PhotoLipSyncAvatar
(+ AvatarMotion, FaceRig)
SpeechTimelineanalyses each scheduled chunk (25 ms Hann window, 10 ms hop, FFT): level, first and second formant estimates, hiss, nasal energy and spectral flux.LipSyncAnalyzerreads the timeline 30 ms ahead of what is heard. Vowel shapes are soft blends normalised to the speaker, shapes coarticulate, stop closures are anticipated, and the jaw and lip gap move as springs.FaceRigbuilds a Delaunay mesh over the photo with weight fields for the jaw, lips, corners, eyelids, brows, irises and head, andPhotoLipSyncAvatardraws it withCanvas.drawVertices.
Contributing #
flutter pub get
flutter analyze
flutter test
Tuning tools (instructions in each file header):
test/src/viseme_calibration_test.dartruns a WAV file or a labelled consonant corpus through the pipeline and prints a report.test/src/photo_preview_test.dartrenders mouth shapes on the photo to PNG.
Working with an AI coding agent? See AGENTS.md.
Credits #
test/fixtures/face.jpgandexample/assets/face.jpg: "Portrait de Blanche Angers, actrice" (Bacard fils, Paris, c. 1880), Paris Musées / Musée Carnavalet, CC0, via Wikimedia Commons.- MediaPipe Tasks Vision (Apache 2.0) is loaded from a CDN at runtime, not bundled.
License #
MIT. See LICENSE.