avatar_lipsync library

Real-time lip sync for Flutter: a talking avatar driven by streamed speech.

Implement a LipSyncSource over your audio playback. SpeechTimeline analyses each chunk as it is scheduled and LipSyncAnalyzer turns it into smoothed LipSyncFrames. Show them with LipSyncAvatar (a drawn face, any platform) or PhotoLipSyncAvatar (a rigged photo, web), or feed them to your own character through onFrame.

Classes

AvatarMotion
The living part of the avatar, from conversation research:
AvatarPose
Everything about the face that is not the mouth, for one frame.
FaceLandmarkDetector
Finds the face points of an image.
FaceLandmarks
The face points of one image, in image pixels.
FaceMeshIndex
MediaPipe face-mesh indices the rig uses.
FaceRig
A photo turned into a deformable mesh, driven by mouth frames and poses.
LipSyncAnalyzer
Turns the speech timeline into mouth frames.
LipSyncAvatar
A simple drawn face whose mouth follows source in real time.
LipSyncAvatarColors
Colors of the drawn face.
LipSyncDriver
Connects a LipSyncSource to a LipSyncAnalyzer.
LipSyncFrame
One animation frame of the mouth, already smoothed.
LipSyncSource
The speech the avatar lip-syncs to.
MouthPalette
Colors for the inside of the mouth, taken from the photo itself so the cavity and teeth match its lighting, white balance and skin tone.
PhotoLipSyncAvatar
A photo of a face that talks: the mouth follows source in real time and the face blinks, breathes, nods and glances like a person on a call.
RigOverlay
Draws how the photo rig works on top of a PhotoLipSyncAvatar: the triangle mesh, the face landmarks and a heat map.
SpeechChunk
A chunk of speech and when it will be heard.
SpeechFeatures
What the voice is doing during one 25 ms window, centred at time.
SpeechTimeline
Speech features along the audio clock, computed as chunks are scheduled.

Enums

RigRegion
What a RigOverlay heat map shows.
Viseme
The mouth shapes the avatar blends between.

Functions

paintRigOverlay(Canvas canvas, FaceRig rig, Float32List positions, RigOverlay overlay, double scale) → void
Paints overlay for rig posed at positions, in image coordinates (scale is the image-to-screen scale, to keep strokes a constant size).
pcm16ToSamples(Uint8List bytes) → Float32List
Converts little-endian 16-bit PCM to samples in -1..1.