avatar_lipsync library
Real-time lip sync for Flutter: a talking avatar driven by streamed speech.
Implement a LipSyncSource over your audio playback. SpeechTimeline
analyses each chunk as it is scheduled and LipSyncAnalyzer turns it into
smoothed LipSyncFrames. Show them with LipSyncAvatar (a drawn face, any
platform) or PhotoLipSyncAvatar (a rigged photo, web), or feed them to
your own character through onFrame.
Classes
- AvatarMotion
- The living part of the avatar, from conversation research:
- AvatarPose
- Everything about the face that is not the mouth, for one frame.
- FaceLandmarkDetector
- Finds the face points of an image.
- FaceLandmarks
- The face points of one image, in image pixels.
- FaceMeshIndex
- MediaPipe face-mesh indices the rig uses.
- FaceRig
- A photo turned into a deformable mesh, driven by mouth frames and poses.
- LipSyncAnalyzer
- Turns the speech timeline into mouth frames.
- LipSyncAvatar
- A simple drawn face whose mouth follows source in real time.
- LipSyncAvatarColors
- Colors of the drawn face.
- LipSyncDriver
- Connects a LipSyncSource to a LipSyncAnalyzer.
- LipSyncFrame
- One animation frame of the mouth, already smoothed.
- LipSyncSource
- The speech the avatar lip-syncs to.
- MouthPalette
- Colors for the inside of the mouth, taken from the photo itself so the cavity and teeth match its lighting, white balance and skin tone.
- PhotoLipSyncAvatar
- A photo of a face that talks: the mouth follows source in real time and the face blinks, breathes, nods and glances like a person on a call.
- RigOverlay
-
Draws how the photo rig works on top of a
PhotoLipSyncAvatar: the triangle mesh, the face landmarks and a heat map. - SpeechChunk
- A chunk of speech and when it will be heard.
- SpeechFeatures
- What the voice is doing during one 25 ms window, centred at time.
- SpeechTimeline
- Speech features along the audio clock, computed as chunks are scheduled.
Enums
- RigRegion
- What a RigOverlay heat map shows.
- Viseme
- The mouth shapes the avatar blends between.
Functions
-
paintRigOverlay(
Canvas canvas, FaceRig rig, Float32List positions, RigOverlay overlay, double scale) → void -
Paints
overlayforrigposed atpositions, in image coordinates (scaleis the image-to-screen scale, to keep strokes a constant size). -
pcm16ToSamples(
Uint8List bytes) → Float32List - Converts little-endian 16-bit PCM to samples in -1..1.