utd_video_effects_kit

Real-time video filters & beauty effects for LiveKit-based Flutter apps — modeled on a native beauty-effects plugin, but built on flutter_webrtc + native GPU pipelines so it integrates with utd_live_room_kit.

The public surface is one class — VideoEffectsProcessor — which is a livekit_client TrackProcessor<VideoProcessorOptions>. That means it drops into any LocalVideoTrack capture with zero glue.

Status: v0.8.0 — the full effect stack ships on both platforms; the 0.8.0 deep-audit hardening pass is in (leaks, races, makeup correctness, parity), and the newly-defaulted paths (iOS bilateral smoothing, unified effect order) still await one on-device A/B via the example app's Diagnostics screen. The native in-place pipeline implements:

• Color / beauty (no landmarks): 58 LUT color filters (with intensity), frequency-separation skin smoothing (+ texture detail), whitening, skin-tone presets (dual-LUT, both platforms), signed skin-tone shift, clarity, and the color-post tail — glow / grain / vignette — plus 12 one-tap presets.
• Background blur: MediaPipe selfie segmentation → blur background, keep the person sharp (feathered, temporally smoothed mask).
• Face effects (MediaPipe FaceLandmarker, async/cached): eye color, makeup (lipstick + finish/style, two-tone eyeshadow + shimmer, blush, brows, eyeliner, contour, highlight), retouch (teeth / eye-white / under-eye), reshape (liquify warp), and nine PNG accessories.
• Body pose (optional): MediaPipe PoseLandmarker, 33 landmarks, pull-based.

Platforms:

• Android — CPU/I420 pipeline (CpuEffects + EffectsVideoProcessor): decode → color grade → background blur → makeup/accessories (Canvas) → reshape warp → glow/grain/vignette → re-encode.
• iOS — CoreImage chain (CIColorCubeWithColorSpace + custom CIKernels + CIBlendWithMask blur) then Core Graphics face overlays, in an ExternalVideoProcessingDelegate, in the same canonical effect order.

The face/blur/pose pass bundles four Apache-2.0 MediaPipe models under assets/models/ and nine transparent accessory PNGs under assets/accessories/ (realistic 3D renders). MediaPipe raises the Android minSdk to 24. Detection runs async off the capture thread; overlays draw from the last cached, motion-extrapolated result.

Why a TrackProcessor (and not a process(VideoFrame) callback)

The shipped livekit_client 2.7/2.8 TrackProcessor interface has no per-frame hook. It exposes lifecycle — init / restart / destroy / onPublish / onUnpublish — plus a processedTrack getter. LiveKit swaps in processedTrack only when it is non-null; otherwise the source track flows unchanged. So:

• All pixel work is native. This Dart object is a lifecycle + control wrapper.
• A null processedTrack is a safe passthrough — never a broken stream.

Usage

Standalone

import 'package:livekit_client/livekit_client.dart';
import 'package:utd_video_effects_kit/utd_video_effects_kit.dart';

final fx = VideoEffectsProcessor.create();

final track = await LocalVideoTrack.createCameraTrack(
  CameraCaptureOptions(processor: fx),
);

// Control effects at any time (cheap native calls once the pipeline lands):
await fx.setEnabled(true);
await fx.setSmoothing(0.6);
await fx.setWhitening(0.2);
await fx.setFilter('warm', intensity: 0.8);
await fx.setBackgroundBlur(0.4);

// Face effects (MediaPipe FaceLandmarker). A null color clears the effect.
await fx.setEyeColor('#3366CC', opacity: 0.6);
await fx.setLipstick('#CC2244');
await fx.setEyeshadow('#8844AA', opacity: 0.4);
await fx.setBlush('#E87070');
await fx.setGlasses(true);    // also: setCrown / setCatEars
await fx.setGlasses(false);   // toggle off
await fx.setLipstick(null);   // clear makeup

// Bind UI to the live state:
ValueListenableBuilder<EffectsState>(
  valueListenable: fx.state,
  builder: (_, s, __) => Slider(value: s.smoothing, onChanged: fx.setSmoothing),
);

// Save / restore a whole look (0.8.0): EffectsState round-trips through a map,
// and apply() pushes a complete state in ONE native call.
final saved = fx.state.value.toMap();
await fx.apply(EffectsState.fromMap(saved));

With utd_live_room_kit

The kit takes a factory (not an instance) because LiveKit destroys the processor on every restartTrack (camera flip / reconnect):

UTDLiveRoom(
  // …,
  config: UTDLiveRoomConfig(
    buildVideoProcessor: () => VideoEffectsProcessor.create(),
  ),
);

The kit attaches the processor to the host self-preview and to the SDK's default camera options, so it survives camera switch, preview→go-live, reconnect, and guest go-live. Rendering needs no changes — VideoTrackRenderer shows processedTrack automatically.

To toggle effects at runtime you usually call the processor's own setters (e.g. setEnabled(false)); to attach/detach the processor on the live track entirely, the kit also exposes controller.mediaController.setVideoProcessor(...).

One-tap filter presets (0.7.0)

VideoEffectsPresets.all is a catalog of 12 curated TikTok-style looks — each a stack of LUT + intensity + smoothing + clarity + glow + grain + vignette applied atomically:

await fx.applyPreset(VideoEffectsPresets.byKey('clean_glow'));
await fx.applyPreset(null); // reset the preset-owned fields

VideoEffectsSheet gets a leading Presets tab with preview chips; makeup / accessories / reshape are never touched by a preset.

Body-pose tracking (0.7.0, optional)

setPoseTracking(true) runs MediaPipe PoseLandmarker (33 landmarks, bundled lite model, off by default) behind the same async/coast discipline as face tracking. Poll poseLandmarks() → PoseFrame {xy, vis, confidence} for skeleton overlays or gesture logic; nothing is drawn natively.

Diagnostics screen (example app)

The example app's Diagnostics page (AppBar → troubleshoot icon) validates every device-gated toggle in one place: render-parity test frame, perf capture with a GO/NO-GO verdict, head-pose sign check, iOS colour check, segmentation mask visualizer, GPU-delegate latency A/B, and the pose skeleton. Run it once per platform after upgrading; flip the documented defaults it green-lights.

Method-channel contract (utd_video_effects_kit/control)

Method	Args	Returns
isSupported	—	bool
attach	{trackId, state}	{supported, sessionId, tier}
detach	{sessionId}	—
setEffects	{sessionId, state}	—
getPerfStats	{sessionId, reset}	perf snapshot map (5.1)
renderTestFrame	{sessionId, rgba, width, height}	graded rgba or null (4.2)
getPoseLandmarks	{sessionId}	{xy, vis, count, tsMs, confidence} or null
setDebugFlags	{sessionId, flags}	— (diagnostics)
getDebugInfo	{sessionId}	tracking diagnostics map
getSegMask	{sessionId}	{person, width, height, rotation} or null

state is EffectsState.toMap(). tier is the native device performance tier (0 Budget … 3 Flagship) surfaced as VideoEffectsProcessor.deviceTier. A companion EventChannel (utd_video_effects_kit/status) pushes frame-budget degrade events natively → Dart as {sessionId, exceeded, level}, surfaced as budgetExceeded / degradeLevel.

Dart API surface (VideoEffectsProcessor)

Every member is dartdoc'd in lib/src/video_effects_processor.dart; this is the map. All setters are cheap native calls; ranges are 0.0..1.0 unless noted.

Lifecycle & state

Member	What it does
VideoEffectsProcessor.create({initial, entitled})	Build the native-backed processor (entitled gates all pixel work)
state	ValueListenable<EffectsState> — live settings snapshot (bind UI)
apply(EffectsState)	Bulk-apply a complete state in ONE native push (pairs with EffectsState.fromMap)
isSupported / entitled	Passthrough gates (platform capability / paid activation)
setEnabled(bool)	Master on/off
dispose()	Final teardown (distinct from LiveKit's per-capture destroy)

Color post & beauty

Member	What it does
setSmoothing / setTextureDetail	Frequency-separation skin smoothing / pore texture retained
setWhitening	Brightness lift
setFilter(key, intensity:)	LUT color grade (58-filter catalog), blended 0..1
applyPreset(FilterPreset?)	One-tap preset — LUT + smoothing + clarity + glow + grain + vignette + whitening atomically
setBackgroundBlur	Segmentation-masked background blur
setGlow / setGrain / setVignette	Color-post tail: Orton bloom, animated film grain, radial vignette
setClarity	Signed -1..1 local contrast (+punchy / −soft)

Retouch & skin

Member	What it does
setTeethWhitening	Desaturate + lift inside the inner-lip opening
setEyeBrightening	Sclera brighten/de-redden (blink-gated)
setUnderEyeBrightening	Tear-trough lift (face-rotation-aware ellipse)
setSkinColor(presetKey)	Dual-LUT skin-tone preset (VideoEffectsSkinTones, both platforms)
setSkinToneShift	Signed -1..1 warm/cool hue rotation, skin-masked
setSegmentationSkin(bool)	Opt-in multiclass face-skin mask for skin effects (bigger model)

Makeup (null color clears; setMakeupLook sets-or-clears every field)

Member	What it does
setEyeColor(hex, opacity:)	Iris recolor
setLipstick(hex, opacity:, finish:, style:)	matte/satin/gloss finish; full/ombre/blurred style
setEyeshadow(hex, opacity:, color2:, shimmer:)	Two-tone gradient + shimmer
setBlush(hex, opacity:)	Cheek tint
setBrows(hex, opacity:)	Hair-gated brow fill
setEyeliner(hex, opacity:, style:)	classic or wing
setContour / setHighlight	Dodge/burn sculpt (cheek/nose burn; high-point sheen)
setMakeupLook(MakeupLook?)	Whole coordinated look in one call

Reshape (landmark-driven liquify warp)

Member	What it does
setEyeEnlarge / setFaceSlim / setNoseNarrow / setLipPlump / setChinShorten	Per-feature warp strengths

Accessories (bool toggles)

Member	What it does
setGlasses / setSunglasses / setCrown / setFlowerCrown / setPartyHat / setCatEars / setBunnyEars / setMustache / setHalo	PNG overlay on/off

Pose, telemetry & diagnostics

Member	What it does
setPoseTracking(bool) / poseLandmarks()	Optional 33-landmark body pose (pull-based PoseFrame)
budgetExceeded / degradeLevel / deviceTier	ValueListenables from the native frame-budget system
perfStats({reset})	Native frame-time telemetry snapshot (EffectsPerfStats)
renderTestFrame(rgba, width:, height:)	On-device CPU-vs-GPU parity harness (requires setEnabled(false))
setDebugFlags / debugInfo / debugSegMask	Runtime validation flags + tracking/mask readouts (Diagnostics screen)

Bundled filters & the third-party asset set

This package ships 58 LUT color filters as plain 512×512 square LUT PNGs under assets/luts/ — a standard, engine-agnostic format the DIY native LUT pass consumes directly (no third-party engine or license required). These are the filters extracted from a third-party beauty-effects resource bundle (ColorfulStyleResources + FaceWhiteningResources) plus 31 color-science looks generated in-house (teal_orange, golden_hour, noir_bw, sepia, cyberpunk, cold_brew, paris, y2k_chrome, … — see tool/lutgen.py). Every filter has a 512² preview thumbnail in assets/previews/, rendered by applying the LUT to a bundled reference portrait.

Use them via the catalog:

for (final f in VideoEffectsFilters.all) {
  // f.key ('fresh'), f.label ('Fresh'), f.asset, VideoEffectsFilters.previewFor(f.key)
}
await fx.setFilter('teal_orange'); // resolves to assets/luts/teal_orange.png natively

Makeup shades & looks

Beyond raw hex, VideoEffectsMakeup provides curated lipstick / eyeshadow / blush / eyeColor shades and 6 coordinated full looks (each with an AI preview thumbnail). Lipstick supports a finish — matte, satin, gloss — and a style — full, ombre, blurred. As of 0.8.0 a MakeupLook spans the entire makeup surface — lipstick (+finish/style), two-tone eyeshadow (+eyeshadowColor2/shimmer), blush, eye color, brows, eyeliner (+style), contour, highlight — and setMakeupLook sets-or-clears every field, so nothing from a previous custom mix leaks into an authored look:

await fx.setLipstick('#B11226', opacity: 0.7, finish: 'gloss');
await fx.setMakeupLook(VideoEffectsMakeup.lookByKey('glam')); // whole look at once
await fx.setMakeupLook(null);                                 // clear all makeup

Not bundled (proprietary-engine-only). The rest of the third-party asset set — makeup (.lua), 3D pendants/stickers (.obj + .lua), the AI .model files, and the skin-color/rosy/clarity/teeth bundles — only work with that vendor's native effects engine (a native create(appID, appSign) + setResources call). The host app deliberately removed that vendor SDK, so those are out of scope for the DIY path. To use them you'd add a native-effects-engine adapter behind VideoEffectsProcessor and re-introduce the vendor SDK + license (a v2 "buy" decision — see PLAN.md §2/§6). Skin smoothing / whitening / background blur in this package are DIY shader effects and need no asset.

Face effects: models & accessories

The face/blur/pose pass bundles four Apache-2.0 MediaPipe models under assets/models/ — face_landmarker.task (~3.8 MB), selfie_segmenter.tflite (~250 KB), selfie_multiclass.tflite (~16.4 MB, only loaded when setSegmentationSkin(true)), pose_landmarker_lite.task (~5.8 MB, only loaded when setPoseTracking(true)) — and nine PNG accessories under assets/accessories/ (transparent, realistic 3D renders): glasses, crown, cat_ears, sunglasses, flower_crown, party_hat, bunny_ears, mustache, halo. To swap art, replace the PNGs in place (same names) — each is a transparent, front-facing, horizontally-centered render. Use the catalog:

for (final a in VideoEffectsAccessories.all) {
  // a.key ('glasses'), a.label ('Glasses'), a.asset (Image.asset path)
}
await fx.setGlasses(true);    // also: setCrown / setCatEars / setSunglasses /
                              // setFlowerCrown / setPartyHat / setBunnyEars /
                              // setMustache / setHalo

Build & validate (on device)

Both platforms build against the bundled MediaPipe SDKs (the example app's debug APK and iOS-simulator Runner.app compile clean). Runtime behavior still needs real hardware. The example/ runners are generated with flutter create . (prerequisite for a device run):

flutter pub get
cd example
flutter create --platforms=android,ios .   # one-time, generates android/ ios/ runners
# Android (minSdk 24 — required by MediaPipe):
flutter run                                 # real Android device — see ABI note below
# iOS (first build runs pod install: WebRTC-SDK + flutter_webrtc + MediaPipeTasksVision):
flutter run                                 # real iOS device (simulator hits the I420 fallback)

Validate: (1) host go-live still works with no effect (passthrough); (2) LUT / smoothing / whitening grade the LOCAL preview AND what remote viewers see; (3) eye color / makeup / accessories track the face and background blur masks the person — across head movement and camera flip (LiveKit re-runs init, and the providers close()/re-init cleanly with no leak); (4) overlays track the face in portrait + landscape (MediaPipe back-projects landmarks to the original image, so FaceEffects.map is a plain scale — see NATIVE.md); (5) no fps collapse at 720p (detection is async/cached; the per-frame Bitmap/CGContext + CPU blur are the heavy steps — downscale/throttle if needed).

Android ABI requirement for the MediaPipe effects. The prebuilt MediaPipe 32-bit (armeabi-v7a) library needs aligned_alloc, which exists only on API 28+, so on a pre-Android-9 32-bit device it can't load and the face effects + background blur are a no-op (LUT / smoothing / whitening / skin-tone still work — they don't use MediaPipe). Use an arm64-v8a device to exercise them; for production ship arm64-v8a only (or minSdkVersion 28): android { defaultConfig { ndk { abiFilters 'arm64-v8a', 'x86_64' } } }.

Roadmap

• M0 — frame seam: ✅ in-place processor on the flutter_webrtc track (addProcessor / addProcessing), affects preview + encoder.
• M1 — LUT color filter: ✅ Android CPU/I420 square-LUT (CpuEffects) + iOS CIColorCube, loading assets/luts/<key>.png. (A GPU/GL revival on Android is a plan item — Phase 5.2 in IMPROVEMENT_PLAN_OSS.md, gated on measured device perf via perfStats(); GlEffects.kt is not on the production path.)
• M2 — smoothing / whitening / skin-color: ✅ Android CPU (bilateral frequency-separation smoothing, whitening lift, dual-LUT skin-tone with a YCbCr mask using assets/skin/**); iOS parity via custom CIKernels (bilateral smoothing, whiten, dual-LUT skin-tone). See NATIVE.md.
• M3 — Background blur: ✅ MediaPipe selfie segmentation + blend (Android CPU box-blur; iOS CIBlendWithMask).
• M4 — Face effects: ✅ MediaPipe FaceLandmarker (async) → eye color, makeup (lipstick / eyeshadow / blush), PNG accessories (glasses / crown / cat-ears). DIY landmark quality is "decent, not Snap-grade" (see PLAN.md §1.3). Requires Android minSdk 24.
• Perf: detection runs async off the capture thread; the per-frame Bitmap/CGContext + CPU blur are the heaviest steps — profile at 720p on low-end devices, downscale/throttle if needed.
• v2 — Buy adapter: face reshaping / realistic makeup / 3D stickers / bg replacement behind the same VideoEffectsProcessor API (Banuba / Tencent XMagic / DeepAR).

See PLAN.md for the full architecture, native frame-format handling, the shader pipeline, build-vs-buy analysis, and milestone estimates.

Platforms

Android + iOS (mobile). Web returns isSupported == false (passthrough).