utd_video_effects_kit 0.2.0

Real-time video filters and beauty effects (LUT color grading, skin smoothing, whitening, background blur, plus MediaPipe face effects — eye color, makeup, accessories) for LiveKit Flutter apps, as a [...]

utd_video_effects_kit

Real-time video filters & beauty effects for LiveKit-based Flutter apps — modeled on ZEGOCLOUD's zego_effects_plugin, but built on flutter_webrtc + native GPU pipelines so it integrates with utd_live_room_kit.

The public surface is one class — VideoEffectsProcessor — which is a livekit_client TrackProcessor<VideoProcessorOptions>. That means it drops into any LocalVideoTrack capture with zero glue.

Status: effects implemented (v0.2.0); needs on-device validation. The Dart API, method-channel contract, and utd_live_room_kit integration are complete. The native in-place pipeline implements:

• Color / beauty (no landmarks): LUT color filters + skin smoothing + whitening + skin-tone (skin-tone is Android-only).
• Background blur: MediaPipe selfie segmentation → blur background, keep the person sharp (Android CPU box-blur; iOS CIBlendWithMask).
• Face effects (MediaPipe FaceLandmarker, async/cached): eye color, makeup (lipstick / eyeshadow / blush), and PNG accessories (glasses / crown / cat-ears).

Platforms:

• Android — CPU/I420 pipeline (CpuEffects + EffectsVideoProcessor): decode → color grade → background blur → face overlays (Canvas) → re-encode.
• iOS — CoreImage chain (CIColorCubeWithColorSpace + smoothing/whitening + CIBlendWithMask blur) then Core Graphics face overlays, in an ExternalVideoProcessingDelegate.

The new face/blur pass bundles the Apache-2.0 MediaPipe models under assets/models/ and placeholder accessory art under assets/accessories/ (replace with real artwork). MediaPipe raises the Android minSdk to 24. Detection runs async off the capture thread; overlays draw from the last cached result. Validate on a device — see below (rotation sign, framerate, mask edges).

Why a TrackProcessor (and not a process(VideoFrame) callback)

The shipped livekit_client 2.7/2.8 TrackProcessor interface has no per-frame hook. It exposes lifecycle — init / restart / destroy / onPublish / onUnpublish — plus a processedTrack getter. LiveKit swaps in processedTrack only when it is non-null; otherwise the source track flows unchanged. So:

• All pixel work is native. This Dart object is a lifecycle + control wrapper.
• A null processedTrack is a safe passthrough — never a broken stream.

Usage

Standalone

import 'package:livekit_client/livekit_client.dart';
import 'package:utd_video_effects_kit/utd_video_effects_kit.dart';

final fx = VideoEffectsProcessor.create();

final track = await LocalVideoTrack.createCameraTrack(
  CameraCaptureOptions(processor: fx),
);

// Control effects at any time (cheap native calls once the pipeline lands):
await fx.setEnabled(true);
await fx.setSmoothing(0.6);
await fx.setWhitening(0.2);
await fx.setFilter('warm', intensity: 0.8);
await fx.setBackgroundBlur(0.4);

// Face effects (MediaPipe FaceLandmarker). A null color clears the effect.
await fx.setEyeColor('#3366CC', opacity: 0.6);
await fx.setLipstick('#CC2244');
await fx.setEyeshadow('#8844AA', opacity: 0.4);
await fx.setBlush('#E87070');
await fx.setGlasses(true);    // also: setCrown / setCatEars
await fx.setGlasses(false);   // toggle off
await fx.setLipstick(null);   // clear makeup

// Bind UI to the live state:
ValueListenableBuilder<EffectsState>(
  valueListenable: fx.state,
  builder: (_, s, __) => Slider(value: s.smoothing, onChanged: fx.setSmoothing),
);

With utd_live_room_kit

The kit takes a factory (not an instance) because LiveKit destroys the processor on every restartTrack (camera flip / reconnect):

UTDLiveRoom(
  // …,
  config: UTDLiveRoomConfig(
    buildVideoProcessor: () => VideoEffectsProcessor.create(),
  ),
);

The kit attaches the processor to the host self-preview and to the SDK's default camera options, so it survives camera switch, preview→go-live, reconnect, and guest go-live. Rendering needs no changes — VideoTrackRenderer shows processedTrack automatically.

To toggle effects at runtime you usually call the processor's own setters (e.g. setEnabled(false)); to attach/detach the processor on the live track entirely, the kit also exposes controller.mediaController.setVideoProcessor(...).

Method-channel contract (utd_video_effects_kit/control)

Method	Args	Returns
isSupported	—	bool
attach	{trackId, state}	{supported, sessionId, processedTrackId}
detach	{sessionId}	—
setEffects	{sessionId, state}	—

state is EffectsState.toMap().

Bundled filters & the ZEGO asset set

This package ships the LUT color filters extracted from the ZEGO Effects resource bundle (Resources/ColorfulStyleResources/* + FaceWhiteningResources) as plain 512×512 square LUT PNGs under assets/luts/. These are a standard, engine-agnostic format the DIY native LUT pass consumes directly — no third-party engine or license required.

Use them via the catalog:

for (final f in VideoEffectsFilters.all) {
  // f.key ('fresh'), f.label ('Fresh'), f.asset (Image.asset path)
}
await fx.setFilter('fresh'); // resolves to assets/luts/fresh.png natively

Bundled keys: fresh, night, autumn, brighten, sunset, cool, sweet, cozily, creamy, film_like, whitening.

Not bundled (ZEGO-engine-proprietary). The rest of the ZEGO asset set — makeup (.lua), 3D pendants/stickers (.obj + .lua), the AI .model files, and the skin-color/rosy/clarity/teeth bundles — only work with ZEGO's native Effects engine (ZegoEffects.create(appID, appSign) + setResources). The host app deliberately removed ZEGO, so those are out of scope for the DIY path. To use them you'd add a ZegoEffectsEngine adapter behind VideoEffectsProcessor and re-introduce the ZEGO SDK + license (a v2 "buy" decision — see PLAN.md §2/§6). Skin smoothing / whitening / background blur in this package are DIY shader effects and need no asset.

Face effects: models & accessories

The face pass bundles two Apache-2.0 MediaPipe models under assets/models/ (face_landmarker.task, selfie_segmenter.tflite) and three placeholder PNG accessories under assets/accessories/ (glasses.png, crown.png, cat_ears.png). The accessory art is intentionally simple — replace the PNGs in place (same names) with production artwork. Use the catalog:

for (final a in VideoEffectsAccessories.all) {
  // a.key ('glasses'), a.label ('Glasses'), a.asset (Image.asset path)
}
await fx.setGlasses(true);

Build & validate (on device)

Both platforms build against the bundled MediaPipe SDKs (the example app's debug APK and iOS-simulator Runner.app compile clean). Runtime behavior still needs real hardware. The example/ runners are generated with flutter create . (prerequisite for a device run):

flutter pub get
cd example
flutter create --platforms=android,ios .   # one-time, generates android/ ios/ runners
# Android (minSdk 24 — required by MediaPipe):
flutter run                                 # real Android device — see ABI note below
# iOS (first build runs pod install: WebRTC-SDK + flutter_webrtc + MediaPipeTasksVision):
flutter run                                 # real iOS device (simulator hits the I420 fallback)

Validate: (1) host go-live still works with no effect (passthrough); (2) LUT / smoothing / whitening grade the LOCAL preview AND what remote viewers see; (3) eye color / makeup / accessories track the face and background blur masks the person — across head movement and camera flip (LiveKit re-runs init, and the providers close()/re-init cleanly with no leak); (4) overlays track the face in portrait + landscape (MediaPipe back-projects landmarks to the original image, so FaceEffects.map is a plain scale — see NATIVE.md); (5) no fps collapse at 720p (detection is async/cached; the per-frame Bitmap/CGContext + CPU blur are the heavy steps — downscale/throttle if needed).

Android ABI requirement for the MediaPipe effects. The prebuilt MediaPipe 32-bit (armeabi-v7a) library needs aligned_alloc, which exists only on API 28+, so on a pre-Android-9 32-bit device it can't load and the face effects + background blur are a no-op (LUT / smoothing / whitening / skin-tone still work — they don't use MediaPipe). Use an arm64-v8a device to exercise them; for production ship arm64-v8a only (or minSdkVersion 28): android { defaultConfig { ndk { abiFilters 'arm64-v8a', 'x86_64' } } }.

Roadmap

• M0 — frame seam: ✅ in-place processor on the flutter_webrtc track (addProcessor / addProcessing), affects preview + encoder.
• M1 — LUT color filter: ✅ Android GL (zero-copy texture) + iOS CIColorCube, loading assets/luts/<key>.png.
• M2 — smoothing / whitening / skin-color: ✅ Android GL (bilateral smoothing, whitening lift, dual-LUT skin-tone with a YCbCr mask using assets/skin/**); iOS smoothing + whitening (skin-tone deferred). See NATIVE.md.
• M3 — Background blur: ✅ MediaPipe selfie segmentation + blend (Android CPU box-blur; iOS CIBlendWithMask).
• M4 — Face effects: ✅ MediaPipe FaceLandmarker (async) → eye color, makeup (lipstick / eyeshadow / blush), PNG accessories (glasses / crown / cat-ears). DIY landmark quality is "decent, not Snap-grade" (see PLAN.md §1.3). Requires Android minSdk 24.
• Perf: detection runs async off the capture thread; the per-frame Bitmap/CGContext + CPU blur are the heaviest steps — profile at 720p on low-end devices, downscale/throttle if needed.
• v2 — Buy adapter: face reshaping / realistic makeup / 3D stickers / bg replacement behind the same VideoEffectsProcessor API (Banuba / Tencent XMagic / DeepAR).

See PLAN.md for the full architecture, native frame-format handling, the shader pipeline, build-vs-buy analysis, and milestone estimates.

Platforms

Android + iOS (mobile). Web returns isSupported == false (passthrough).