utd_video_effects_kit 0.3.0

Real-time video filters and beauty effects (LUT color grading, skin smoothing, whitening, background blur, plus MediaPipe face effects — eye color, makeup, accessories) for LiveKit Flutter apps, as a [...]

Changelog

0.3.0 (more accessories, LUTs, curated makeup looks)

• Accessory art upgraded to realistic 3D renders. Replaced the placeholder glasses / crown / cat_ears PNGs with transparent, high-quality renders (generated via Vertex AI Imagen, matted with rembg).
• Six new accessories: sunglasses, flowerCrown, partyHat, bunnyEars, mustache, halo. Each is wired end-to-end:
  • Dart: EffectsState toggles + VideoEffectsProcessor setters (setSunglasses, setFlowerCrown, setPartyHat, setBunnyEars, setMustache, setHalo) + VideoEffectsAccessories catalog entries.
  • Native Android/iOS: lazy-loaded bitmaps and landmark-anchored draw passes (e.g. mustache at the philtrum, halo floating above the crown of the head).
• Calibration note: the new accessories ship with sensible widthScale / topFrac defaults mirroring the existing ones; fine-tune per device in FaceEffects.draw* if needed.
• 16 new color filters (LUTs) generated with color science as clean 64-level square LUTs (correct on both the Android GL and iOS CIColorCube paths): teal_orange, golden_hour, sunset_glow, vibrant_pop, crisp_clean, vintage_warm, faded_film, matte_fade, moody_blue, cold_winter, dreamy_pink, emerald, cyberpunk, sepia, noir_bw, silver_bw (filter catalog now 43).
• All filter previews re-rendered at 512² (was 256²) from a new high-quality reference portrait (Imagen), including the 5 skin-tone previews (dual-LUT skin mask). Existing LUTs were audited — all valid; none rebuilt (infrared white-clips by design).
• Curated makeup catalog (VideoEffectsMakeup): realistic lipstick / eyeshadow / blush / eye-color shades plus 6 coordinated full looks (natural, glam, soft_rose, bold_berry, sunkissed, smoky_night), each with an AI-rendered preview thumbnail. Apply a whole look in one call with VideoEffectsProcessor.setMakeupLook.
• More realistic makeup rendering: new lip finish (matte / satin / gloss, default satin) via setLipstick(..., finish:) + EffectsState.lipFinish, wired through Android & iOS drawLips; softer Gaussian-style blush falloff on both platforms (removes the visible ring).

0.2.0 (face effects — eye color, makeup, accessories, background blur)

• New face-effect pass driven by MediaPipe FaceLandmarker (478 landmarks), adapted into the existing in-place TrackProcessor frame seam (no per-frame bytes cross the method channel):
  • Eye color — iris recolor (landmarks 468–477) with a soft radial falloff.
  • Makeup — lipstick / eyeshadow / blush (landmark-region fills + gradients).
  • Accessories — glasses / crown / cat-ears PNG overlays, scaled/rotated from the temple + nose landmarks.
• Background blur finally implemented (the long-declared backgroundBlur field): MediaPipe selfie segmentation → blur the background, keep the person sharp. Android = CPU downscale box-blur over the frame; iOS = CIBlendWithMask.
• Detection runs async (LIVE_STREAM) off the capture thread with a cached, atomically-published landmark/mask snapshot — onFrame runs under the lock that gates frame delivery, so it never blocks the stream (≈1 frame of lag).
• Dart API (additive, backward compatible): flat EffectsState fields (eyeColor/lipstick/eyeshadow/blush + *Opacity, glasses/crown/ catEars) and VideoEffectsProcessor setters (setEyeColor, setLipstick, setEyeshadow, setBlush, setGlasses, setCrown, setCatEars). New VideoEffectsAccessories catalog.
• Assets: bundles the Apache-2.0 MediaPipe models (assets/models/ face_landmarker.task + selfie_segmenter.tflite, ≈4 MB) and placeholder accessory art (assets/accessories/*.png — replace with production artwork).
• Android minSdk raised 21 → 24 (required by MediaPipe Tasks Vision).
• Graceful degradation: if the MediaPipe native lib can't load (e.g. a pre-API-28 32-bit armeabi-v7a device, where its .so needs aligned_alloc), the providers set a permanent failed flag — face effects / background blur become a clean no-op (no crash, no per-frame retry) while LUT / smoothing / whitening / skin-tone keep working. Use an arm64-v8a device (or API 28+) for the MediaPipe effects; for production ship arm64-only or minSdkVersion 28.
• On-device fixes (Android): overlay rotation — FaceEffects.map() is now a plain normalized→pixel scale (MediaPipe Tasks already back-projects landmarks to the original image; the earlier inverse-rotation made overlays appear rotated/ offset). Builds on both platforms (example debug APK + iOS-simulator app); remaining items to validate: framerate at 720p and seg-mask edges (see NATIVE.md).

0.1.0 (GPU pipeline + M2)

• Android: zero-copy OpenGL ES pipeline (GlEffects + EffectsVideoProcessor) replacing the CPU/I420 path — OES→FBO passes, per-frame output texture, holds 30fps. Effects: square-LUT color filter, bilateral skin smoothing, whitening lift, and skin-tone (dual-LUT with a YCbCr skin mask).
• iOS M2: CoreImage chain extended with Gaussian/dissolve smoothing + brightness whitening (skin-tone deferred — needs a CoreImage skin mask).
• Dart: EffectsState.skinColor + VideoEffectsProcessor.setSkinColor + VideoEffectsSkinTones catalog (5 presets). SkinColor textures now declared as assets (assets/skin/**).
• Android Kotlin verified to compile in the app Gradle context.

0.1.0 (native LUT)

• Native in-place frame seam: registers an ExternalVideoFrameProcessing (Android) / ExternalVideoProcessingDelegate (iOS) on the flutter_webrtc LocalVideoTrack (found by id via FlutterWebRTCPlugin.sharedSingleton), so effects appear in both the local preview and the encoded stream. processedTrack stays null by design.
• LUT color filters implemented: Android CPU/I420 LUT; iOS CoreImage CIColorCubeWithColorSpace. Loads assets/luts/<key>.png (512² square LUTs).
• 13 bundled filters (ColorfulStyle + whitening/rosy/clarity) from the ZEGO resource set (LUTs only — no ZEGO engine).
• Android plugin verified to compile in the app Gradle context (webrtc-sdk 144.7559.01 + auto-wired :flutter_webrtc). iOS pending on-device build.
• SkinColor textures staged under assets/skin/** (blend documented in NATIVE.md; shader not yet implemented).

0.1.0 (scaffold)

• Initial scaffold.
• Public Dart API: VideoEffectsProcessor (a livekit_client TrackProcessor<VideoProcessorOptions>), EffectsState, and the utd_video_effects_kit/control method-channel contract.
• Safe-passthrough behavior: until the native GPU pipeline lands, isSupported is false and processedTrack is null, so the original camera frames flow unmodified and video always publishes.
• Android (Kotlin) + iOS (Swift) plugin scaffolds implementing the channel contract as passthrough, with documented Milestone-0/1 TODOs.
• Integrates with utd_live_room_kit via its UTDLiveRoomConfig.buildVideoProcessor factory — no dependency from the kit onto this package.