embed_flutter 0.3.0

A powerful Flutter SDK that provides a powerful voice-enabled AI-agent functionality with real-time widget tree monitoring for screen context fetching and realtime voice communication.

Changelog

0.3.0

Tree-capture hardening, Android/backend wire parity, and multi-step mission parity. Verified against the revrag-core wire contract. Minor version bump: capture semantics, defaults and stable IDs change (see Behaviour changes).

Requires Flutter >= 3.29 / Dart >= 3.7 (SemanticsData.role).

Behaviour changes

• Missions no longer depend on dataCapture, widget visibility or the route allowlist. Mission captures and responses (find / verify / info queries / candidates) are blocked only by explicit host exclusions (disabledRoutes, an open excluded popup), as on Android. flags.dataCapture: false stops telemetry pushes only. A refused mission response is replaced by a failure frame (screen_not_capturable / payload_too_large) instead of being dropped.
• transmitOnHiddenWidget defaults to on (Android parity). Only an explicit false under flags or treePush turns it off.
• Named popup routes in disabledRoutes block capture while open, but no longer hide the FAB (popups don't change the page route). Their route-change analytics are restored.
• Screen names prefer the route name (or host-provided name) over widget inference; tabs of an IndexedStack / PageView shell are named route/tab from stable signals (ValueKey<String>, Semantics identifier, heading). Class names are never used in release builds, so --obfuscate no longer scrambles screen names, stable-ID prefixes or cache keys.
• Stable IDs changed (number-led label words no longer enter IDs and collisions get content-derived suffixes; tappable images are buttons; full-screen labelled wrappers are groups; overlay nodes live under screen.overlay.*; role and screen-name changes below). The on-device cache format is bumped to schemaVersion: 2; older screen files are discarded locally and never uploaded.
• The stale-node prune is new: a node is dropped only after it has been absent from eligible changing captures spanning 14 days; a quiet stretch, an open overlay or a partial capture never counts as absence.
• Cache uploads no longer include _firstSeen / _lastSeen / _seenCount bookkeeping or tokens.

Wire parity

• Nodes carry disabled (measured), placeholder, slider range (when the host provides min/max), parent_id, and the setText action on editable fields. Roles use Android's vocabulary (scrollable, switch, radio, navigation, modal, overlay, listitem); raw Flutter role names no longer leak.
• ui_diff keeps its delta shape: moved entries now carry path (the key the backend reads), updated entries carry subtree_text, normalized_text, layer, editable, secure, parent_id, visible_fraction, and fraction-only visibility changes are reported.
• A node is visible only when at least 5% of it is on screen.
• Snapshot frames carry app_user_id, app_version and sdk.
• verification_result carries the achieved slider value and strategy; slider failure codes match the backend/Android (invalid_params, value_out_of_range, range_unknown, set_slider_not_applied).
• Every terminal mission_status carries step_index (on succeeded, the last step), so the agent can say which chained step failed.
• continue_on_failure missions emit step_failed and keep going.
• New flags.nativeBridge gates the synthetic-pointer tap fallback.

Multi-step missions

• Cancels and newer missions take effect immediately (waits, retry delays and attempts race the cancel signal); a pre-cancelled mission no longer cancels the running one; a cancelled step emits only cancelled.
• A blind step right after a settle wait polls up to 2 s for a slow next screen before failing.
• Malformed steps fail invalid_mission; unexpected errors fail exception; terminal statuses are never evicted from the outbound queue.
• Standalone highlights (guided walkthrough) run as internal hl_ missions: full find ladder, scroll into view with align, re-find before drawing; duration_ms: 0 stays until cleared. Highlights yield while a mission runs.

Execution safety

• Taps: the synthetic pointer needs >=50% visibility and a hit test that reaches the target; a tap with no visible effect is never retried (no double side effects); agent taps are not counted as user clicks.
• set_text on a non-editable target writes only to a field BOUND to the matched label (or takes the slider path), with no focus-tap side effect; clears are verified.
• A measured-disabled target fails fast with target_disabled; the enabled twin wins ties.
• back uses the host hook / Navigator first and restores a custom tab only when nothing can pop. Retries reuse the node found on the first attempt.
• Text-match promotion never climbs past a control; find_candidates honours scope, role and forbid filters; scrolling prefers the topmost layer's vertical scroller, skips pagers, and waits for pagination.
• The stable-ID rung cross-checks text, so a shifted ID cannot tap the wrong sibling ("3 Months" vs "6 Months").

Result accuracy

A mission is reported succeeded only when the INTENDED target was acted on and verification supports it; otherwise it is failed with a specific reason (the terminal vocabulary is unchanged).

• set_text never types into a field whose own label/hint names something else ("PIN" for "Full Name") and never redirects to merely the nearest field: the field must sit in the label's own container, directly below / right of it with nothing labelled in between, or be the one field whose label agrees. Otherwise target_mismatch, nothing typed. Obscured fields are skipped unless the request asks for a PIN / password.
• Side-effect verbs (set_text, click / tap / navigate, check, select, set_slider) match whole words only ("PIN" ≠ "Pincode"), refuse an ambiguous match (ambiguous_target, candidates still sent on disambiguating) and refuse a qualified label for a write ("Name" → "Nominee Name"). Read-only verbs keep the lenient ladder.
• Taps: a tappable twin must have the same text ("Name" never taps "Change Name"); a container's only button is tapped only when it agrees with the request (target_mismatch otherwise). A stable id now naming a different number ("12 Months" for "6 Months") is not acted on. A retry re-finding by rectangle must agree on the label.
• Verification is target-scoped: a tap needs a route change, an overlay opening / closing, or a change on the target, its option group, its own subtree or its bounded container. Changes elsewhere on the page — and anything that was already changing before the tap (a baseline capture ~200 ms earlier) — no longer count. navigate needs a change of place; optional params.expect_screen must match.
• set_text verifies the requested field's VALUE (never its label), which must have changed unless it already held exactly the text (set_text_already_satisfied); formatter tolerance (case, spaces, separators, masks) is kept.
• When it cannot look, the SDK says so: verification_unavailable (one retry for idempotent verbs, none after a tap / back), check_not_applied, select_not_applied, back_had_no_observable_effect. A tap after which something changed elsewhere on screen (not on the target, and not a self-updating clock or countdown) is reported verification_unavailable with evidence unattributed_change — never success, and never "nothing happened".
• Second look, observation only. When a dispatched tap / navigate / back cannot be confirmed, the SDK re-captures once ~1.5 s later and re-runs the verifier — a late, network-driven update becomes a verified success. The action is never dispatched a second time; still unconfirmed → the original failure is reported.
• read_field on an empty field fails field_has_no_text instead of answering with the label.
• Additive wire keys: verification_result gains requested_stable_id, requested_text (label only, ≤60 chars, never for obscured fields), acted_stable_id, acted_role, redirected, match_source, verified, value_matched (a boolean, never the value); terminal mission_status gains steps_completed, steps_total, steps_skipped, steps_failed and mirrors the identity keys on failure. Inbound set_text values are redacted in the debug wire log.
• More conservative on device: taps whose only visible effect is elsewhere on the screen (a counter, a badge, a status line outside the tapped card) now report tap_had_no_observable_effect instead of success.

Icon-only controls and custom tab bars

• Small icon-only tappables are named from what they draw: an image's asset file name (ic_home_active.png → home), a Material / Cupertino icon's literal name (Icons.pie_chart → pie chart), or — for a tab — "Tab i of n". Large wrappers are never named; a real label is never overridden.
• Custom bottom-navigation rows are reported as tab (excluding Material button bars, grids and keypads) and kept first under the 15 KB budget.
• Custom tab rows report selected when exactly one item is drawn differently, else from the tab the session saw lead to the current screen.
• back after a custom-tab switch re-selects the previous tab.

Fixes

• A modal no longer hides on-screen UI outside it; an inline Dialog no longer turns its page into an overlay; sheet scrims attach to their sheet.
• Effect durations of 0 no longer crash the highlight / border / awakening animations (clamped to 1 ms).
• mission_exploration and mission responses are trimmed to fit the 15 KB transport cap instead of being dropped.
• Cache: on-screen nodes are never capped away; over-cap overlays no longer re-upload on every capture; a 200 with zero rows written is retried; a file the backend keeps rejecting is quarantined instead of blocking sync; sync enables as soon as the gate and API key allow it.
• Lifecycle flushes send a diff, not a full baseline; the active border is dismissed on stop.
• Release builds: tracker IDs no longer collapse to Instance of … (0.2.0 formats kept for strings, numbers, enums, dates and lists); WireLog and inbound frame logging are silent.
• The SDK's own panel is no longer captured as host UI.

0.2.0

Native-parity release: the Flutter SDK's config wiring, call/agent modes, back navigation, and widget UX brought in line with the Android and React Native SDKs and the revrag-core wire contract. Verified live on a physical device.

Breaking changes — migration required

• Multi-agent (flowName) support removed — React / React Native parity. One integration is one agent, so the per-flow scoping is gone:
  • embedInitialize, EmbedProvider and embedEvent (and embedTrack*) no longer take a flowName.
  • EmbedWidget.enabledRoutes changes shape from Map<String, List<String>> (flow → routes) to a flat List<String>: enabledRoutes: {'main': ['home','account']} → enabledRoutes: ['home','account'].
  • EmbedWidget.groupContinuity / groupDelays (flow-keyed maps) become the single continuity: / buttonDelay:.
  • EmbedRouteManager's flow surface (currentFlow, catalogFlow, getFlowForRoute, setCurrentFlow, flowRoutes, continuityForFlow / delayMsForFlow / delayPolicyForFlow) is replaced by a flat API (activeGroup, isRouteEnabled, groupForRoute, enabledRoutes, continuity, delayMs, delayPolicy). Config, cache and event scoping collapse to one agent; existing on-device caches stay valid (the default scope is unchanged). Multi-agent apps must split into separate integrations.
• EmbedWidget apiKey / embedUrl / primaryColor / groupInsets removed (and the EmbedButtonInset type). Credentials come from embedInitialize; the FAB position (widget_config.position) and colors (widget_config.colorPalette) come from the backend — the host no longer overrides them on the widget. Use a per-widget key only via embedInitialize.
• EmbedWidget / EmbedProvider rightPadding / bottomPadding removed. FAB position is now driven solely by the backend widget_config (right_padding / bottom_padding / widget_position) — the host no longer overrides it in code. Set the position from the dashboard instead; remove the two params from your EmbedWidget call.
• embedInitialize is now the single, all-named intake — embedInitialize(apiKey: …, appVersion: …, embedUrl: …). apiKey moved from positional to a required named parameter, and appVersion is now required (embedSetAppVersion() and the package_info_plus auto-derivation are gone).
• EmbedProvider removed. It only forwarded apiKey / embedUrl / appVersion into embedInitialize and mounted an EmbedWidget; with embedInitialize as the single intake it was a redundant wrapper. Call embedInitialize + wrap with EmbedWidget directly.
• Server-driven panels/cards now render only in workflow mode. A call the user starts from the FAB (or the inactivity auto-trigger) is a co-pilot call and drops inbound panel frames. To keep panels, start the call from host code with EmbedWidget.startCall(agentTriggerMode: AgentTriggerMode.WORKFLOW).
• other_data flags removed — no peer SDK ever read them: widget_tree_snapshot_enabled and screen_capture_enabled (widget-tree snapshot sends now gate on action_config.flags.dataCapture), and embed_route_allowlist (the platform route allowlist is now host-prop-only via EmbedWidget(usePlatformRouteAllowlist: true)). Tenants relying on the old flags must move the opt-out to dataCapture / the host prop.

New

• AgentTriggerMode (co-pilot / workflow) — host-declared at EmbedWidget.startCall(agentTriggerMode:) and latched per call; a pure resolveCallPlan decides who owns the UI, whether panels render, and whether the avatar surface opens. The enum values are CO_PILOT / WORKFLOW (identifier-for-identifier with Android), and the mode's cross-SDK wire string is lowercase copilot / workflow — serialize with mode.wire, read back with parseAgentTriggerMode(String?) (case- and separator-insensitive, mirrors Android's parser).
• Visibility groups (buttonVisibility) — React Native parity. EmbedWidget(buttonVisibility: EmbedButtonVisibilityConfig(groups: [...])) lets different sets of screens carry different FAB continuity / pre-show delay / delay-policy (EmbedButtonGroupConfig{id, screens, continuity, delayMs, delayPolicy}). Resolves per current route (first matching group wins); a group's screens also join the allow-list. Screens in no group fall back to the widget-level continuity / buttonDelay / embedButtonDelayMs.
• EmbedWidget.expandWidget() / embedExpandWidget() — programmatic inverse of collapseWidget() (emits agent_tap_to_open); startCall() does not auto-expand.
• EmbedWidget.openAvatarScreen() — programmatically open the full-screen agent surface (starts a workflow call if none is running, else re-opens the surface after a collapse).
• action_config coverage completed — dataCapture master gate, awakeningEffect / activeBorderGlow / highlighting / clickTracking kill-switches, the three themed gradients, and all 26 effects.* tuning knobs (unset knobs reproduce the built-in look exactly). New: an active-border watchdog and nudge-time FAB breathing.
• widget_config completed — position.* and agentAvatar.avatarSize merged; sizing follows the native AvatarSizing rule and bar typography scales with it. isCollapsed opening state honored once per lifetime.
• call_id latched from /token and sent flat on agent_conversation_started / _ended (Android parity).
• System-back rung — back now sweeps every reachable navigator (deepest-first, PopScope-aware) so nested / per-tab / activity-per-screen hosts no longer exhaust the ladder.

Fixes & behavior (native parity)

• Inactivity: nudgeDelay <= 0 = off; show_popup shows only with a non-empty tooltip; shouldAutoCollapseCard (captions suppress the fold and cancel a running countdown).
• Widget UX: one centered 90/60/40 expanded-bar width; awakening plays only while no call is live and collapse returns the FAB to its dock; static collapsed avatar; avatar-open icon hidden in co-pilot; ink-square and glow-bloom collapse artifacts fixed; a silent config revalidate no longer collapses an open bar; avatar cannot surface on a co-pilot call (stale-plan race closed).
• Memory (OOM guard): the ExplorationLog node union is now call-scoped (reset on AgentRuntime.stop, not only at mission start) and capped LRU to 12 screens, so it can no longer grow unbounded across calls for the life of a long-lived host process.
• Internal: lib/src/embed.dart split into focused part files; comment cleanup; CLAUDE.md and docs/CONFIG_WIRING.md added.

Fixes (device-verified)

• Orb-flight performance. The awakening overlay no longer paints a full-screen radial-gradient bloom or two rainbow sweep-shader rings per frame: the bloom is bounded to a circle and a single burst ring is drawn, with smaller blur radii. On a mid-tier device this cut janky frames from ~10% to ~6% (raster p95 22.8 ms → 16.9 ms) — the bottleneck was fill-rate/shader churn, not blur.
• Mic and speaker toggles are single-flight. Rapid mute/unmute (or speaker) taps no longer stack concurrent native setMicrophoneEnabled / setSpeakerOn calls that churned the audio session and lagged the agent's speech; a burst now collapses to one native op at a time, and the stale-isMuted read is gone.
• FAB collapse no longer dips. On call end the FAB settles straight at its dock instead of dropping ~14 px and springing back — the elasticOut landing bounce that restarted after the FAB had already docked is removed.

Previously staged (0.0.21-dev)

Behavioral change — retry policy. A tap/navigate whose SEMANTIC dispatch fired but produced zero observable evidence now fails after ONE attempt (VerificationOutcome.unknown, terminal): retrying an already-fired action re-fires it physically (double/triple submits). Synthetic-pointer fallbacks and null after-snapshots keep their retries — nothing semantic was delivered. Hosts that tuned retriesPerStep for this case should rely on the backend's failure_reason: tap_had_no_observable_effect to re-plan instead.

Source-breaking for StepVerifier subclasses. StepVerifier.verify gained an optional named parameter UiNode? resolvedTarget (the find-ladder winner the runner acted on). Custom verifier overrides must add it (and may ignore it).

Also in this release: failure_reason emitted alongside error on mission_status (the backend reads only the former); node model gains value/checked/editable/secure with capture-time (sticky) secure redaction — typed field contents no longer ride text or stable ids; find ladder edge fixes (rung-1 promotion, single-token subtree hatch, IoU floor, normalised forbid_text, container-anchored inside_text); config cache un-brick (other_data versioned + TTL + revalidate; isLive hides only on explicit false); wait clamp; stale-node tap refusal; CI + lib/core test suite (180+ tests).

0.0.20

Fixes

• Fixed agent audio dropping when a wired/Bluetooth headset is connected mid-call. The SDK no longer forces the audio route on a polling timer; it relies on LiveKit's native headset prioritization and reacts only to real device-change events.
• The inactivity tooltip no longer appears while a call is connecting/connected (previously it could pop up after the bar auto-collapsed).
• The inactivity tooltip now auto-dismisses on any user interaction (tap, scroll, drag), not only a FAB tap.

Features

• The expanded voice bar auto-collapses back to the FAB after a call connects while keeping the call live. Delay is configurable via collapsedView.collapsedTime (seconds, default 15s).
• Added a "live call" pulse around the collapsed FAB that appears only after the bar fully collapses and only while the user or agent is speaking. Color is configurable via colorPalette.pulseColor (default purple #B391F3).
• The default avatar fallback now uses a Lottie animation instead of the Material chat/person icons.

Tweaks

• Reduced the expanded bar height, Connect-button padding, and mute/end-call button sizes.
• Bumped the SDK version reported to the backend (sdk_version) to 0.0.20.

0.0.19

Fixes

• Relaxed the livekit_client constraint from the exact pin 2.6.5 back to ^2.5.3. The exact pin transitively forced meta ^1.17.0 (required by livekit_client 2.6.5), which broke pub get for apps on older Flutter SDKs that pin meta 1.16.0. The range lets pub select a compatible livekit_client version per app.

0.0.18

FAB pre-show delay & per-flow delay policy

• Added embedButtonDelayMs (on EmbedWidget / EmbedProvider) — global delay before the FAB appears after navigating to an enabled route. Default 0 (no delay).
• Added groupDelays (Map<String, EmbedButtonDelay>) — per-flow delay override with EmbedButtonDelayPolicy: perScreen, oncePerGroupEntry, oncePerAppSession. Falls back to embedButtonDelayMs.
• Added React-API aliases on EmbedButtonContinuity: continuous (= keepVisible) and perScreen (= reset), plus EmbedButtonContinuity.fromString / EmbedButtonDelayPolicy.fromString parsers.

Fixes

• Nudge/inactivity countdown now starts when the FAB becomes visible (after embedButtonDelayMs), not when the route is first entered.
• inactivity_behaviour: "auto_trigger" now maps to InactivityMode.AUTO_TRIGGER (auto-starts the call like start_call); previously it silently fell back to showing the popup. Added InactivityMode.fromBehaviour.
• FAB no longer renders off-screen on the app's initial/home route — position is recomputed once the screen size is available (post first layout) and on metrics changes.
• Fixed agentTapToOpen / agentTapToClose enum identifiers (backend values agent_tap_to_open / agent_tap_to_close unchanged).

Docs

• Added doc/fab-visibility-advanced.md (Flutter FAB visibility guide) and doc/embed-button-delay.md.

0.0.17

• Typewriter transcription panel: vertical expansion from bottom to top when type == "calculation" data received via LiveKit
• FAB positioning: widget_position (bottom-right, bottom-left, top-right, top-left) now respected from backend config
• Fixed bottomPadding and rightPadding not applying correctly to FAB position
• Bounce animation on FAB when transcription panel closes
• Analytics event names updated: agent_tap_to_open / agent_tap_to_close
• SdkEventName enum added for all internally-fired analytics event names (exported as public API)
• All public event APIs (embedEvent, embedTrackEvent, etc.) now return Future<bool> for send-success feedback
• EmbedRouteListener widget added for GoRouter / ShellRoute automatic route detection
• EmbedNavigatorObserver now accepts optional nameExtractor for GoRouter alias support
• Route matching now strips leading slashes — GoRouter paths (/home) match aliases (home)
• Cache-first config strategy restored in _fetchUIConfig with API fallback on cold start
• Removed all debug logs; all exceptions handled internally — zero crash guarantee for host app
• Removed dead _buildTranscriptionPanel overlay (superseded by inline vertical expansion)

0.0.16

• Graceful exception handling across SDK — zero crash guarantee
• Fixed event routing and SDK key resolution issues
• Stability Improvements

0.0.15

• Stability Improvements

0.0.14

• Minor Bug Fixes and Stability Improvements

0.0.13

• Minor Bug Fixes and Stability Improvements

0.0.12

• Added capability to send ANALYTICS_DATA event from screens which are not part of enabled routes.

0.0.11

• Added ANALYTICS_DATA event for tracking structured analytics events with a specific event identifier

0.0.10

• Minor Bug Fixes and UI improvements

0.0.9

• Added rightPadding and bottomPadding to set the initial position of Embed Widget

0.0.8

• Added onPermissionStatusChanged callback to recieve the permission status change events
• Minor Bug Fixes and Stability Improvements

0.0.7

• Minor Bug Fixes and Stability Improvements

0.0.6

• Bluetooth Headset Support added
• Added Inactivity Actions
• Minor Bug Fixes and Stability Improvements

0.0.5

• Added Multi Flow Support with enabledroutes

0.0.4

• Added Support for Re-initialisation of embed agent

0.0.3

• Added Support for Buttons
• Added Route based activation
• Added embedEvent call for user generated USER_DATA nd SCREEN_STATE events
• Added Embed Config Service for management of apiKey and embedUrl
• Improved widget context extraction
• Updated Documentation

0.0.2

• Removed the requirement for bluetooth permissions
• Updated License

0.0.1

• Initial release of Embedded Agent SDK
• Core functionality for automatic widget tree traversal
• Real-time widget monitoring with collapsible interface
• Comprehensive widget property extraction
• Cross-platform support for Android and iOS
• Example applications demonstrating SDK usage
• JSON output format for widget information
• Performance-optimized widget tree analysis