voicebot_flutter

Drop a VoiceBot store assistant — voice and text — into any Flutter app. The same assistant that runs as the VoiceBot web widget, native in your app: the user taps a launcher, talks or types, and the bot searches your catalog, navigates, filters, and adds to cart through handlers you register.

Status — early access (0.4.x). Headless core, text path, auth (Modes A & B), context enrichment, and the optional UI kit are implemented and tested. Native audio (iOS AVAudioEngine + AEC, Android AudioRecord + AEC) is code-complete and compiled in CI, pending real-device latency sign-off. See Status.

Why

Voice + text, one session, one socket. WebSocket + PCM16 — no WebRTC, no SFU to run.
Headless by design. You own all UI and all tool handlers; the SDK is a typed event core.
Behaviorally identical to the web widget — same lifecycle, same wire protocol, same bot.
No AI model on device. Voice runs server-side on Gemini Live; the SDK only moves audio bytes.
One key for web + app. Your existing VoiceBot key authorizes the mobile app too.
Type-safe, no dynamic in the public API. Sealed states, typed events, typed failures.

Platform support

	Package floor	Voice / audio path
iOS	13+	16+ (AVAudioEngine + `.voiceChat` hardware AEC)
Android	API 21+	API 24+ (AudioRecord `VOICE_COMMUNICATION` + AEC)

Audio is PCM16 mono — 16 kHz uplink, 24 kHz downlink (fixed by the Gemini bridge; no negotiation). The text-only path uses zero native code. WSS only (ws:// rejected except on localhost).

Uses a native plugin, so not Expo Go — use a development build / config plugin (RN parity note).

Voice readiness & known limitations

The native audio path is implemented end-to-end and compiled in CI (flutter build apk / flutter build ios): mic permission, PCM16 capture → WS uplink, hardware AEC, call/Siri interruption with auto-resume, and Bluetooth/headset route changes. Final sign-off requires a physical-device run (iOS 16+ / Android 24+) — CI compiles the modules but does not run them on hardware.

Speakerphone self-echo (known limitation). Voice activity detection runs server-side over the mic stream, so on loud speakerphone the bot's own output can leak past acoustic echo cancellation and self-interrupt. The mitigation is the built-in hardware AEC (iOS .voiceChat, Android MODE_IN_COMMUNICATION + AcousticEchoCanceler), which is enabled by default; effectiveness depends on the device. (This is not fixed or worsened by the WS transport choice.)

Install

flutter pub add voicebot_flutter

iOS — add a mic-usage string to ios/Runner/Info.plist:

<key>NSMicrophoneUsageDescription</key>
<string>Microphone access lets you talk to the assistant. Your voice is sent to the VoiceBot service to answer your questions.</string>

Android — the plugin declares RECORD_AUDIO; request it at runtime before starting voice.

Before submitting to a store, follow docs/STORE_SUBMISSION.md (mic string, privacy manifest, 5.1.2 / GenAI third-party-AI disclosure, Data safety).

Quickstart

import 'package:voicebot_flutter/voicebot_flutter.dart';

// `tokenProvider` mints the short-lived session JWT. Pick a mode below;
// StaticTokenProvider works today for dev / chat-only.
final client = VoicebotClient.init(
  baseUrl: 'https://api.monoverse.tech',
  tokenProvider: StaticTokenProvider.value(devSessionJwt),
);

// Client-executed actions (PROTOCOL §9): native callbacks or deep links.
client.registerToolHandler('add_to_cart', (args) async {
  await myCart.add(args['id']);
  return {'success': true, 'cartItemCount': myCart.count};
});
client.registerDeepLinkMap('open_product', (args) => 'myapp://product/${args['id']}');

// One session per assistant launch.
final session = client.startVoiceSession(config: const SessionConfig(lang: 'uk'));

session.transcripts.listen((t) => print('${t.role.name}: ${t.text}'));
session.deepLinks.listen((d) => router.go(d.uri));      // route navigations
session.errors.listen((e) => print('error: ${e.message}'));

session.sendText('скільки коштує ноутбук?');            // text turn — works without a mic

// session.minimize()   → collapse UI; session + WS stay alive (sends NO `end`)
// session.endSession() → user closed the chat → sends `end`, tears down

Text-only surface (no native audio engine at all):

final session = client.startTextSession(lang: 'uk');

UI kit (optional)

The headless core above is the primary API — you own all UI. But if you just want a working assistant fast, an optional, themeable Flutter UI kit ships in a separate library so headless apps never pull it in:

import 'package:voicebot_flutter/voicebot_flutter.dart';
import 'package:voicebot_flutter/ui.dart';   // ← opt-in UI kit

Turnkey — `VoicebotLauncher`

Drop it once near the app root. It floats a launcher button over every screen, manages the chat panel (open/minimize/close), and owns the session lifecycle:

VoicebotLauncher(
  client: VoicebotClient.init(baseUrl: '…', tokenProvider: …),
  config: const SessionConfig(lang: 'uk'),
  theme: const VoicebotTheme(corner: VoicebotCorner.bottomRight, accent: Colors.teal),
  // Register handlers/deep links once the client is ready:
  onClientReady: (c) {
    c.registerDeepLinkMap('open_product', (p) => '/product/${p['id']}');
    c.registerToolHandler('add_to_cart', (a) async => {'success': true});
  },
  // Feed live, zero-PII context per session:
  onSessionStarted: (s) => s.updateContext(const AppContext(screen: 'home', locale: 'uk')),
)

The launcher floats above all routes via a root OverlayEntry. For programmatic control, hold a GlobalKey<VoicebotLauncherState> and call open() / minimizePanel() / close() / toggle().

Pieces — `VoiceButton` + `ChatPanel`

For custom layouts, use the two widgets directly (place them in your own Stack):

VoiceButton — the floating launcher (FAB). Reflects the live phase from the session streams: idle · connecting (spinner) · listening (animated pulse ring) · bot-speaking (accent). Compute the phase with VoiceButtonPhase.from(status:, audio:, open:).
ChatPanel — the assistant window: header (title + minimize + close), scrollable transcript (live streaming partials), footer (text input + send + mic toggle). Respects safe area + keyboard.

Theming & localization — `VoicebotTheme`

VoicebotTheme ships sensible defaults that blend with the ambient Material ColorScheme; override any field (corner, margin, buttonSize/buttonElevation/buttonIcon, buttonBackground/ buttonForeground/accent/pulseColor, panel/bubble colors + radii, typography). All user-facing strings live in VoicebotLabels (theme.labels) — localize by passing translated labels (nothing is hardcoded in the widgets). Wrap a subtree in VoicebotThemeProvider to set it once, or pass theme: per widget.

Lifecycle wiring (built in)

The kit wires the minimize ≠ close contract for you: the panel's minimize collapses to the launcher and keeps the session alive (no end); the panel's close (X) calls endSession() for a real teardown. A bot/server-initiated end auto-collapses and resets for the next open.

Fully replaceable

The kit is sugar, not a lock-in. Every widget binds only to the public VoicebotSession streams (status, audioState, transcripts, …) — ignore the kit entirely and build your own UI on the same headless API (see Quickstart and example/).

Auth & the VoiceBot API key

The SDK mirrors the web widget exactly (HANDOFF §6 / PROTOCOL §1). A tenant has one VoiceBot key = a paired tenant_id + a shared secret, created at pairing. The same key authorizes both the website widget (origin-bound) and the mobile app (bundle-id-bound). The SDK never holds the shared secret in the app binary; it asks a TokenProvider for a session JWT and stores it in the Keychain / EncryptedSharedPreferences. Pick one of two modes:

Mode A — server-minted token (recommended, true web-parity)

Your backend holds the shared secret and calls POST /api/v1/mobile/issue-token, HMAC-signing the request. The secret never ships in the binary. The SDK's signer builds the exact canonical preimage the backend verifies:

HMAC_SHA256(secret, "{METHOD}\n{path}\n{timestamp}\n{nonce}\n{sha256hex(body)}").hex()

with headers X-VoiceBot-Tenant-Id / -Timestamp / -Nonce / -Signature + X-VoiceBot-App-Id (timestamp = epoch seconds, ±300s; nonce single-use). The default IssueTokenSigner is the real HMAC-SHA256 — no wiring needed; it's injectable only if you delegate to a hardware-backed key.

final client = VoicebotClient.init(
  baseUrl: 'https://api.monoverse.tech',
  tokenProvider: MobileIssueTokenProvider(
    issueTokenUrl: Uri.parse('https://your-backend.example/api/v1/mobile/issue-token'),
    credential: const AppCredential(
      tenantId: '…',                 // your VoiceBot tenant id
      appId: 'com.yourco.app',       // iOS bundle id / Android package
      sharedSecret: '…',             // server-side only — do NOT ship in the app
    ),
    poster: myHttpPoster,            // inject your HTTP client (keeps the SDK dep-free)
  ),
);

Mode B — publishable key (convenience, no backend)

For apps without a backend. Embed a publishable key vb_pk_… — an identifier, not the secret, safe to ship. The SDK calls the public POST /api/v1/mobile/issue-token-public:

final client = VoicebotClient.init(
  baseUrl: 'https://api.monoverse.tech',
  tokenProvider: PublishableKeyTokenProvider(
    issueTokenUrl: Uri.parse('https://api.monoverse.tech/api/v1/mobile/issue-token-public'),
    publishableKey: 'vb_pk_…',
    appId: 'com.yourco.app',
    poster: myHttpPoster,
  ),
);

App Attest / Play Integrity is optional hardening layered on Mode B (attestationProvider), not the primary gate.

Billing kill-switch. Tokens are minted only while the connection is status == "connected". If a subscription lapses, issuance fails and live sessions are revoked within ~30s — surfaced as a FatalConnectionFailure (close 4401). "Account alive → key works."

Lifecycle — minimize ≠ close

The single most important behavior to wire correctly (identical to the web widget):

Call	Meaning	Wire
`startVoiceSession()` / `startTextSession()`	open the assistant	issue-token → WS → (mic)
`session.minimize()`	UI-only collapse to launcher — session + WS stay alive	sends no `end`
`session.restore()`	reverse a minimize	—
`session.endSession()`	user closed the chat (the X)	sends `{type:"end"}`, tears down
bot ends / fatal close	server-initiated	emits `SessionEnded` / `SessionFailed`
app backgrounded	socket may drop	server holds a 60s reattach window; return resumes the same `conversation_id`

Wire "collapse" → minimize() and "X" → endSession(). Listen to the sealed session.states (SessionConnecting / SessionReady / SessionReconnecting / SessionEnded / SessionFailed) or the coarse session.status.

Tool handlers & deep links

Server-side catalog tools (search, get product, compare, …) run on the backend — the bot just speaks the results. Client-executed actions (navigate, open product, add to cart, apply filter, …) map to either:

registerDeepLinkMap(action, resolver) — return a route URI; emitted on session.deepLinks.
registerToolHandler(action, handler) — a native callback returning a result map.

Unregistered actions never crash — the SDK auto-replies {success: false, error: "no_handler"} and the bot adapts. The SDK always sends the action_ack for you.

Context enrichment (zero-PII)

Make the bot sharper by feeding it live client-side context (PROTOCOL §10) — reuses existing backend frames, no backend change:

// On screen / product changes:
session.updateContext(const AppContext(
  screen: 'product',
  route: '/product/42',
  currentProductId: '42',
  category: 'laptops',
  locale: 'uk',
  custom: {'promo': 'summer'},          // arbitrary non-PII signals
));

// On cart changes:
session.sendCartEvent(const CartEvent(
  type: CartEventType.added,
  items: [CartLineItem(productId: '42', quantity: 1, price: 15000, currency: 'UAH')],
));

AppContext is a typed, ZERO-PII payload (no names/emails/addresses; loggedIn is a boolean, never an id). The latest context is retained and re-sent automatically after a reconnect/resume.

Voice & privacy

Voice is processed server-side on Gemini Live — there is no AI model on device. The SDK only captures mic → PCM16 16 kHz → WebSocket and plays back PCM16 24 kHz, with hardware echo cancellation (.voiceChat / VOICE_COMMUNICATION) and barge-in flush. Session tokens live in the Keychain / EncryptedSharedPreferences and are never logged. The shared secret never reaches the device in Mode A. Foreground-only — no background audio. The bundled ios/Resources/PrivacyInfo.xcprivacy declares Audio Data (App Functionality; no required-reason APIs).

Store submission: because voice is sent to a third-party AI (Gemini), Apple Guideline 5.1.2 and Play's GenAI/Data-safety policies require you to disclose it. Follow docs/STORE_SUBMISSION.md — a copy-paste iOS + Android checklist.

Status

Area	State
Headless core (`VoicebotClient` / `VoicebotSession`, sealed state machine)	✅ implemented + tested
`WsTransport` (backoff, ~25s heartbeat, fatal-close set, ≤3 reconnect, 60s reattach)	✅
Protocol codec (all PROTOCOL.md envelopes)	✅
Auth — Mode A (HMAC, exact PROTOCOL §1 preimage) + Mode B (publishable key)	✅
Context enrichment (`updateContext` / `sendCartEvent`)	✅
Tool registry + deep links	✅
Native audio (iOS/Android)	🧪 code-complete + compiled in CI; real-device latency sign-off pending
Optional UI kit (`VoicebotLauncher` / `VoiceButton` / `ChatPanel`, `package:voicebot_flutter/ui.dart`)	✅ implemented + tested
CI (analyze + test) / tag-driven OIDC publish	✅ workflows in `.github/workflows/`
pub.dev publish (verified publisher)	⏳ first publish pending — see Publishing

Docs

docs/PROTOCOL.md — the WebSocket wire contract (ground truth; build against this).
docs/HANDOFF.md — the implementation contract (scope, decisions, lifecycle).
docs/STORE_SUBMISSION.md — App Store + Play store-compliance checklist for merchants.
example/ — a runnable app: voice + text against a configurable baseUrl.

Publishing (maintainers)

CI runs dart analyze --fatal-infos + flutter test on every push and PR (.github/workflows/ci.yml). Releases publish to pub.dev automatically over OIDC — no API token / secret is stored (.github/workflows/release.yml, the official dart-lang/setup-dart/.github/workflows/publish.yml).

One-time pub.dev setup (the package owner must do this, in order):

Create the verified publisher monoverse.tech on pub.dev (Account → Create publisher) and verify domain ownership via DNS TXT record or Google Search Console.
Do the FIRST publish manually from a clean checkout with your own Google account: flutter pub publish (this initial publish cannot be automated). Set the package's publisher to monoverse.tech.
On the published package, go to Admin → Automated publishing → Enable publishing from GitHub Actions, set repository to alyokhin-n/voicebot-flutter and the tag pattern to v{{version}}.

After that, every release is just:

# bump `version:` in pubspec.yaml + update CHANGELOG.md, then:
git tag v0.2.0
git push --tags     # → release.yml authenticates via OIDC and publishes, no token

The pushed tag must match pubspec.yaml's version (the workflow trigger is v[0-9]+.[0-9]+.[0-9]+).

License

MIT — see LICENSE.

voicebot_flutter

Why

Platform support

Voice readiness & known limitations

Install

Quickstart

UI kit (optional)

Turnkey — `VoicebotLauncher`

Pieces — `VoiceButton` + `ChatPanel`

Theming & localization — `VoicebotTheme`

Lifecycle wiring (built in)

Fully replaceable

Auth & the VoiceBot API key

Mode A — server-minted token (recommended, true web-parity)

Mode B — publishable key (convenience, no backend)

Lifecycle — minimize ≠ close

Tool handlers & deep links

Context enrichment (zero-PII)

Voice & privacy

Status

Docs

Publishing (maintainers)

License

Libraries

voicebot_flutter package

voicebot_flutter

Why

Platform support

Voice readiness & known limitations

Install

Quickstart

UI kit (optional)

Turnkey — VoicebotLauncher

Pieces — VoiceButton + ChatPanel

Theming & localization — VoicebotTheme

Lifecycle wiring (built in)

Fully replaceable

Auth & the VoiceBot API key

Mode A — server-minted token (recommended, true web-parity)

Mode B — publishable key (convenience, no backend)

Lifecycle — minimize ≠ close

Tool handlers & deep links

Context enrichment (zero-PII)

Voice & privacy

Status

Docs

Publishing (maintainers)

License

Libraries

voicebot_flutter package

Turnkey — `VoicebotLauncher`

Pieces — `VoiceButton` + `ChatPanel`

Theming & localization — `VoicebotTheme`