VoiceOS Flutter SDK

Production-grade Flutter SDK for AI voice calling powered by the VoiceOS Rust AI Voice Engine.

Features

Feature	iOS
Outgoing AI calls	✅
Incoming calls via PushKit	✅
Native CallKit call screen	✅
AVAudioEngine duplex audio	✅
16 kHz capture / 22.05 kHz playback	✅
WebSocket real-time streaming	✅
Auto-reconnection	✅
Speaker / Earpiece / Bluetooth routing	✅
Keychain credential storage	✅
Structured event stream	✅
Android support	Phase 2

---

Quick Start

1. Add the dependency

# pubspec.yaml
dependencies:
  voiceos_flutter:
    path: ../packages/voiceos_flutter  # local
    # OR once published:
    # voiceos_flutter: ^0.1.0

2. iOS — Required Setup

Info.plist

<!-- Microphone access -->
<key>NSMicrophoneUsageDescription</key>
<string>VoiceOS needs the microphone for AI voice calls.</string>

<!-- VoIP background mode -->
<key>UIBackgroundModes</key>
<array>
    <string>voip</string>
    <string>audio</string>
</array>

Entitlements

Add PushKit and CallKit entitlements in Xcode:

• aps-environment = development (or production)

Minimum deployment target

Set iOS deployment target to 14.0 or later.

3. Initialize the SDK

// main.dart
import 'package:voiceos_flutter/voiceos_flutter.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();

  await VoiceOS.initialize(
    const VoiceOSConfig(
      serverUrl: 'http://your-voiceos-server:8080',
      enableCallKit: true,
      enablePushKit: true,
      debugMode: kDebugMode,
    ),
  );

  runApp(const MyApp());
}

4. Listen to events

VoiceOS.instance.events.listen((event) {
  switch (event) {
    case IncomingCallEvent(:final callerName, :final callId):
      print('Incoming call from $callerName');

    case CallStateChangedEvent(:final state):
      print('Call state: $state');

    case TranscriptEvent(:final text, :final speaker, :final isFinal):
      print('[$speaker] $text');

    case MetricsEvent(:final totalMs):
      print('Round-trip: ${totalMs}ms');

    case PushTokenEvent(:final token):
      // Send this token to your backend!
      sendTokenToServer(token);

    case VoiceOSErrorEvent(:final error):
      print('Error: ${error.message}');

    default: break;
  }
});

5. Start a call

// Optional: authenticate first
await VoiceOS.instance.login('user-123', 'jwt-token');

// Register for VoIP push (iOS only)
await VoiceOS.instance.registerForVoIPPush();

// Start an outgoing call
final callId = await VoiceOS.instance.callAI(
  const CallConfig(
    language: 'en',
    voice: 'en_US-lessac-medium',
    callerDisplayName: 'AI Agent',
  ),
);

6. Control the call

await VoiceOS.instance.mute();
await VoiceOS.instance.unmute();
await VoiceOS.instance.enableSpeaker();
await VoiceOS.instance.disableSpeaker();
await VoiceOS.instance.endCall();

---

Architecture

Flutter App
  │
  ▼
VoiceOS (Dart SDK)
  │  MethodChannel: com.voiceos/sdk
  │  EventChannel:  com.voiceos/sdk/events
  ▼
VoiceOSPlugin.swift (Flutter Plugin Entry Point)
  │
  ▼
VoiceSessionManager (Coordinator)
  ├── VoicePushKitManager    — PushKit VoIP push
  ├── VoiceCallKitManager    — CallKit native call UI
  ├── VoiceAudioSessionManager — AVAudioSession (VoiceChat mode)
  ├── VoiceAudioEngine       — AVAudioEngine capture + playback
  └── VoiceSignalingClient   — URLSession WebSocket → VoiceOS gateway
            │
            ▼
     VoiceOS Rust Gateway (ws://host:8080/ws/audio/{session_id})
            │
            ▼
     Conversation Engine: EnergyVAD → Whisper STT → Ollama LLM → Piper TTS

Audio Data Flow

Microphone (native rate)
  → AVAudioEngine tap
  → resample to 16 kHz Int16 mono
  → 512-sample chunks (640 bytes)
  → WebSocket binary frames
  → VoiceOS Rust Gateway (VAD + STT)

VoiceOS Rust Gateway (TTS PCM output)
  → WebSocket binary frames (Int16 LE, 22.05 kHz)
  → AVAudioPlayerNode
  → Speaker / Earpiece / Bluetooth

---

Incoming Call Flow (PushKit)

VoiceOS Backend
  → APNs VoIP push → iOS wakes app (even if killed)
  → VoicePushKitManager.pushRegistry(_:didReceiveIncomingPushWith:)
  → VoiceSessionManager.handleIncomingCallPush(sessionId:callerName:callUUID:)
  → CXProvider.reportNewIncomingCall(...)   ← MUST happen synchronously
  → Native CallKit incoming call screen
  → User taps Accept
  → AVAudioSession activated by CallKit
  → AVAudioEngine starts
  → WebSocket connects to gateway
  → ClientMessage.startCall sent
  → AI conversation begins

---

Performance Targets

Metric	Target	Implementation
SDK idle memory	< 5 MB	No persistent connections
Active call memory	< 40 MB	Reuses AVAudioEngine buffers
Incoming call UI	< 1 s	PushKit + CallKit
Call acceptance	< 500 ms	Pre-created sessions
WebSocket connect	< 1 s	Connection on accept
Voice latency	< 200 ms	20 ms audio chunks
Cold start	< 2 s	Lazy init

---

Apple Compliance

• PushKit used only for VoIP calls (not general notifications)
• Every PushKit push immediately calls reportNewIncomingCall (required by iOS 13+)
• CallKit used for all incoming and outgoing calls
• No persistent background socket connections when idle
• No fake VoIP pushes
• AVAudioSession deactivated after each call

---

Security

• Auth tokens stored in iOS Keychain (kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly)
• Bearer token forwarded in WebSocket Authorization header
• TLS enforced for production (use https:// / wss:// server URLs)
• PushKit token stored in Keychain; never logged in production

---

API Reference

VoiceOSConfig

Property	Type	Default	Description
serverUrl	String	required	VoiceOS gateway base URL
apiKey	String?	null	Bearer token for authentication
enableCallKit	bool	true	Show native CallKit call screen
enablePushKit	bool	true	Enable VoIP push registration
debugMode	bool	false	Verbose logging
connectionTimeoutMs	int	10000	WebSocket connect timeout
reconnectDelayMs	int	1000	Initial reconnect back-off
maxReconnectAttempts	int	5	Max reconnect retries

CallConfig

Property	Type	Default	Description
language	String?	null	BCP-47 language tag
voice	String?	null	TTS voice ID
llmProvider	String?	null	"local", "cloud", or "mock"
llmModel	String?	null	LLM model override
callerDisplayName	String	"AI Agent"	CallKit display name

Events

Event	Payload	Description
IncomingCallEvent	callId, callerName, sessionId	Incoming AI call
CallStateChangedEvent	state, callId	Call lifecycle change
TranscriptEvent	text, speaker, isFinal	STT / LLM transcript
LlmTokenEvent	token	Streaming LLM token
MetricsEvent	sttMs, llmFirstTokenMs, ttsMs, totalMs	Latency breakdown
AudioRouteChangedEvent	route	Earpiece/speaker/BT change
MuteChangedEvent	isMuted	Mic mute state
PushTokenEvent	token	New VoIP push token
VoiceOSErrorEvent	error	SDK error

---

Testing

Step-by-step guide to run the example, test outgoing CallKit calls, and set up PushKit incoming calls:

docs/TESTING.md

---

License

MIT — see LICENSE