wixzel_voice_flutter

Flutter widgets for Wixzel Voice. Every Wixzel Voice agent has a mascot, an animated face. This package draws it in your app and makes it move and lip-sync while the agent talks.

It works with wixzel_voice, the Dart SDK for the Wixzel Voice API.

• AgentMascotView draws an agent's mascot from the Agent the SDK returns.
• MascotView and MascotStill draw any mascot spec, live or as one frame.
• MascotController makes it play moves, look somewhere, change state and move its mouth.
• MascotRealtimeLink makes it follow a realtime call: listening, thinking, talking in sync with the agent's audio.
• Its eyes follow the phone's tilt.

The mascot runtime is the same one the Wixzel Voice app uses. Tests check it against the web runtime number for number, so a mascot looks and moves the same in your app, on its share page and in the website widget.

Install

dependencies:
  wixzel_voice: ^0.8.0
  wixzel_voice_flutter: ^0.1.0

This package is not on pub.dev yet. Until it is, depend on it from a checkout of the repository:

dependencies:
  wixzel_voice_flutter:
    path: ../wixzel-voice/packages/sdk-flutter

Inside the repository it depends on wixzel_voice by path (../sdk-dart). When it is published, that becomes a hosted dependency on wixzel_voice: ^0.8.0.

It needs Flutter 3.27 or newer. It depends on sensors_plus (for tilt) and visibility_detector (so a mascot only animates while it is on screen). It does not record or play audio.

Show an agent's mascot

Fetch the agent with the Dart SDK, then draw it:

import 'package:flutter/widgets.dart';
import 'package:wixzel_voice/wixzel_voice.dart';
import 'package:wixzel_voice_flutter/wixzel_voice_flutter.dart';

Widget agentFace(Agent agent) => AgentMascotView(agent: agent, size: 200);

AgentMascotView draws agent.mascot.spec. An agent without a spec gets the default look. A near-black or near-white mascot stands on a stage of its own colour, so it does not disappear into a dark or light screen. Turn that off with stage: false.

In a list or a grid, use still: true. It draws one frame and runs no animation:

Widget agentRow(Agent agent) => Row(children: [
      AgentMascotView(agent: agent, size: 40, still: true),
      Text(agent.name),
    ]);

Keep your API key off the phone. Fetch agents on your own server and send the JSON to the app, where Agent.fromJson reads it.

Draw any mascot

A mascot is a small JSON spec: a shape, a colour, two eyes and a personality. MascotView animates one. MascotStill draws one frame.

Widget mascots() => Column(children: [
      MascotView(
        spec: defaultMascot,
        size: 160,
        interactive: true, // a tap plays an expression
        followTilt: true, // its eyes follow the phone's tilt
      ),
      MascotStill(spec: presets[2], size: 48),
    ]);

spec takes decoded JSON or a MascotSpec. normalizeSpec turns any input into a spec that draws: unknown fields are dropped and numbers are clamped. The API runs the same function, so a spec reads the same everywhere.

void readSpec(Map<String, dynamic> json) {
  final spec = normalizeSpec(json);
  print('${spec.name} ${spec.color} ${spec.shape.kind.name} ${spec.eyes.style.name}');
}

The default look and the presets

defaultMascot is the look every agent starts with: a lime (#a0d425) squircle with dark round eyes. It is plain JSON, so you can also send it to the API.

presets holds eight ready-made looks (Pebble, Peak, Puff, Onyx, Mint, Clove, Wix and Jelly) to start from.

Widget presetGrid() => Wrap(children: [
      for (final preset in presets) MascotStill(spec: preset, size: 64),
    ]);

Customize a look

Change the JSON and draw it again. A controller can swap the look without restarting the animation, which suits a live editor:

void tryLook(MascotController controller) {
  controller.setSpec({
    ...defaultMascot,
    'name': 'Coral',
    'color': '#ff6a5c',
    'shape': {'kind': 'blob', 'seed': 42},
    'eyes': {'style': 'pill', 'color': '#ffffff'},
  });
}

Shapes are circle, drop, triangle, cloud, pill, flower, squircle, blob, heart and custom. Eye styles are pill, round, block and line.

To save it, send the spec from your server with the Dart SDK:

Future<void> saveLook(WixzelVoice client, String agentId, Map<String, dynamic> spec) async {
  await client.agents.updateMascot(agentId, UpdateAgentMascot(spec: spec));
}

Drive it with a controller

Pass a MascotController to MascotView or AgentMascotView to drive it from your code:

class Greeter extends StatefulWidget {
  const Greeter({super.key, required this.agent});
  final Agent agent;

  @override
  State<Greeter> createState() => _GreeterState();
}

class _GreeterState extends State<Greeter> {
  final mascot = MascotController();

  @override
  void initState() {
    super.initState();
    WidgetsBinding.instance.addPostFrameCallback((_) => mascot.play('hello'));
  }

  @override
  void dispose() {
    mascot.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) => GestureDetector(
        onDoubleTap: () => mascot.play('dance'),
        child: AgentMascotView(agent: widget.agent, controller: mascot),
      );
}

Method	What it does
play(move)	Plays a move. Returns false for an unknown name.
setState(MascotState)	idle, listening, thinking or speaking.
lookAt(Offset?, hold:)	Looks toward a point, -1 to 1 on both axes from its centre. null lets it look around on its own.
lookAtPoint(local, size)	Looks toward a point in the view, such as a tap.
setSpec(spec)	Changes the look without restarting.
react(text)	Plays a gesture that fits a line of speech, if one fits: a nod for "yes", a wave for "hello".
feedAudio(AgentAudioChunk) / clearAudio()	Lip-sync from agent audio, and forgetting it on a barge-in.
setMicLevel(level)	The user's microphone level, 0 to 1. It leans in while they talk.
setMouth(MouthShape?)	Moves the mouth by hand. null hands it back to the audio.
playRandomExpression()	What a tap does.

Moves are listed in moveNames. The ones a tap picks from are in expressions: happy, surprised, sad, think, shrug, wiggle, dance, jump, nod, shake, look and hello.

Lip-sync to a realtime call

An agent can talk inside your app with no phone number. Your server mints a session with client.realtime.createSession, and the app connects with RealtimeConnection from the Dart SDK. See Talk without a phone number in the SDK README.

MascotRealtimeLink connects the call to the mascot. It is thinking while the call connects, waves hello, listens while the user talks, thinks while the agent prepares an answer, and talks with its mouth in sync with the agent's audio. It goes idle when the call ends.

Future<MascotRealtimeLink> talk(RealtimeSession session, MascotController mascot) async {
  final conn = RealtimeConnection.forSession(session);
  final link = MascotRealtimeLink(mascot, conn);

  conn.events.listen((event) {
    switch (event) {
      case RealtimeAudio(:final mulaw):
        player.feed(mulawToPcm16(mulaw)); // your PCM player, 8 kHz mono
      case RealtimeAudioClear():
        player.clear(); // the user spoke over the agent
      default:
    }
  });

  mic.listen((Int16List frame) {
    // your recorder: 16-bit PCM, 8 kHz mono, 160 samples (20 ms) a frame
    conn.sendAudio(pcm16ToMulaw(frame));
    link.micAudio(frame); // it leans in while the user talks
  });
  return link;
}

Show it with MascotView(spec: null, controller: mascot) or AgentMascotView(agent: agent, controller: mascot). When the talk is over, call link.dispose(). It leaves the mascot idle. It does not end the call or dispose the controller.

The link does not play or record audio. Use any recorder plugin that gives 16-bit PCM (for example record), and any PCM player (for example flutter_pcm_sound). Turn on the platform's echo cancellation, or the agent hears itself and interrupts itself.

Exact timing

By default the link decodes each audio frame itself and assumes your player starts it about 120 ms after it arrives (playbackDelay). That is close for a player that plays audio as soon as it gets it.

If your player schedules audio itself, tell the link exactly when each chunk plays, on the player's own clock:

void exact(RealtimeConnection conn, MascotController mascot) {
  final link = MascotRealtimeLink(mascot, conn, estimatePlayback: false);

  // In your player, each time it schedules a chunk:
  void scheduled(Int16List pcm, double playAt, double now) {
    link.agentAudio(pcm, playAt: playAt, currentTime: now, sampleRate: 8000);
  }

  scheduled(Int16List(160), 0.1, 0);
}

Tilt

With followTilt: true (the default) the mascot's eyes follow the phone's tilt, the way they follow the pointer on the web. It reads the accelerometer only while the mascot is on screen. A phone held at any angle looks straight ahead; only tilting it moves the eyes. It is off when the platform asks for reduced motion, and a device without the sensor just looks around on its own.

TiltGaze is the part that does this, if you want it for something else:

TiltGaze eyesOnTilt(MascotController mascot) =>
    TiltGaze((at) => mascot.lookAt(at, hold: 0.6))..start();

A stage for dark and light mascots

stageColor(color) returns a background for a near-black or near-white mascot, and null for any other colour. mascotBackdrop(color) is the soft wash of the mascot's colour that AgentMascotView puts behind it.

Widget onStage(Map<String, dynamic> spec) => DecoratedBox(
      decoration: mascotBackdrop('${spec['color']}', borderRadius: BorderRadius.circular(24)),
      child: MascotView(spec: spec),
    );

The same mascot on the web

The share link and the website widget draw the same spec with the web runtime. Turn them on with agents.updateMascot and sharing. See Mascots in the SDK README.

Example

example/lib/main.dart shows an agent's mascot and sketches a Talk button: it mints a session through your server, links the mascot to the call, and marks where your player and recorder go.

Public API

Name	What it is
AgentMascotView	An SDK Agent's mascot, with the default look and a stage.
MascotView	A live mascot. Animates only while on screen.
MascotStill	One frame, no animation. For lists.
MascotController	Moves, states, gaze, look and lip-sync.
MascotRealtimeLink	Drives a controller from a RealtimeConnection.
MascotCallEvents	The lower-level hooks the link uses, for a call that is not a RealtimeConnection.
defaultMascot, agentMascotSpec(agent)	The default spec, and the spec to draw for an agent.
presets, normalizeSpec, MascotSpec	Ready-made looks, validation and the typed spec.
moveNames, expressions, gestureFor	Move names, the tap expressions, and the gesture for a line of speech.
AgentAudioChunk, ChunkLipSync	Agent audio for lip-sync, and the analyser.
TiltGaze	Accelerometer to gaze.
stageColor, stageTheme, mascotBackdrop	Colours for dark and light mascots.
drawMascot, MascotPainter, buildOutline, MascotAnimator	The renderer and animator, for drawing into your own canvas.

The package does not export the runtime's keyframe class (Key), so it never clashes with Flutter's Key.

Related

• Documentation and the dashboard
• wixzel_voice, the Dart SDK

License

MIT. See LICENSE.