token_text_controller

A token-aware TextEditingController for Flutter. It detects and styles semantic text while preserving a normal editable string, cursor, selection, clipboard behavior, and IME composition.

Use it with any TextField, TextFormField, or EditableText. No custom input widget is required.

Features

  • Every built-in detector enabled by default, with fine-grained activation.
  • Reusable token groups for messaging, social, contact, commerce, structured data, suggestions, links, and Markdown.
  • Stable light/dark token colors independent of theme brand colors.
  • A dedicated optional TextStyle for every built-in token kind.
  • Regex, function, trigger, and fully custom token detectors.
  • Priority-based overlap resolution.
  • Unicode grapheme handling for joined, modified, keycap, and flag emoji.
  • Current-token queries for mention, hashtag, command, and cashtag suggestions.
  • Optional per-kind value allowlists for mentions, commands, and custom tokens.
  • Token list, newly detected token, and active query callbacks.
  • Semantic token insertion with arbitrary custom metadata.
  • A reusable TokenText renderer for non-editable content.
  • Correct composing-range rendering for international keyboards and IMEs.

Installation

Add the published package to pubspec.yaml:

dependencies:
  token_text_controller: ^0.3.3

Minimal usage

final controller = TokenTextController();

TextField(
  controller: controller,
  decoration: const InputDecoration(hintText: 'Message'),
)

By default, every built-in token kind is active, including links, contact values, social tokens, structured values, lightweight Markdown, commands, and emoji. controller.text remains an ordinary string.

Markdown rendering

Recognized inline formatting renders without its delimiters by default:

final controller = TokenTextController(
  text: '**Important** _note_ ~~old~~ ||spoiler|| `code` [Flutter](https://flutter.dev)',
);

Bold supports **text** and __text__; italic supports _text_ and *text*. Markdown links show only their label. Their destination is available in token.data; token.displayText and token.contentRange expose the visible portion. token.text, token.range, controller.text, and serialized documents always retain the original source, including syntax.

In an editable field, hidden syntax has zero visual width but keeps its UTF-16 positions. It is not deleted or replaced by widget placeholders, so selections and the IME keep working with the original value. Composing ranges underline the visible content without exposing the hidden syntax. Copying from the editable field keeps the source; read-only/selectable TokenText copies rendered text.

Use showMarkdownSyntax: true in either constructor to show delimiters. The controller property can also change while editing:

controller.showMarkdownSyntax = true;

Disabling Markdown token kinds leaves their syntax literal. Inline code does not interpret nested links or formatting; escaped and incomplete markers are not treated as formatting. Prefixes such as @, #, $, and template braces remain visible. These are lightweight, non-overlapping inline detectors, not a full CommonMark/block editor. Formatting-like punctuation inside detected URLs, email addresses, handles, hashtags, and variables stays literal. TokenTextSpanBuilder preserves source length by default; only read-only consumers should set preserveSourceLength: false.

Styling

Every built-in token accepts a complete Flutter TextStyle. Unspecified styles fall back to theme-aware defaults.

The default palette adapts to light/dark brightness without using the primary, secondary, or tertiary brand colors:

Tokens Light Dark
Links, contact details, social tokens, commands and references #4150F7 #708DFF
Dates, times, amounts, IDs and template variables Inherited Inherited

Blue follows the familiar link convention. Links and contact details are underlined; social references use semibold weight. Informational tokens, code and spoilers inherit the surrounding foreground. Highlighting in an editor does not add tap actions. Use per-type styles or a resolver if informational tokens need an action treatment in your application. Check contrast on custom backgrounds; use TokenTextStyles(actionColor: ...) to override the shared action color. Per-type styles and the resolver take precedence.

Emoji default to font size 20 and line-height multiplier 0.6 in both mixed text and emoji-only values. Set emoji or emojiOnly to customize either value.

final controller = TokenTextController(
  styles: TokenTextStyles(
    base: TextStyle(fontSize: 16),
    url: TextStyle(decorationThickness: 2),
    email: TextStyle(fontWeight: FontWeight.w500),
    mention: TextStyle(
      color: Colors.blue,
      fontWeight: FontWeight.w700,
    ),
    hashtag: TextStyle(color: Colors.green),
    command: TextStyle(color: Colors.deepPurple),
    emoji: TextStyle(fontSize: 20),
    emojiOnly: TextStyle(fontSize: 30),
  ),
);

TokenTextStyles also provides styles for phone numbers, references, dates, times, currencies, percentages, UUIDs, IP addresses, variables, Markdown links, inline code, bold, italic, strikethrough, spoilers, and custom kinds.

Use TokenTextStyles.unstyled() when no built-in visual treatment is desired. For per-value or metadata-driven styling, provide a resolver:

TokenTextController(
  styleResolver: (details) {
    if (details.token.data == 'verified') {
      return details.suggestedStyle.copyWith(
        decoration: TextDecoration.underline,
      );
    }
    return details.suggestedStyle;
  },
)

Suggestions and active queries

onActiveTokenChanged and activeQueryListenable report the trigger token at the caret, including incomplete values such as @lu, #flu, or /he.

final controller = TokenTextController(
  onActiveTokenChanged: (query) {
    if (query?.kind == TokenKind.mention) {
      loadProfiles(query!.query);
    }
  },
);

A suggestion can replace the complete active range and retain domain data:

controller.replaceActiveQuery(
  '@Luis',
  kind: TokenKind.mention,
  data: {'profileId': 'profile-123'},
);

The inserted token appears in controller.semanticTokens and remains linked to its metadata until an edit intersects that token.

Allowed values

Use allowedValues when only known values should become styled tokens. A disallowed value remains ordinary editable text and is omitted from token snapshots and callbacks. Active queries remain available so a suggestion UI can still search while the user types.

final controller = TokenTextController(
  activeTokens: TokenGroups.assistantChat,
  allowedValues: {
    TokenKind.mention: const ['maria', '@alex'],
    TokenKind.command: const ['help', '/settings'],
  },
);

The trigger prefix is optional for mentions, hashtags, cashtags, and commands, and those built-in values are matched case-insensitively. Other kinds use trimmed, exact matching. A null map, a missing kind, or an empty collection leaves that kind unrestricted.

The policy can change without recreating the field:

controller.allowedValues = {
  TokenKind.mention: currentAliases,
};

For an asynchronous source, explicitly restrict the supported kinds before results exist. This distinguishes "loading or no valid values" from the legacy empty-list behavior, which remains unrestricted:

final allowedValues = TokenAllowedValuesController(
  restrictedKinds: {
    TokenKind.mention,
    TokenKind.command,
  },
);
final controller = TokenTextController(
  allowedValuesController: allowedValues,
);

TokenAllowedValuesController is observable. Calling addAll, replace, remove, clear, restrict, or unrestrict reparses the field immediately. It can also drive read-only TokenText instances through allowedValuesListenable.

TokenQueryCoordinator optionally connects activeQueryListenable to async resolvers. Resolver keys explicitly define which token kinds may query a remote source. It debounces input, caches equivalent queries, ignores stale responses, and adds successful values without removing earlier valid tokens:

final coordinator = TokenQueryCoordinator(
  textController: controller,
  allowedValuesController: allowedValues,
  resolvers: {
    TokenKind.mention: (query) async {
      final profiles = await directory.search(query.query);
      return [
        for (final profile in profiles)
          TokenResolvedValue(profile.alias, data: profile),
      ];
    },
  },
  onResolved: (query, results) {
    // Update the suggestion UI with results and their optional domain data.
  },
);

Dispose the coordinator, text controller, and allowlist controller with their owner. When membership changes, use replace or remove and invalidate the coordinator cache as needed. Async work remains outside parsing and buildTextSpan, so rendering stays synchronous and deterministic.

TokenText and TokenTextParser also accept restrictedKinds or a complete TokenAllowedValuesPolicy. A per-call policy passed to TokenTextParser.parse overrides the parser-level policy.

Serialization

Persist plain text and resolved semantic metadata together:

final json = controller.document.toJson();
final document = TokenTextDocument.fromJson(json);
final restoredController = TokenTextController.fromDocument(document);

Use encodeData and decodeData when token metadata needs conversion to a JSON-compatible representation.

Token events

TokenTextController(
  onTokensChanged: (tokens) {
    // Current non-overlapping token snapshot.
  },
  onTokenDetected: (token) {
    // A token has newly appeared in the editable value.
  },
  onActiveTokenChanged: (query) {
    // The token query at the current collapsed selection changed.
  },
)

The same state is available through tokensListenable and activeQueryListenable for widget composition without callbacks.

Active tokens and groups

Use activeTokens to enable only the token kinds needed by a field. The default is TokenGroups.all.

final controller = TokenTextController(
  activeTokens: TokenGroups.chat,
);

Groups are intentionally scoped to a concrete kind of field:

Group Intended use
links Web URLs and deep links
contact Email addresses and phone numbers
social Posts and comments with links, mentions, hashtags, and emoji
chat Human chat without commands, finance, or structured tokens
messaging Alias for chat
assistantChat AI or bot chat with commands and Markdown
commerce Cashtags, currencies, and percentages
temporal Dates and times
identifiers References, UUIDs, IP addresses, and variables
structured All built-in identifiers, dates, and times
markdown Lightweight Markdown syntax
developerContent Links, identifiers, and markup in technical text
socialSuggestions Mention and hashtag suggestion triggers
financeSuggestions Cashtag suggestion triggers
commandSuggestions Slash-command suggestion triggers
suggestions / triggers Every built-in suggestion trigger
all Every detector included by the package

Groups are regular constant lists, so they can be used directly or combined:

final controller = TokenTextController(
  activeTokens: TokenGroups.combine(
    [TokenGroups.social, TokenGroups.markdown],
    include: const [TokenKind.reference],
    exclude: const [TokenKind.cashtag],
  ),
);

Activation can change while the field is alive. Existing text, styles, token events, and the active suggestion query are recalculated immediately.

controller.activeTokens = TokenGroups.suggestions;
controller.activateTokens(const [TokenKind.emoji]);
controller.deactivateTokens(const [TokenKind.cashtag]);
controller.activateAllTokens();

availableTokens reports the kinds backed by the controller's detector catalog, while isTokenActive(kind) checks the current selection.

Detector presets

TokenTextController(detectors: TokenDetectors.social());
TokenTextController(detectors: TokenDetectors.chat());
TokenTextController(detectors: TokenDetectors.messaging());
TokenTextController(detectors: TokenDetectors.assistantChat());
TokenTextController(detectors: TokenDetectors.commerce());
TokenTextController(detectors: TokenDetectors.structured());
TokenTextController(detectors: TokenDetectors.markdown());
TokenTextController(detectors: TokenDetectors.all());

Use detectors to replace or customize the detector catalog. All supplied detectors are active unless activeTokens is also provided. TokenDetectors also exposes forKind and forKinds for building a catalog from token kinds.

Custom tokens

Token kinds are value objects rather than a closed enum:

const orderKind = TokenKind('order');

final controller = TokenTextController(
  detectors: [
    ...TokenDetectors.defaults(),
    RegexTokenDetector(
      kind: orderKind,
      id: 'order',
      priority: 150,
      pattern: RegExp(r'ORD-\d+'),
    ),
  ],
  styles: TokenTextStyles(
    custom: {
      orderKind: TextStyle(
        color: Colors.orange,
        fontWeight: FontWeight.bold,
      ),
    },
  ),
);

Implement TokenDetector directly or use FunctionTokenDetector when a regular expression is not sufficient. Higher priorities win when ranges overlap.

Rendering outside a field

TokenText(
  message,
  activeTokens: TokenGroups.chat,
  styles: controller.styles,
  semanticTokens: controller.semanticTokens,
)

This package performs token-aware styling and semantic range management. It is not a document editor and does not alter the source string with hidden markup.

Libraries

token_text_controller
Token-aware text editing, parsing, styling, and rendering for Flutter.