tom_d4rt_ast 0.1.6

Serializable AST model for Dart code interpretation without analyzer dependency.

tom_d4rt_ast

Analyzer-free Dart interpreter runtime that executes pre-compiled SAstNode bundles with full bridging, sandboxing, and standard library support.

Overview

tom_d4rt_ast is the pure-runtime half of the D4rt interpreter ecosystem. It accepts pre-parsed SAstNode trees (the serializable mirror AST defined in tom_ast_model) and executes them directly — no analyzer package required.

Why no analyzer?

The Dart analyzer package is large and incompatible with Flutter's tree-shaker constraints. Shipping it inside a mobile app is not practical. tom_d4rt_ast breaks that dependency: the analyzer is used only at build time (in tom_ast_generator) to convert Dart source into a compact JSON representation. The resulting .ast bundle can be distributed separately — downloaded at runtime, stored in assets, or fetched from a server — and interpreted on-device by this package.

The Flutter use case

A Flutter app embeds tom_d4rt_ast (no analyzer weight). Server-side tooling uses tom_ast_generator to convert Dart scripts to .ast bundles once. The app downloads or bundles those .ast files, loads them with AstBundle.fromFile / AstBundle.fromBytes, and calls D4rtRunner.executeBundleAs<T>. The script runs on the device, producing a typed result. New script logic can be deployed without submitting an app update to the store.

Relationship to tom_d4rt

tom_d4rt is the original analyzer-based interpreter that parses and executes Dart source directly. tom_d4rt_ast contains the same InterpreterVisitor, Environment, bridging infrastructure, and standard library — they are kept in strict 1:1 sync. The difference is the AST source: tom_d4rt builds its AST from the analyzer's CompilationUnit; tom_d4rt_ast reads SAstNode trees from tom_ast_model. Any interpreter fix applied to one package must be applied to the other. The D4 helper class (static bridge utilities) exists in both packages with an identical API surface (lib/src/runtime/generator/d4.dart in this package, lib/src/generator/d4.dart in tom_d4rt).

Installation

dart pub add tom_d4rt_ast

pubspec.yaml:

dependencies:
  tom_d4rt_ast: ^0.1.5

The package requires Dart SDK ^3.10.4. Its only runtime dependencies are archive (ZIP/gzip bundle I/O) and tom_ast_model (zero-dependency SAstNode definitions).

Features

• InterpreterVisitor — two-pass AST walker (declaration pass + interpretation pass) that evaluates all Dart statement and expression node kinds defined in tom_ast_model, including async/await, generators, pattern matching, extension types, and records.
• Environment — lexical scoping with a linked-chain model. Each function call, block, or class body gets its own Environment whose enclosing reference chains back to the global scope. Supports define, get, assign, defineBridge, defineBridgedEnum, and lazy GlobalGetter / GlobalSetter entries.
• BridgedClass / BridgedInstance — native Dart classes are exposed to interpreted code via adapter maps for constructors, instance methods, static methods, getters, and setters. BridgedInstance<T> wraps the native object alongside its BridgedClass descriptor. A static supertype registry (BridgedClass.registerSupertypes) enables hierarchy-aware isSubtypeOf checks without dart:mirrors.
• BridgedEnum / BridgedEnumValue — native enums with per-value instance getter and method adapters, plus static getter support (e.g. WidgetState.any).
• Permission sandbox — five concrete Permission subclasses guard filesystem, network, process, isolate, and dangerous operations. D4rtRunner.grant / revoke / checkPermission control what interpreted code may do at runtime.
• Callable system — Callable (abstract), InterpretedFunction, InterpretedClass, NativeFunction, and several bridged adapter callables form a uniform call protocol used throughout the interpreter.
• Runtime types — RuntimeType / RuntimeValue interfaces, InterpretedClass, InterpretedInstance, InterpretedRecord, TypeParameter, and the type-coercion helpers on the D4 class.
• AstBundle — a transportable ZIP archive containing one or more SCompilationUnit modules with a manifest.json. Supports JSON, gzip-compressed JSON, and ZIP serialization. Auto-detects format on load.
• AstModuleLoader — resolves import directives against the bundle's pre-loaded module map with zero file I/O. Handles dart:* stdlib registration, bridged-library wiring, re-export chains, and per-module scoped environments.
• Standard library bridges — dart:core (String, int, double, num, bool, List, Map, Set, Iterable, DateTime, Duration, RegExp, Uri, BigInt, …), dart:async (Future, Stream, StreamController, Completer, Timer), dart:typed_data (ByteData, Uint8List, Float32List, Int32List, and all typed array variants), dart:convert (JSON, UTF-8, Base64, Latin-1, ASCII, …), dart:collection (HashMap, HashSet, LinkedHashMap, SplayTreeMap, Queue, …), dart:math (Random, Point, Rectangle, constants), dart:io (File, Directory, Process, Platform, stdout/stderr, HttpClient, Socket), and dart:isolate stubs. Platform-conditional entry point selects stdlib_io.dart on VM / stdlib_web.dart on web.
• D4 helper class — static utilities consumed by generated bridge code: unwrapAs<T>, unwrapInterpreterValue, extractBridgedArg<T>, coerceList<T>, getRequiredArg, getRequiredNamedArg, validateTarget<T>, withActiveVisitor, generic-type wrapper registration, and the native-to-interpreted Expando map.
• registerExtensions / finalizeBridges — ordered extension hook for bridge packages that have post-registration wiring dependencies.
• Introspection — DeclarationInfo sealed class hierarchy (FunctionInfo, ClassInfo, VariableInfo, EnumInfo, ExtensionInfo) for inspecting what a script declares.

Quick Start

Minimum: execute a hand-built bundle

import 'package:tom_d4rt_ast/runtime.dart';

void main() {
  // Build a minimal AST manually (or load from a .ast file).
  final mainFn = SFunctionDeclaration(
    offset: 0,
    length: 0,
    name: SSimpleIdentifier(offset: 0, length: 4, name: 'main'),
    functionExpression: SFunctionExpression(
      offset: 0,
      length: 0,
      parameters: SFormalParameterList(offset: 0, length: 0),
      body: SBlockFunctionBody(
        offset: 0,
        length: 0,
        block: SBlock(
          offset: 0,
          length: 0,
          statements: [
            SReturnStatement(
              offset: 0,
              length: 0,
              expression: SIntegerLiteral(offset: 0, length: 2, value: 42),
            ),
          ],
        ),
      ),
    ),
  );

  final unit = SCompilationUnit(
    offset: 0, length: 0,
    declarations: [mainFn],
  );

  final bundle = AstBundle(
    entryPointUri: 'package:demo/main.dart',
    modules: {'package:demo/main.dart': unit},
  );

  final runner = D4rtRunner();
  final result = runner.executeBundleAs<int>(bundle);
  print(result); // 42
}

Load a pre-compiled .ast file

import 'package:tom_d4rt_ast/runtime.dart';

void main() {
  final bundle = AstBundle.fromFile('path/to/script.ast');

  final runner = D4rtRunner();
  runner.grant(FilesystemPermission.read); // grant only what the script needs

  final result = runner.executeBundleAs<String>(bundle, name: 'buildLabel');
  print(result);
}

Load from bytes (e.g. downloaded over HTTP in Flutter)

import 'package:tom_d4rt_ast/runtime.dart';

Future<void> runScript(List<int> bytes) async {
  final bundle = AstBundle.fromZip(bytes);   // or fromBytes() for gzip JSON
  final runner = D4rtRunner();
  final result = await runner.executeBundleAsAsync<Map<String, dynamic>>(bundle);
  print(result);
}

Parse a JSON AST string

final runner = D4rtRunner();
final ast = runner.parseJson(jsonString); // returns SCompilationUnit
final result = runner.execute(ast: ast, name: 'compute');

Registering a native class bridge

import 'package:tom_d4rt_ast/runtime.dart';

final colorBridge = BridgedClass(
  nativeType: Color,
  name: 'Color',
  constructors: {
    '': (visitor, positional, named) {
      final value = positional[0] as int;
      return Color(value);
    },
  },
  getters: {
    'red':   (visitor, target) => (target as Color).red,
    'green': (visitor, target) => (target as Color).green,
    'blue':  (visitor, target) => (target as Color).blue,
  },
);

final runner = D4rtRunner();
runner.registerBridgedClass(colorBridge, 'dart:ui');

Using registerExtensions and finalizeBridges

Some bridge packages have wiring that must run after another package's registrations complete. Use registerExtensions to declare those callbacks; the runner fires them in registration order before the first script execution.

final runner = D4rtRunner();

// Register primary bridges inline.
runner.registerBridgedClass(widgetBridge,   'package:flutter/widgets.dart');
runner.registerBridgedClass(materialBridge, 'package:flutter/material.dart');

// Queue post-material wiring; it runs before the first executeBundle* call.
runner.registerExtensions('my_flutter_package', () {
  registerInterfaceProxyOverrides(runner);
});

// Optional: finalize early for deterministic timing.
runner.finalizeBridges();

final result = runner.executeBundleAs<Widget>(bundle);

finalizeBridges is idempotent — subsequent calls are no-ops. Calling registerExtensions after finalizeBridges throws StateError.

Warming up: warmup() (cold-start flakiness, OPEN B.11 / U25)

The first script run after a test harness' setUpAll used to flake under host load because the interpreter infrastructure — extension finalization, the stdlib bridges, and the registered bridged-class/enum definitions — cold-started during that first build. warmup() pays that cost up front so the first real build behaves like a warm one:

final runner = D4rtRunner();
// ... register all bridges / extensions ...
runner.warmup();              // finalizeBridges() + build a throwaway environment
final result = runner.executeBundleAs<Widget>(bundle); // first build, no cold start

warmup() runs finalizeBridges() and then builds (and discards) a global environment, exercising the full Stdlib(...).register() + bridged-definition registration path. It is idempotent and script-neutral — the warmup environment leaves no script declarations behind, and the next execute*/executeBundle* call rebuilds a fresh environment as usual.

D4rtRunner has no Dart source parser (that lives in tom_d4rt_exec's D4rt, whose warmup() additionally warms the analyzer front-end by parsing + executing a trivial throwaway script), so the runner warms only the bridge/stdlib half — the portion the parser-less Flutter runtime and a test app's /warmup endpoint share. The analyzer-based VM twin tom_d4rt's D4rt.warmup() mirrors the same contract.

Architecture and Key Concepts

SAstNode-driven execution

tom_ast_model defines the SAstNode hierarchy — a fully serializable mirror of the Dart AST. Every node is JSON-serializable with no reference to the analyzer package. InterpreterVisitor extends GeneralizingSAstVisitor<Object?> and implements visit* methods for each node kind. The DeclarationVisitor performs a first pass that registers class and function declarations into the environment before any statements execute.

The 1:1-with-analyzer principle

When a bug is found in the interpreter logic (type coercion, enum handling, isSubtypeOf chain walk, etc.), the fix goes into tom_ast_model or into both tom_d4rt and tom_d4rt_ast simultaneously. The rule is: fix the AST model or the shared interpreter logic, not a one-off workaround in one package. The _copilot_guidelines/sync_with_tom_d4rt.md document in this package enforces this contract.

Bridging: native-to-interpreted interop

Interpreted code can call native Dart constructors and methods through registered BridgedClass / BridgedEnum entries. When a bridged constructor is called, the adapter function returns a native instance wrapped in BridgedInstance<T>. InterpreterVisitor recognizes BridgedInstance at getter / method call sites and dispatches through the registered BridgedMethodAdapter / BridgedInstanceGetterAdapter. For interpreted subclasses of bridged types, an InterfaceProxyFactory (registered via D4.registerInterfaceProxy) creates a native proxy that delegates method calls back through InterpreterVisitor.

Environment and lexical scoping

Each Environment holds a Map<String, Object?> for named values and a Map<String, BridgedClass> for type-resolution. The enclosing reference chains scopes: local → function closure → class → global. define writes to the current scope; assign walks the chain to find the binding owner; get walks up until it finds a value or throws RuntimeD4rtException.

AstBundle format

A .ast file is a ZIP archive containing a plain-JSON manifest.json (format version, entry point URI, file-to-URI mapping) and one gzip-compressed JSON entry per module (0.ast.json, 1.ast.json, …). Optional Dart source files (0.src.dart) can be co-bundled for debugging. AstBundle.fromFile auto-detects format from magic bytes (ZIP PK\x03\x04, gzip \x1F\x8B, or plain JSON fallback).

Permission sandbox

Every operation in the stdlib that touches the filesystem, network, process execution, or isolate spawning calls ModuleContext.checkPermission before proceeding. Permissions are scoped: FilesystemPermission.readPath('/data') grants read access under that prefix; NetworkPermission.connectTo('api.example.com') grants outbound connections to that host only. DangerousPermission.codeEvaluation guards eval-like functionality. All permissions default to denied.

D4.unwrapAs<T> and the return boundary

At the script-to-host boundary, D4rtRunner._bridgeInterpreterValueToNative recursively converts the raw interpreter result: BridgedInstance and BridgedEnumValue leaf nodes are unwrapped to their native objects; List and Map elements are recursed; InterpretedRecord with up to 16 positional fields is converted to a native Dart record. executeBundleAs<T> then applies D4.unwrapAs<T> for the final typed cast, throwing D4UnwrapException (with expectedType and actualType fields) on mismatch.

Ecosystem Position

tom_ast_model          (zero-dependency, serializable SAstNode hierarchy)
    ^
    |  depends on
    |
tom_d4rt_ast           (THIS — analyzer-free interpreter runtime)
    ^
    |  depends on
    |
tom_ast_generator      (analyzer-based Dart source → SAstNode converter)
    ^
    |  depends on
    |
tom_d4rt_exec          (full D4rt execution entry point, 100% API-compatible with tom_d4rt)
    ^
    |  depends on
    |
tom_dcli_exec          (DCli CLI tool, uses tom_d4rt_exec for script execution)

tom_d4rt               (original analyzer-based interpreter; kept in sync with tom_d4rt_ast)

The tom_d4rt_ast package is the only component that a Flutter app needs to embed. Build tooling (tom_ast_generator, tom_d4rt_exec, tom_d4rt) runs on the developer machine or CI server and is never shipped to end users.

Documentation

tom_d4rt_ast runs the same interpreter as the analyzer-based base, so its docs are differences-only (policy P1) and link to tom_d4rt for shared semantics.

• User Guide — what differs in the analyzer-free runtime: AstBundle loading, D4rtRunner, the typed-execute API, and the Flutter/web deployment model.
• Limitations (delta) — runtime-specific limits (no on-device parser, web dart:io absence, bundle-scoped imports); links back to the canon.
• Extension Registration — registerExtensions / finalizeBridges ordering contract.
• Relaxer Usage Logging — opt-in D4 usage instrumentation.
• Runtime Registration Surface — the canonical D4.register* reference (shared with the VM twin).
• Base (shared) docs: tom_d4rt User Guide · Bridging Guide · Limitations (canonical).

Status

Version 0.1.5 — current release on pub.dev (first published at 0.1.4). The package is production-quality in the context of the Tom framework and is kept continuously in sync with the analyzer-based tom_d4rt interpreter.

Repository: https://github.com/al-the-bear/tom_d4rt/tree/main/tom_d4rt_ast

Issues and pull requests should be filed against the parent repository at https://github.com/al-the-bear/tom_d4rt.