Automation 🚀

in_app_automation is an in-app UI testing tool for Flutter. You register tests in Dart, then run them either from an on-device inspector overlay or headlessly in CI. Actions drive real input — synthetic pointer and IME events routed through Flutter's gesture and text pipelines — so a green result means a user could actually perform the action.

Guides: Integrating into your app · User-flow cookbook

Table of Contents

What it does

  • Real input — tap dispatches hit-tested pointer events through GestureBinding; enterText drives the field's real IME pipeline (inputFormatters, onChanged, selection, onSubmitted). Callbacks are not invoked directly, so covered / IgnorePointer / disabled widgets are correctly reported as not actionable instead of silently "passing".
  • Actionability + auto-waiting — before acting, the target must exist, resolve to an enabled element, be visible (honoring Offstage, zero Opacity, and clips), and actually receive events at its center. These are re-checked until a timeout.
  • On-device inspector — a floating overlay (debug builds) to pick and run tests and watch step-by-step progress.
  • Headless runner — the same runner, wired to a process exit code for CI.

Installation

flutter pub add in_app_automation

Or add it to pubspec.yaml:

dependencies:
  in_app_automation: ^0.2.0

Set it up with an AI assistant

This package is designed to be adopted with an AI coding assistant (Claude Code, Cursor, Copilot, etc.). The guides in doc/ are written as step-by-step, checklist-style specs, so an assistant can read them and do the work in your project. Two steps:

1. Let the AI wire the package into your app

Point the assistant at doc/INTEGRATION.md and ask it to apply the guide to your project. It will edit your core files to meet the package's requirements — add the dependency, wrap the root widget in AutomationInspectorWrapper, register tests inside if (kDebugMode), add stable Keys to the widgets your tests will target, and create the integration_test entrypoint and CI workflow.

Example prompt: "Read doc/INTEGRATION.md from the in_app_automation package and apply it to this project. Work through its checklist: update pubspec, wrap my app in AutomationInspectorWrapper, register tests in kDebugMode, add Keys to the widgets I'll drive, and add the integration_test entrypoint and CI workflow. Show me each file change."

2. Let the AI write your user-flow tests

Point the assistant at doc/USER_FLOWS.md and describe the flows you want. Using the cookbook's patterns, it will create the flow test files (e.g. lib/automation/app_tests.dart) as registerTest(...) scenarios that reference your app's real keys and screens.

Example prompt: "Read doc/USER_FLOWS.md. Based on my screens (login, dashboard, profile, cart), write user-flow tests for: login → dashboard → profile; add-to-cart → checkout → confirmation; invalid login shows an error. Follow the cookbook patterns, reuse a shared login() sub-flow, and reset state in beforeEach."

Tips: keep Key names stable (they're the test contract), review what the assistant generates, and finish with flutter test integration_test.

Prefer to do it by hand? Follow the same guides yourself — the manual steps are below.

Setup

Wrap your app with AutomationInspectorWrapper:

import 'package:in_app_automation/in_app_automation.dart';

void main() {
  runApp(AutomationInspectorWrapper(child: const MyApp()));
}

The overlay only renders in debug builds. In release, automation is disabled by default (see Safety).

Writing your first test

Register tests before runApp (guard with kDebugMode so test code and any credentials do not ship in release):

void main() {
  if (kDebugMode) {
    AutomationRegistry.instance.registerTest(
      name: 'Login',
      tags: {'smoke'},
      steps: [
        TestStep(
          description: 'Enter username',
          action: () => AutomationEngine.instance.enterText(const Key('username'), 'tester'),
        ),
        TestStep(
          description: 'Tap login',
          action: () => AutomationEngine.instance.tap(find.byText('LOGIN')),
        ),
        TestStep(
          description: 'Land on the dashboard',
          action: () => Expect.visible(find.byText('Inventory')),
        ),
      ],
    );
  }
  runApp(AutomationInspectorWrapper(child: const MyApp()));
}

Finders

Via the global find:

Finder Matches
find.byKey(key) widget by Key
find.byText('x') Text/EditableText with exactly that text
find.textContaining('x') substring; also accepts a RegExp
find.byIcon(Icons.add) Icon
find.byType(ElevatedButton) exact runtime type
find.byWidget<ButtonStyleButton>() type or subtype
find.byTooltip('Delete') Tooltip by message
find.descendant(of:, matching:) child within a parent

Narrow any finder with .first, .last, or .at(n).

Actions and assertions also accept a Key, String, IconData, or Type directly and resolve it to a finder.

Actions

final engine = AutomationEngine.instance;
await engine.tap(target);                       // real hit-tested tap
await engine.enterText(target, 'hi', submit: true); // real IME input + submit
await engine.scrollUntilVisible(target, axis: Axis.vertical); // animated scroll
await engine.waitFor(target);                   // wait until present
await engine.pumpAndSettle();                   // wait until frames settle

scrollUntilVisible prefers the scrollable that contains the target (so nested and horizontal lists work) and scrolls with real physics.

Assertions

Expect assertions auto-retry until they hold or time out:

await Expect.visible(target);
await Expect.hidden(target);
await Expect.absent(target);
await Expect.count(find.byText('Item'), 10);
await Expect.enabled(target);
await Expect.disabled(target);
await Expect.text(const Key('title'), 'Welcome');   // descends into children
await Expect.textContaining(target, 'Wel');

Collect several without stopping at the first failure:

final soft = SoftAssertions();
await soft.check(() => Expect.visible(a));
await soft.check(() => Expect.text(b, 'x'));
soft.assertAll(); // throws once with every failure

Failures throw typed errors (ElementNotFoundException, NotVisibleException, NotActionableException, AutomationTimeoutException, AutomationAssertionException, …) — all subtypes of AutomationException.

The test runner

TestRunner is the single execution core used by both the inspector and CI:

final results = await TestRunner(
  config: const TestRunConfig(
    defaultTimeout: Duration(seconds: 20),
    retries: 1,                 // retry failures; passing on retry marks flaky
    includeTags: {'smoke'},     // or excludeTags / grep by name
  ),
  listeners: [TestReporter.instance],
).run(
  AutomationRegistry.instance.tests,
  hooks: AutomationRegistry.instance.hooks,
);

Register setup/teardown (use beforeEach to reset app state for isolation):

AutomationRegistry.instance
  ..beforeAll(() { /* once before the run */ })
  ..beforeEach(() { /* reset app state before each test */ })
  ..afterEach(() { /* cleanup */ })
  ..afterAll(() { /* once after the run */ });

Each result carries its outcome, attempts, per-step timings, and error/stack.

Reports and screenshots

When a test fails, a screenshot is captured automatically (before teardown) and attached to its result — AutomationController.runAllTests and the inspector both enable this by default. The HTML report embeds each failure's screenshot inline.

Format results yourself with TestReportFormatter:

TestReportFormatter.toJson(results);     // structured JSON
TestReportFormatter.toJUnitXml(results); // JUnit XML for CI
TestReportFormatter.toHtml(results);     // HTML with embedded failure screenshots

Or write the whole set to disk (reports + a PNG per failed test) via the IO entrypoint:

import 'package:in_app_automation/io.dart';

await AutomationController.instance.runAllTests();
await TestArtifactWriter.write(TestReporter.instance.detailedResults);
// -> build/automation-reports/{report.json, junit.xml, report.html, *.png}

Capture a screenshot manually at any time:

final png = await AutomationScreenshot.capture();

Running headless in CI

Automation runs headlessly through package:integration_test, whose pass/fail becomes the process exit code. See example/integration_test/automation_test.dart for a complete entrypoint that runs the suite and writes JSON/JUnit/HTML artifacts, and .github/workflows/automation-ci.yml for a reference workflow (analyze + unit/widget tests, plus the integration suite on an emulator with artifact upload).

flutter test integration_test    # exits non-zero if any test fails

Safety: release builds

Automation actuates the live app, so it is enabled outside release builds only (debug and profile). In release, tap/enterText/scrollUntilVisible throw AutomationDisabledException and runAllTests no-ops. Opt in deliberately for release automation:

AutomationConfig.enable();   // allow automation in any build mode

Stability

Pre-1.0 (0.x): the API may change between minor versions. Breaking changes are called out in the CHANGELOG. Once the interaction and runner APIs settle, this will move to 1.0 with semver guarantees.

Happy testing! 🎉

Libraries

in_app_automation
io
IO-dependent helpers for the automation package.