self_test 1.0.0

Automated regression testing for Flutter through direct callback invocation. Run tests in live app environments without UI simulation.

1.0.0

First stable release. The public API below is the one this package intends to keep: a change that breaks it from here is a 2.0, not a patch. The 0.x line stopped at 0.1.0 on pub.dev, so every entry in this section is new to anyone installing the package rather than tracking the repository.

Breaking

• RecordingStore has one more method: deleteStep. A store that implements the interface has to add it, which is one line over whatever it already does for deleteScript. Without it there is no way to remove a step, so fixing one wrong step in a recording meant deleting the recording and capturing the whole session again.

• Minimum Flutter is now 3.35.0 (Dart 3.9.0), raised from a declared 3.24.0 that was never true. lib/recording_fields passes widget properties that only exist from 3.35 (Switch.activeThumbColor and friends), so installing 0.1.0 on Flutter 3.24 produced eight compile errors inside this package. 3.35.0 is the oldest version the whole suite is verified against, and CI now runs against it on every push so the number stays honest.

• The build_runner generator has moved to its own package, self_test_gen. Add it to dev_dependencies to keep using annotations. In exchange, the core package no longer depends on analyzer, source_gen, build or dart_style, so none of them reach your app any more. Its only dependencies are Flutter and meta.

• Generated controller methods are camelCase, matching what the README always documented: tapLoginBtn() rather than tap_login_btn(). A private class such as _LoginFormState now generates LoginFormStateTestController, so it can be referenced from a test.

• Do not write a part directive for the generated file and do not import it from its own source. The builder emits a standalone library; import it from your test. Importing it from its own source leaves that library unresolvable, and the generator then finds no annotations at all.

• Recording persistence is an interface. RecordingStore and InMemoryRecordingStore replace DatabaseService, and hive_ce and path_provider are gone. Pass your own implementation to SelfTestManager().useRecordingStore() for recordings that survive a restart. initializeDatabase and clearDatabase are deprecated aliases for initializeRecordingStore and clearRecordings.

• The recording model formerly called TestStep is now RecordedStep. TestStep remains the scenario command type it always was in the public API.

• TestCodeGenerator returns strings instead of writing files, so the caller decides where generated code goes.

• WidgetCatalog and FlowDiscovery are deprecated. Both only ever returned an empty map.

• The bridge no longer depends on go_router. SelfTestBridge took a GoRouter? router, so a testing bridge that was meant to drive any Flutter app only navigated in apps that had picked one particular routing package. The parameter is now BridgeNavigator? navigator, an interface this package owns, with four members the bridge actually needs. Pass NavigatorStateBridgeNavigator(yourNavigatorKey) and it works in any app; a go_router app implements the interface in ten lines, and BridgeNavigator's doc comment gives that adapter in full.

  The parameter was renamed rather than kept as router, because it no longer takes a router. bridge.router is now bridge.navigator.

  Navigation commands that used to report {"success": true} while doing nothing, because no router was configured, now return an error saying so. getFlowGraph returns the routes the navigator reports rather than the empty map deprecated FlowDiscovery handed back.

• The generator writes .self_test.g.dart, not .g.dart. Regenerate and update the import in your tests. source_gen:combining_builder claims .dart -> .g.dart, and every part-based generator (json_serializable, freezed, mobx, hive, drift) routes its output through it. Claiming the same file made build_runner refuse to start in the whole package, with Builders source_gen:combining_builder and self_test_gen:self_test outputs collide, taking the app's existing codegen down with it. Verified against a real app: a 75-file Flutter app using mobx_codegen and hive_ce_generator could not build at all with self_test_gen added, and builds with this change.

Added

• A run is coloured by what its steps do. Every action names a family, and the report page draws each family in its own colour: taps blue, typing teal, assertions violet, a channel mock orange, and getting around in grey so it recedes. A hundred and fifty rows of trigger differ by one word, and that word was the same colour as the rest of the line. The families come from StepVocabulary and reach the page over /report/api/vocabulary, so an action added to the list arrives already sorted, and a test fails if the page has no colour for a family the app names or a colour for one it does not. The verdict is on the edge of a row as well as in its tag, and an exported report is drawn the same way from the same list.

• A script can be edited from the report page. Add, change, reorder and delete the steps of a stored script in the browser. SelfTestManager grew insertStep, editStep, deleteStepAt and moveStep; the page reaches them over /report/api/scripts/<id>/steps. A recording is a first draft, and until now one wrong step in a fifteen-step session meant recording the session again.

• One step vocabulary, shared. StepVocabulary is the single list of what a step can be: the action, its other spellings, whether it needs a target, what its value means, and an example. The importer reads it, the editor is served it over /report/api/vocabulary, and a test compares it with the case labels of the replayer's own switch, so an action cannot be added to one and missed by the others. The README's step table is generated from it and a test fails until the file matches.

• The editor is offered what is on screen. SelfTestManager.suggestTargets returns every widget the app can see with the strongest locator that finds it, a key before the words it paints, indexes filled in. /report/api/screen serves it and /report/api/resolve answers what a locator finds before a run depends on it: how many matches, which one the index picks, and whether that one has anything to press.

• Import and export a script. GET /report/api/scripts/<id>/export answers one script as a file that carries no ids from the app that held it; POST /report/api/scripts/import takes one back, checked by the same rules as a suite installed over the socket. A file can also be dropped anywhere on the page.

• The report page has been rebuilt: one icon set drawn on one grid, a step editor, and a test that reads the page as it is served and fails when the script reaches for an element the markup does not have. They fail together otherwise, silently, and what arrives in the browser is a header over three empty columns.

• A run keeps a record of itself. Every replayed step is recorded with its outcome, its duration, the locator as recorded and the locator as repaired, and optionally a screenshot; RunReport and StepReport hold it and a RunReportStore keeps it on disk past the run. Before this a run left a verdict and nothing else: lastRunStatus was stamped on the script and overwrote the previous run, so a run that repaired two steps and passed was indistinguishable on screen from one that passed clean. The result knew the difference and nobody could see it.

• Screenshots are a choice, not a policy. CaptureMode.failuresOnly is the default and photographs the first step, the last, and any failure; everyStep photographs all of them; never turns it off. Measured on a real app, a picture costs about 185ms and 47KB at pixelRatio 1.5, so photographing all 866 steps of a suite is a real cost to opt into rather than one to impose.

• The report is served over the port the bridge already holds. GET /report answers an HTML page, /report/api/* answers JSON, and /report/shot/<run>/<step>.png answers a picture. The token is checked before the WebSocket upgrade is demanded, so these routes inherit the guard rather than adding a second one. It is served by the app because that is where the pictures are: on a simulator they can be copied out with simctl, on a device they cannot be reached at all.

• The browser can drive the app. Run, stop, rename and delete a script, and delete a run, from the page. A run answers 202 immediately with running: true, because a scenario takes minutes and a browser that waited would time out and read as a failure.

• A run says where it is while it runs. /report/api/status reports the script, the step index, the total and the elapsed time, announced before the step is attempted rather than after it completes: a step can wait seconds for its target, and reported on completion the step a run hangs on is the one step that never appears. The elapsed time is computed by the reader, not frozen into the progress, so it keeps moving during exactly the wait being watched.

• The report exports. GET /report/export/<run> answers one self-contained HTML file with the pictures inlined and no links back to the server, so a failing run can be read on a machine that never had the app.

• assertAbsent, because an absence is its own assertion. Five of the regression cases this package is measured against have an absence as their whole expected result ("the Done button is hidden", "scaffold 0001 is not listed"). The obvious workaround, asserting the text that replaces the control, passes on a screen showing both. An absence assertion also skips the settle loop: it names a widget that must never appear, so waiting for it to appear and settle could only spend the whole budget confirming what was already true on the first poll.

• A recording keeps what the finger did to the page. A scroll is recorded from ScrollNotification rather than from pointer arithmetic, because the finger does not know where the page stopped: a drag hands off to a fling that keeps going after the finger leaves, and a page that reaches its end stops while the finger continues. Only drags are recorded (a form that scrolls a field into view when the keyboard opens does it again on replay), only net movement above a 1px floor (dragging a list already at the top stretches it and lets it go), and the step names the Scrollable, never a row, because the step exists to move the rows.

• A recording keeps the file a native picker answered with. The answer is recorded as a mockChannel step placed before the tap that caused it, since installing it after the tap is installing it after the real picker already opened. The file is copied, not referenced: a picker answers with a path into a directory the app empties. Only answers that name a file are kept, because that is the one class of answer a replay cannot obtain again, while connectivity, location and preferences are answered the same way on the next run; without that rule four taps recorded as twenty four steps with a GPS position and a timestamp frozen inside them.

• dragAt, the nameless sibling of tapAt, and renameTestScript. The first exists because nothing else exercises the scroll recorder: a drag with a locator records its own step and the recorder discards it as a duplicate, so until there was a nameless drag nothing could prove the recorder worked.

• A mocked platform channel now answers the app. Call SelfTestWidgetsFlutterBinding.ensureInitialized() where the app called WidgetsFlutterBinding.ensureInitialized(), and the binding's messenger answers mocked channels in place of the platform: the camera on a simulator, a native picker, a permission dialog. ChannelMocks holds the table and a log of what the app asked. Plugins take the default messenger from the binding the moment it exists, so this is the only place a mock can be put in their path; in an app that already has a binding the call installs nothing and ChannelMocks.instance.intercepting stays false, so a caller can say so. Pigeon channels (image_picker, path_provider, url_launcher...) are mocked with ChannelMockKind.message: their reply is a one-element list.

• The bridge's mockChannel tells the truth. It refuses, naming the binding, when nothing intercepts the app's calls, instead of reporting a mock it had registered for the wrong direction. codec: "message" mocks a Pigeon channel; channelLog lists the calls the app made on mocked channels and what answered each. The MCP tool flutter_mock_channel exposes both.

• One run at a time, and cancelRun(). A replay lives in the app, not in whoever pressed Run, so a driver that dies mid-run left the app pressing its buttons for minutes while a fresh driver's taps all reported success and nothing happened. runTestScript now refuses a second run with a result that says one is under way, isRunning says so, and cancelRun() stops the run at the next step and puts the app back in the mode it was found in.

• Universal locators. SelfTestLocator finds a widget by the text it paints, its ValueKey, its tooltip, its semantics label, its type, or a SelfTestableWidget id, with .at(n) to pick between duplicates. Resolution walks the element tree, so an app needs no wrapper, no annotation and no generated code to be driven. Locators are JSON in both directions, which is how the bridge and the MCP server will carry them.

• Real pointer events. tap, doubleTap, longPress, dragFrom and scrollBy dispatch through GestureBinding.handlePointerEvent, the same entry point the engine uses. Hit testing runs, so a widget behind a dialog is not reachable and a disabled button swallows the tap. Before this, a driven tap invoked the app's callback directly and therefore passed on buttons the user could not even reach.

• Real text entry. typeInto goes through EditableTextState.updateEditingValue, the method the soft keyboard calls, so input formatters run, onChanged fires and a TextFormField validates what was actually typed. It also resolves a field from the label beside it, which is how a person describes it.

• describeScreen() returns every actionable widget on screen with its type, text, tooltip, rect and enabled state, for an agent deciding what to do next.

• exists and isVisible are separate questions. A list keeps items built after they scroll away, and tapping one of those would land on whatever is drawn at those coordinates now, so a driven gesture refuses instead.

• A recording keeps how each widget was addressed, not just an id. RecordedStep.locator carries the locator, so a session recorded on an app with no wrappers replays. A step recorded before this, or against an id, still replays through the id path.

• recordAssertion records an assertion against a locator, so a recorded session can check something. Recording only actions produces a generated test that drives the app and asserts nothing, which passes on a blank screen.

• The code generator emits the locator API and a testWidgets body that pumps between steps, because a driven tap is a real pointer event and nothing it changes is visible until the next frame. It also pumps the app for you when given an appExpression.

• useClock lets a widget test hand the driver tester.pump, which is what a long press needs to be held rather than silently degrading to a tap.

• The bridge speaks locators. Every command that named a widget by its registered id now also takes a locator object, and prefers it when both are sent:

  {"by": "text|key|id|semanticsLabel|type|tooltip",
   "value": "Sign in", "exact": true, "index": 0}
  

  tap, doubleTap, longPress, type/enterText, submit, drag, scroll and clear route through the manager's locator API when given one, so they reach widgets the app never registered. describeScreen, find, exists, isVisible and readText are new and take a locator only; describeScreen answers with WidgetSnapshot JSON. submit is a new command.

  A malformed locator is answered with what is wrong with it - an unknown by, a non-string value, a negative index - rather than a cast error from three layers down.

• Recording, replay and test code generation, merged in from the feature/visual_tests line: a recording control panel, a recording-field registry, and TestCodeGenerator.

• A DevTools extension and a standalone inspector under packages/.

• SelfTestManager.captureScreenshotBytes() for platform-neutral capture, and clearScreenshotKey() so a disposed boundary cannot leave a stale key.

• MIT license, replacing the previous custom terms.

• CI covering every package, including a job that runs the README quickstart.

Security

• self_test is inert in a release build. Every action and every query is gated: no taps, no typing, no screenshots, no recording, no reading the widget tree, and no register of what is on screen is even built. A device farm that means to drive a signed build calls SelfTestManager.enableInReleaseBuilds() deliberately. Nothing flips it by accident, and debugSimulateReleaseBuild exists so the guard is covered by tests rather than asserted in a comment.
• The bridge binds to loopback, not to every network interface. Until now it was InternetAddress.anyIPv4, so anyone on the same wifi could drive a colleague's debug build: read the tree, tap, type, photograph the screen. Pass host: InternetAddress.anyIPv4 to reach it from a real device, and it says out loud what that means.
• The bridge requires a token. One is generated per instance and printed at startup, or the app supplies its own. A connection without it, or with a wrong one, gets an HTTP 403 before the WebSocket upgrade. The comparison is constant time.
• The bridge refuses to start in a release build unless allowInReleaseBuilds: true is passed, because a bridge in a shipped app is a remote control for it.

Fixed

• A mock step exported with a script could not be imported again. The importer built a mock only from spread-out params, and a step read back from a script carries all three values in the single field a step has, so export was a one-way door for any script that answered a channel.
• A step written by hand could name no widget at all. enterText against the empty locator resolved to nothing and reported a pass.

Every screenshot after the first was the same picture. With no boundary registered, capture walks the tree and photographs the first RenderRepaintBoundary it finds, and on a real app that one belonged to a route that had stopped repainting: a nine step run produced seven byte identical files, including the one labelled as the failure. The root the app is already wrapped in now installs the boundary, and capture waits while debugNeedsPaint, because toImage answers the last raster that was painted. The package's own docstring had warned about this.

• A step that names no widget no longer waits for one. Before each step the replay waits for that step's target to appear and settle, with a five second budget. wait, goBack and mockChannel name no widget: the first two carry an empty target that resolves to nothing on every poll, and a mock carries a channel name, which looks like an ordinary target and will never resolve. Both spent the whole budget before running a step that was never about a widget. Nothing failed; the only symptom was the clock. Measured at 327 of the 866 steps in one suite, or 27 minutes of waiting for a widget no step mentions; one scenario went from 4.97s to 0.78s per step.

• A field is named by the label beside it, not by what it shows. The recorder derived a name from the only text a widget painted, which in an empty field is its placeholder, and the replayer refuses a placeholder on purpose because it stops existing the moment the field is filled. So the recorder wrote a locator the replayer was designed to reject. Worse, the fallback was an index, and an index counted with a bottom sheet open is read again with it closed: a recorded EditableText index 1 typed into the date field and the run reported PASSED. A field is now named by the nearest word beside it, accepted only when asking the text input driver what that word means answers this field, which also settles which of two identical labels was meant.

• Typing resolves to a field that can take a keystroke. A locator means the first match, and an app often draws a read only picture of a field over the real one: the first match was the picture, the guard refused it, and the run stopped with the field it wanted one match further along on the same screen. A typing step now skips matches with no field or a read only field, and when none of them accepts text the guard speaks as before.

• A tap a guard refuses is retried once with the target scrolled into view. A recording has no scroll in it unless the finger made one, so a row recorded at the top of a form replays under a footer that covers it. The retry happens only after a refusal, never before a tap that was already going to land, and only once: when there is nothing to scroll, or scrolling does not free the target, the refusal stands.

• A recorded picked file survives the next install. The step stored an absolute path into the app's data container, and iOS renames that container on every install while migrating the files: the file always survives, the path always dies. Three container names in two days. The app then received a path to nothing, which only failed much later inside the image decoder, far from the cause, and looked exactly like recording nothing at all. A step now stores self_test:kept/<name> and the replay joins the name to this install's directory, so a recording made two installs ago still runs.

• A picker that names the same file twice is copied once. file_picker answers with a list of maps naming the file in both path and identifier as a file:// URL. The copy was memoized on the string, and two spellings are two strings, so the step ended up pointing at two different files. Reading the answer also walked only the strings of a list, so a map inside the list yielded no string at all, the whole call was discarded as noise, and the recording kept the tap that opened the gallery and nothing else.

• A replay no longer writes into an open recording. The recorder watches pointers at the root and a replayed tap is a pointer like any other, so running a script while recording appended the replay's own taps to the recording; when the script being replayed was the one being recorded it doubled, and the run still reported PASSED. The only clue was the step count in the panel, by which time it is too late.

• A scroll step is not repaired as though it were a tap. Recording the scroll gave the step a point, and self healing reads a point as a tap: a scroll was rewritten to be named after the row it exists to move away.

• A picture waits for the app instead of a fixed delay. Capture happened 100ms after the step, which is a route transition and not a screen load, so two steps of a passing sixteen step run were photographs of a spinner on an otherwise blank screen. Measured on a real app, the same tap answered in under 170ms warm and had not finished in 500ms cold, so any hand picked number is short on one run and wasted on the other. Capture now waits for the next step's target to settle and for no progress indicator to be on screen, photographs anyway when the budget runs out and says so (capturedWhileBusy), and never waits at all for a failure (the screen that blocked it is the one worth photographing) or for a step nobody photographs. RefreshIndicator is explicitly not a progress indicator: it is the pull to refresh wrapper and sits in the tree of every list that can be pulled, loading or not, so counting it made a fully drawn list claim to be busy and cost the whole budget on every picture.

• find answers the index it was given. It returned the same widget for index 0 and index 7, and returned a widget for an index that does not exist.

• A command sent without a parameter it needs now says which parameter. Sixty five call sites read params['x'] as String directly, so omitting one answered type 'Null' is not a subtype of type 'String' in type cast, which names neither the parameter nor the command and reads like a crash in the bridge rather than a mistake in the request. Found by driving a real app: wait with no condition answered exactly that. The locator parameter already had this treatment; the scalar ones now do too, and a wrong type reports what was actually sent.

• The docs no longer assume you are watching a flutter run console. The bridge token is announced through debugPrint, which reaches nobody under flutter build plus simctl launch or adb shell am start, which is how CI drives an app. Three places told you to use "the token the app printed"; they now tell you to pass your own.

• getByText and getByRole read the element tree, like every other discovery command. They read the registered-node map instead, so on an app that never adopted SelfTestableWidget, which is every app before it adopts this package, they answered [] with no error. Found by driving a real production app: describeScreen returned a Text reading "Scaffolder", a locator exists on that same string answered true, and getByText('Scaffolder') answered nothing. getByRole also inferred the role by looking for "checkbox" or "switch" inside the widget's id string, so a button named checkbox_help was a checkbox and a real Checkbox with an unhelpful id was not; roles now map to the Flutter types that answer to them. Its name: filter searches the button's descendants, because a Flutter button paints no text of its own.

• self_test_bridge no longer caps web_socket_channel at 2.x. It only uses WebSocketChannel and WebSocketChannel.connect, both unchanged in 3.x, so the caret pin bought nothing and cost the consumer a major version: adding the bridge to a real app downgraded its web_socket_channel from 3.0.3 to 2.4.0 and dragged build_runner's shelf stack down with it. Now >=2.4.0 <4.0.0, with the bridge's 60 tests verified on 3.0.3.

• The bridge answered "done" before it had done anything. tap, type and eleven other commands called SelfTestManager.trigger and enterText without awaiting them. Those methods became async when they started dispatching real pointer events, so an agent that tapped and then read the screen was told the tap was finished and shown the screen from before it. unawaited_futures is now enforced in that package so it cannot come back.

• Time-travel capture-on-interaction and test-step recording were implemented and then never called, so both recorded nothing. Recorded comments were dropped between the command and the step.

• The widget rebuild profiler reported every rebuild as "Frame sample" instead of the reason it had worked out.

• Starting memory profiling twice abandoned the first timer, leaving two running and interleaving their samples.

• A recorded tap fired the app's handler twice. The recording wrappers called the driving API to record the action, which invoked the wrapper's callback, and then called the child's callback as well. Recording now records, and the child's own handler runs once; SelfTestableWidget.onTap is a fallback for a child that carries none.

• Driving a widget wrapped in SelfTestableWidget with no matching recording builder recursed until the stack ran out, because the fallback wrapper called trigger from inside the tap it was handling.

• Under integration_test, driven gestures did nothing at all and said nothing: that binding drops pointer events that did not come from a WidgetTester. The driver now detects it and names the one line that fixes it, binding.shouldPropagateDevicePointerEvents = true.

• Annotations on class members are found. LibraryReader.annotatedWith only visits top-level declarations, so annotating a handler, which is what the README asks for, previously generated nothing at all.

• Recorded values are escaped when generating test code. A value containing a quote, a dollar sign or a newline produced source that would not compile, or that picked up an unintended interpolation.

• Two assertText steps in one recording no longer declare the same local twice, which made the generated file fail to compile.

• SelfTestRoot no longer draws its recording controls in a widget test. They are a live overlay, so pumpAndSettle on a wrapped app never returned. The new showControls flag makes the choice explicit.

• ScreenshotBoundary installs a real RepaintBoundary and registers it with the manager. It was a StatelessWidget that returned its child, so capture photographed whichever boundary happened to come first in the tree, or nothing.

• The generator could not run on current Flutter at all. self_test_gen pinned analyzer below 9, and analyzer 7 throws Missing implementation of visitDotShorthandPropertyAccess while serialising any library that reaches the Flutter framework, which on Dart 3.12 and later is every library. build_runner build therefore worked on the declared floor, Flutter 3.35, and crashed on Flutter 3.47. It now takes analyzer >=8.1.1 <15.0.0, source_gen ^4.2.4 and build >=3.0.2 <5.0.0, which resolves to analyzer 10 on Dart 3.9 and analyzer 14 on Dart 3.13. Generated output is byte-identical under both. The quickstart CI job now runs on both ends of the range rather than on one of them, which is why nothing caught this.

• The core package no longer publishes the rest of the monorepo inside itself. A .pubignore keeps the bridge, the generator, the inspector and the MCP server's Node tree out of the self_test archive, and keeps extension/devtools/build in, because that is how DevTools finds the panel.

0.1.0

• Direct callback invocation for automated testing
• Code generation with build_runner and annotations
• SelfTestableWidget wrapper for manual setup
• TestScenario and TestStep for multi-step test flows
• Text assertion support for input field validation
• Screenshot capture during test execution
• Memory-safe test node management
• Runtime testing in debug/profile builds
• Test mode support for unit/integration tests
• Generated test controllers with full type safety

0.0.1

• Initial experimental release