Skip to main content

tarpit 0.1.0 copy "tarpit: ^0.1.0" to clipboard
tarpit: ^0.1.0 copied to clipboard

Find widget-test code that can hang in FakeAsync, and fail a stalled testWidgets fast. An analyzer-based CLI (text, JSON, SARIF) plus a watchdog.

tarpit #

Find the widget-test code that hangs your CI, before it runs, and make a stalled testWidgets fail in seconds with a reason instead of sitting until the timeout.

CI license: MIT

Why #

testWidgets runs the test body in a FakeAsync zone. Timers there fire only when the test pumps, and the results of real async work (files, sockets, isolates, the engine's image pipeline) wait in the fake zone for a pump that never comes. So one innocent line:

await File('test/fixtures/user.json').readAsString();

turns into a test that never finishes. Locally it looks like a frozen runner; in CI the job sits until the ten-minute test timeout or the job timeout, and the log does not say which line (flutter/flutter#5728, flutter/flutter#139870).

tarpit has two parts:

  1. A checker (dart run tarpit) that reads your tests with the Dart analyzer and points at the lines that will stall, with why and how to fix each one.
  2. A watchdog (package:tarpit/flutter_test.dart) that fails a widget test which makes no progress for 30 seconds, says what state it was stuck in, and lets the rest of the file run.

Rules #

Rule Severity Reports
fake-delay error await Future.delayed(...), Future(...) or pumpEventQueue()
real-io error an awaited dart:io call: files, directories, processes, sockets
isolate-hop error an awaited compute, Isolate.run/spawn, ReceivePort read
engine-async error an awaited toByteData, toImage, getNextFrame, precacheImage
blocking-sleep warning sleep(...) from dart:io
endless-settle warning pumpAndSettle() right after pumping an indeterminate progress indicator
asset-string info an awaited AssetBundle.loadString (uses an isolate at 50 KB and up)
bad-suppression warning a tarpit:allow comment without a known rule id or a reason

Only testWidgets (and tarpitWidgets) bodies are checked, and anything inside tester.runAsync(...) is allowed. The same pattern inside a nested closure the test may not await, or in a file whose imports do not resolve, is reported as info rather than an error. doc/rules.md has the details, examples and what each rule deliberately leaves alone.

Install #

flutter pub add --dev tarpit

Or install the command once for all projects (with the dart that comes with Flutter):

dart pub global activate tarpit

Run the checker #

dart run tarpit                 # checks test/ (every *_test.dart)
dart run tarpit test/widgets    # or given directories and files
tarpit                          # when activated globally

Run flutter pub get first: tarpit uses your project's resolved types, so file.readAsString() is known to be dart:io and not your own repository class. Output from the example project:

$ cd example && dart run tarpit
test/stalls_test.dart:22:11  error  fake-delay
  Awaits Future.delayed(...); its fake timer fires only when the test pumps, and the test is waiting instead of pumping.
  fix: Advance fake time with `await tester.pump(duration)` instead. If you really need wall-clock time, wrap it in `tester.runAsync(...)`.

test/stalls_test.dart:28:24  error  real-io
  Awaits File.readAsString() from dart:io outside tester.runAsync; the result arrives on the real event loop and the test never resumes.
  fix: Wrap the call in `await tester.runAsync(() => ...)`, or do the I/O in setUp/setUpAll, or use an in-memory fake (for example package:file).

test/stalls_test.dart:40:11  warning  endless-settle
  pumpAndSettle() right after pumping an indeterminate CircularProgressIndicator (line 38); the indicator animates forever, so this never settles.
  fix: Use `await tester.pump(duration)` while the indicator is visible, give the indicator a `value` in the test, or wait for the content with a bounded loop of pumps.

Why:
  fake-delay: testWidgets runs the body in a FakeAsync zone. Future.delayed, Future(...) and pumpEventQueue() schedule fake timers, which fire only when the test pumps. While the body is awaiting one, nothing pumps, so the await never returns.
  real-io: dart:io completes on the real event loop, but the continuation is queued in the FakeAsync zone, which only drains when the test pumps. The test waits for I/O that has finished but is never delivered.
  endless-settle: An indeterminate CircularProgressIndicator, LinearProgressIndicator, RefreshProgressIndicator or CupertinoActivityIndicator animates forever, so pumpAndSettle keeps pumping frames until its own timeout and then fails with "pumpAndSettle timed out".

3 findings (2 errors, 1 warning, 0 info) in 2 files.
$ echo $?
1

Options and exit codes #

Option Meaning
-f, --format text|json|sarif Output format (default text).
--fail-on error|warning|info Lowest severity that makes the exit code 1 (default warning).
-o, --output <file> Write the report to a file instead of stdout.
--rules List the rules.
--version, -h, --help

Exit code 0: nothing at or above --fail-on. 1: findings. 2: bad arguments, a path that does not exist, or a report that cannot be written.

--format json prints {"format": "tarpit-json", "formatVersion": 1, ...} with one object per finding (rule, severity, path, line, column, message, why, fix). --format sarif prints SARIF 2.1.0 for GitHub code scanning:

- run: dart run tarpit --format sarif --output tarpit.sarif || test $? -eq 1
- uses: github/codeql-action/upload-sarif@v4
  with:
    sarif_file: tarpit.sarif
- run: dart run tarpit

Suppressing a finding #

When a pattern is fine in one place, say why next to it:

await File('golden.txt').exists(); // tarpit:allow real-io runs only with the live binding

// tarpit:allow fake-delay,real-io this helper pumps from a fake timer
await waitForSync();

The comment works on its own line or on the line above. // tarpit:allow-file <rule-id> <reason> covers a whole file. The reason is required; without one, or with an unknown rule id, the comment is ignored and reported as bad-suppression.

The watchdog #

Change one import in a test file:

import 'package:tarpit/flutter_test.dart'; // instead of package:flutter_test

Every testWidgets in that file now runs under a watchdog. Or use tarpitWidgets(...) from package:tarpit/tarpit.dart for single tests, and watchTester(tester, description, body) inside a custom wrapper.

A test that stops making progress (the fake clock does not move and no frame is pumped) for Tarpit.stallAfter (30 seconds) fails with a TarpitStall, and the next test runs normally. A test that keeps pumping is never stopped, however long it takes. From flutter test in the example:

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TarpitStall was thrown running a test:
tarpit: "reads the name from a file" made no progress for 3.0s and was stopped.

  real time in the test body: 3.0s
  fake clock moved:           0.0s (it moves only when the test pumps)
  frames pumped:              0
  running tickers:            0
  frame requested, not pumped: no

The test body is awaiting a Future that nothing in the FakeAsync zone of testWidgets will complete.
Usual causes:
  - Future.delayed, Future(...) or pumpEventQueue() awaited instead of tester.pump(duration)
  - file, socket or process I/O, compute() or Isolate.run() awaited outside tester.runAsync(...)
  - a Completer or stream that waits for a platform channel, plugin or service the test does not
fake

Run `dart run tarpit` on the test file to find likely lines. If the test was inside tester.runAsync,
flutter_test still waits for that real work before it moves on. If the test is just slow without
pumping, raise the limit with stallAfter: or Tarpit.stallAfter.

Set the limit for a whole suite in test/flutter_test_config.dart:

import 'dart:async';

import 'package:tarpit/tarpit.dart';

Future<void> testExecutable(FutureOr<void> Function() testMain) async {
  Tarpit.stallAfter = const Duration(seconds: 20); // Duration.zero turns it off
  await testMain();
}

or per test with stallAfter:.

What the watchdog cannot do:

  • Engine image calls (toByteData, toImage, getNextFrame) block the test isolate itself, so no timer in it can fire. engine-async in the checker is the guard for those.
  • Inside tester.runAsync, the failure is reported on time, but flutter_test waits for the real work before it starts the next test.
  • experimentalLeakTesting is not passed through; use flutter_test's testWidgets for those tests.

How it works #

The checker builds an AnalysisContextCollection over the files, gets each file's resolved AST from package:analyzer, finds testWidgets calls whose body is a function literal, and walks the body: awaited expressions are classified by the library and class of the element they call (dart:io, dart:isolate, dart:ui, package:flutter), not by text. Without resolved types it only reports what it can recognise by name, as info.

The watchdog runs a real timer in the root zone next to the test's fake one. When the fake clock and the frame count stay still for the limit, it reports a TarpitStall into the test's zone, which flutter_test's binding handles like any other test failure, and drains the binding's pending microtasks so the test completes.

Contributing #

Issues and pull requests are welcome; see CONTRIBUTING.md. New rules need a reproduction that hangs or fails a real widget test. tarpit is the successor to imaji_test_hang.

License #

MIT, see LICENSE.

0
likes
0
points
861
downloads

Publisher

unverified uploader

Weekly Downloads

Find widget-test code that can hang in FakeAsync, and fail a stalled testWidgets fast. An analyzer-based CLI (text, JSON, SARIF) plus a watchdog.

Repository (GitHub)
View/report issues

Topics

#testing #flutter #lint #ci #static-analysis

License

unknown (license)

Dependencies

analyzer, args, flutter, flutter_test, meta, path

More

Packages that depend on tarpit