expectedFailureProblems function

List<String> expectedFailureProblems(String report, List<MatrixExpectedFailure> expected, { String? directory, })

The problems of a run of flutter test --reporter json in an app whose tests of expected must fail, and whose other tests must pass; report is what the run wrote. A problem shows a test file by its path in the app when the report has it in directory, the directory of the app.

An expected failure is met when its test ran in the test file that it names, did not pass, and failed first on an expectation, a thrown TestFailure such as that of a failed expect, whose message contains its reason. package:test reports such a failure as an error of the test that is a failure (isFailure). flutter_test reports each failure of a widget test as an error that refers to what it printed before, Test failed. See exception logs above., with the result error, so the first failure of such a test is the first thing that flutter_test printed as thrown: The following TestFailure was thrown … for a failed expectation.

The problems are: the tests passed; an expected test did not run, did not finish, passed or was skipped; it failed first on an expectation whose message lacks the reason, or with an error rather than a failed expectation, such as an exception, a timeout or a test file that does not load, each quoting the first line of its message; another test did not pass or did not finish, as when a test file does not compile, a hook such as setUpAll fails, or the bug fails a test that it must not; and the run did not finish.

A test that its declaration skips, with skip: of the test or of its group, which the report shows when the test starts, is skipped in every app alike, so it is no problem. A test that skips itself while it runs may do so because of the bug, so it is one.

Implementation

List<String> expectedFailureProblems(
  String report,
  List<MatrixExpectedFailure> expected, {
  String? directory,
}) {
  final run = _Run.parse(report);
  final problems = [
    if ((run.passed ?? false) && expected.isNotEmpty)
      'The tests passed, but ${expected.join(', ')} must fail.',
  ];
  final matched = <_Test>{};
  for (final failure in expected) {
    final test = run.testOf(failure);
    if (test == null) {
      problems.add('$failure did not run.');
      continue;
    }
    matched.add(test);
    if (_expectedProblem(test, failure) case final problem?) {
      problems.add(problem);
    }
  }
  for (final test in run.tests) {
    if (matched.contains(test)) continue;
    if (_otherProblem(test, directory) case final problem?) {
      problems.add(problem);
    }
  }
  if (run.passed == null) problems.add('The run of the tests did not finish.');
  return problems;
}