runFailingApps function

Future<int> runFailingApps(List<MatrixFailingApp> apps, { required String directory, MatrixAppTests appTests = const MatrixAppTests([]), MatrixCommands commands = const MatrixCommands(), MatrixFlutter flutter = _flutter, FileSystem fileSystem = const LocalFileSystem(), })

Generates each of apps in directory, with the options of CI, adds to it the tests of appTests that apply to it, and runs them: its expected failures must fail as they expect, and its other tests must pass; see runFailingAppTests. Returns the exit code: 0 if the contract harness builds each app without errors (MatrixFailingApp.check), smf create generates it with every module and every step that the options of CI do not leave for later, the app with its tests has no issue, and its tests fail as expected; 1 otherwise.

So the tests of the apps of the matrix show that they can fail: the app of a provider of a role that has a known bug must fail the tests of the role on that bug, and on nothing else.

The apps stay in directory, with the tests. The log of commands gets what happens, and their create generates each app, by default from the registry of the app; flutter runs the commands of the tests on the files of fileSystem. The analyze and the test of commands do not run.

Implementation

Future<int> runFailingApps(
  List<MatrixFailingApp> apps, {
  required String directory,
  MatrixAppTests appTests = const MatrixAppTests([]),
  MatrixCommands commands = const MatrixCommands(),
  MatrixFlutter flutter = _flutter,
  FileSystem fileSystem = const LocalFileSystem(),
}) async {
  // coverage:ignore-start
  // The default prints to the terminal, as the runs in CI do; the tests
  // give their own.
  final say = commands.log ?? _print;
  // coverage:ignore-end
  final problems = <String>[];
  var count = 0;
  for (final (index, failing) in apps.indexed) {
    final (:app, problems: checked) = await failing.check();
    if (app == null) {
      problems.addAll([
        for (final problem in checked) '${failing.name}: $problem',
      ]);
      continue;
    }
    final name = 'app_${index + 1}';
    say('\n=== $name: $app');
    final (generated, appProblems) = await _generate(
      // coverage:ignore-start
      // The default creates the app with smf, as CI does; the tests give
      // their own.
      commands.create ?? _smfCreate(failing.modules),
      // coverage:ignore-end
      app,
      name,
      directory,
    );
    problems.addAll(appProblems);
    if (generated == null) continue;
    count++;
    final (testProblems, output) = await runFailingAppTests(
      generated,
      app,
      [
        for (final test in appTests.tests)
          if (test.appliesTo(app)) test,
      ],
      failing.failures,
      flutter: flutter,
      fileSystem: fileSystem,
    );
    say(output.trim());
    problems.addAll([
      for (final problem in testProblems) '$name ($app): $problem',
    ]);
  }
  say('\n$count apps generated in $directory.');
  return _exitCode(problems, say);
}