compareScenarioRuns function

ScenarioRunDrift compareScenarioRuns(ScenarioRunResult before, ScenarioRunResult after, { String? baseline, })

Two runs of the same suite, compared by what their steps recorded.

A suite can be entirely green while a third of its screenshots move every pass, and nothing else in a run report shows it: every scenario passes, every assertion holds, and the pictures underneath differ. On a real 125-scenario suite the causes were a fixture chosen by hashing a freshly-minted id, generated ids drawn on screen as text, and two production writes reaching for DateTime.now() instead of the injectable clock — a day to find by eye, an afternoon with this.

It matters beyond tidiness: a comparison against a base checkout is only as useful as the suite's determinism, because a suite that moves on its own drowns the change that was actually asked about.

What is compared. The captured surface, and the facets recorded beside it that the surface does not contain — see ScenarioDriftFacet. A status bar that stopped being tinted moves no pixel of a capture that never included the status bar, and a comparison that reads that as "nothing moved" is worse than one that says nothing at all.

Which steps. A step carrying a digest in both runs. A step with no digest is not compared on its other facets either, and a scenario with no digested step at all is not looked at: a notification beat captured no bytes, and a run written by a flutterware that predates digests carries none anywhere, which answers here as two runs with nothing to compare rather than as a suite that moved entirely.

How they are paired. By package, axis point, file, scenario, branch and then — within the branch — by the Shot's name. Never by ScenarioRunStep.index, which shifts under any insertion, and not by ScenarioRunStep.position alone, which shifts under one too: its ordinal counts captures since the branch began, so inserting a step renumbers every step after it and the comparison reports a cascade — one changed, one added and one removed per step below the cut — that says nothing about either run. A name does not move when a step is inserted beside it. An unnamed step is pinned to the nearest name above it plus its distance from it, so an insertion disturbs only the unnamed steps between two names; one before its branch's first name has nothing to hold on to and falls back to its position. ScenarioRunDrift.nameMatched reports how much of the comparison was anchored rather than guessed, which is the number that says whether the rest of the report can be trusted.

Only scenarios both runs contain are looked at. One of them may have been a selective run — one file, one name, one tag — and a scenario the other never executed is not a scenario that moved. A scenario genuinely added or deleted between the two is a change to the suite, which the suite already knows about; this function is for unintended change.

baseline names before for a reader who did not choose it — the run directory, worktree-relative. It defaults to the one before recorded for itself.

Implementation

ScenarioRunDrift compareScenarioRuns(
  ScenarioRunResult before,
  ScenarioRunResult after, {
  String? baseline,
}) {
  var changed = <ScenarioStepDrift>[];
  var added = <ScenarioStepDrift>[];
  var removed = <ScenarioStepDrift>[];
  var compared = 0;
  var nameMatched = 0;
  var unanchored = 0;

  var shared = _scenarios(before).intersection(_scenarios(after));
  // A run on the real clock drew its pictures off live data, so two of them
  // differ by weather. Everything the pixels do not carry — status bar tint,
  // keyboard height, whether a step settled — is still compared.
  var live =
      before.packages.any((p) => p.isRealTime) ||
      after.packages.any((p) => p.isRealTime);

  var old = <String, _Compared>{};
  for (var step in _walk(before)) {
    if (step.record != null && shared.contains(step.scenario)) {
      old[step.key] = step.record!;
    }
  }

  var seen = <String>{};
  for (var step in _walk(after)) {
    if (step.record == null || !shared.contains(step.scenario)) continue;
    seen.add(step.key);
    var was = old[step.key];
    if (was == null) {
      added.add(step.drift);
      continue;
    }
    compared++;
    if (step.pairing == _Pairing.name) nameMatched++;
    if (step.pairing == _Pairing.position) unanchored++;
    var moved = was.difference(step.record!, pixels: !live);
    if (moved.isNotEmpty) changed.add(step.drift.movedIn(moved));
  }

  for (var step in _walk(before)) {
    if (step.record != null &&
        shared.contains(step.scenario) &&
        !seen.contains(step.key)) {
      removed.add(step.drift);
    }
  }

  return ScenarioRunDrift(
    compared: compared,
    nameMatched: nameMatched,
    unanchored: unanchored,
    baseline: baseline ?? before.packages.firstOrNull?.output,
    pixelsIgnored: live,
    changed: changed,
    added: added,
    removed: removed,
  );
}