measureMatrix function

Future<ParityMatrix> measureMatrix(Iterable<MatrixEntry> entries, { Tolerance? tolerance, })

Replays one set of goldens across several backends and reports all of them.

Each entry is measured at the tolerance its own manifest implies, which is the only way the row means anything: an int8 export and a float32 one of the same model are not wrong by the same amount and should not answer to the same bound. tolerance overrides that for every entry, which is what a caller wants when the question is how the backends compare rather than whether each one meets its own recipe.

Entries whose goldens do not line up are refused rather than measured. A matrix over different inputs is a table of unrelated numbers arranged to look like a comparison.

Implementation

Future<ParityMatrix> measureMatrix(
  Iterable<MatrixEntry> entries, {
  Tolerance? tolerance,
}) async {
  final list = entries.toList();
  if (list.isEmpty) {
    throw ArgumentError.value(
      entries,
      'entries',
      'a matrix over no backends measures nothing; supply at least one',
    );
  }

  final ids = [for (final c in list.first.goldens.cases) c.id];
  for (final entry in list.skip(1)) {
    final theirs = [for (final c in entry.goldens.cases) c.id];
    if (!_sameOrder(ids, theirs)) {
      throw ArgumentError.value(
        entry.model.backend,
        'entries',
        'carries goldens ${theirs.join(", ")} where the first entry carries '
            '${ids.join(", ")}. A matrix compares one set of inputs across '
            'backends, and these are different questions side by side',
      );
    }
  }

  final reports = <DriftReport>[];
  for (final entry in list) {
    reports.addAll(
      await measureParity(
        entry.model,
        goldens: entry.goldens,
        tolerance: tolerance,
      ),
    );
  }

  return ParityMatrix(
    backends: [for (final e in list) e.model.backend],
    goldenIds: ids,
    reports: reports,
  );
}