genUiFuzz function
- required CatalogImplementation catalog,
- required GenUiPump pump,
- GenUiFuzzHost host = _plainHost,
- Set<
String> only = const <String>{}, - Set<
String> skip = const <String>{}, - int maxCasesPerComponent = 80,
- bool reportEmpty = false,
Renders everything catalog's own examples allow, mutated, and reports
what broke.
A catalog is a contract with something that cannot be recompiled, and the only thing holding a model to it is a JSON schema. Every widget test writes the input the author had in mind; this writes the inputs the schema permits and the author did not. An optional property left out, a binding that never resolves, a call to a function nobody registered, a list with two hundred entries: each of those is a message a real agent can send, and each one renders here before it renders in front of someone.
The cases are built by mutating each component's own generated example rather than by reading the schema, because the example is valid by construction and already names every required property. Each case then goes through the catalog's own validation, as an agent's message would. A case the schema rejects is the protocol working, so it is skipped; what is reported is a case the schema accepts and the widget cannot take. A hand-written GenUiComponent is fuzzed as well as a generated one, as long as it carries an example.
testWidgets('the catalog survives what its schema allows', (tester) async {
final findings = await genUiFuzz(
catalog: exampleCatalog,
pump: tester.pumpWidget,
);
expect(findings, isEmpty, reason: findings.join('\n\n'));
});
reportEmpty also reports a case that rendered nothing at all. It is off
by default because most of those are correct: take the items away from a
list and it has nothing to draw. Turn it on when you want to find the
component that swallows a wrong-typed property and leaves a blank space
where the agent asked for something.
host puts each surface inside the app it is drawn in. Left out, that is
a plain MaterialApp. Pass the app's own theme, and whatever else its
components read from above, when they depend on it:
final findings = await genUiFuzz(
catalog: exampleCatalog,
pump: tester.pumpWidget,
host: (surface) => MaterialApp(
theme: appTheme,
home: Scaffold(body: SingleChildScrollView(child: surface)),
),
);
Keep the surface in a parent that lets it grow, as the scroll view above does, or a tall component is reported as overflowing.
skip leaves components out by name, for the ones whose failure you have
already accepted. only restricts the run to a few while you are fixing
one. maxCasesPerComponent bounds a component with many properties; the
cases are generated in a fixed order, so a run is reproducible and a bound
always drops the same tail.
Implementation
Future<List<GenUiFuzzFinding>> genUiFuzz({
required CatalogImplementation catalog,
required GenUiPump pump,
GenUiFuzzHost host = _plainHost,
Set<String> only = const <String>{},
Set<String> skip = const <String>{},
int maxCasesPerComponent = 80,
bool reportEmpty = false,
}) async {
final findings = <GenUiFuzzFinding>[];
var surface = 0;
for (final ComponentImplementation component in catalog.components.values) {
if (component is! GenUiComponent) continue;
if (skip.contains(component.name)) continue;
if (only.isNotEmpty && !only.contains(component.name)) continue;
final List<_Json>? example = _parseExample(component);
if (example == null) continue;
final int rootIndex = example.indexWhere((c) => c['id'] == 'root');
final _Json root = example[rootIndex < 0 ? 0 : rootIndex];
final int index = rootIndex < 0 ? 0 : rootIndex;
// The untouched example first: it is the baseline every case is compared
// against, and a component whose own example throws needs no fuzzing to
// be worth reporting.
final _Render baseline = await _render(
pump,
host,
catalog,
example,
's${surface++}',
);
if (baseline.rejection != null || baseline.error != null) {
findings.add(
GenUiFuzzFinding(
component: component.name,
mutation: 'its own generated example',
kind: baseline.rejection != null
? GenUiFuzzKind.rejected
: GenUiFuzzKind.threw,
detail: baseline.rejection ?? baseline.error!,
componentJson: jsonEncode(root),
),
);
continue;
}
for (final _Case c in _casesFor(root).take(maxCasesPerComponent)) {
final List<_Json> components = <_Json>[...example];
components[index] = c.component;
final _Render r = await _render(
pump,
host,
catalog,
components,
's${surface++}',
);
// The schema turned it away, which is the protocol doing its job: an
// agent that sends it gets an error back, and nothing renders.
if (r.rejection != null) continue;
if (r.error != null) {
findings.add(
GenUiFuzzFinding(
component: component.name,
mutation: c.label,
kind: GenUiFuzzKind.threw,
detail: r.error!,
componentJson: jsonEncode(c.component),
),
);
} else if (reportEmpty && baseline.hasSize && !r.hasSize) {
findings.add(
GenUiFuzzFinding(
component: component.name,
mutation: c.label,
kind: GenUiFuzzKind.renderedNothing,
detail: 'the untouched example took up space and this took up none',
componentJson: jsonEncode(c.component),
),
);
}
}
}
return findings;
}