golden_toolkit 0.3.0

  • Readme
  • Changelog
  • Example
  • Installing
  • 78

Golden Toolkit #

Build Statuscodecov

This project contains APIs and utilities that build upon Flutter's Golden test functionality to provide powerful UI regression tests.

It is highly recommended to look at sample tests here: golden_builder_test.dart

Table of Contents #

Key Features #

GoldenBuilder #

The GoldenBuilder class lets you quickly test various states of your widgets given different sizes, input values or accessibility options. A single test allows for all variations of your widget to be captured in a single Golden image that easily documents the behavior and prevents regression.

Consider the following WeatherCard widget:

screenshot of widget under test

You might want to validate that the widget looks correct for different weather types:

testGoldens('Weather types should look correct', (tester) {
  final builder = GoldenBuilder.grid(columns:2)
          ..addScenario('Sunny', WeatherCard(Weather.sunny))
          ..addScenario('Cloudy', WeatherCard(Weather.cloudy))
          ..addScenario('Raining', WeatherCard(Weather.rain))
          ..addScenario('Cold', WeatherCard(Weather.cold));
  await tester.pumpWidgetBuilder(builder.build());
  await screenMatchesGolden(tester, 'weather_types_grid');
});

The output of this test will generate a golden: weather_types_grid.png that represents all four states in a single test asset.

example GoldenBuilder output laid out in a grid

A different use case may be validating how the widget looks with a variety of text sizes based on the user's device settings.

testGoldens('Weather Card - Accessibility', (tester) {
  final widget = WeatherCard(Weather.cloudy);
  final builder = GoldenBuilder.column()
          ..addScenario('Default font size', widget)
          ..addTextScaleScenario('Large font size', widget, textScaleFactor: 2.0)
          ..addTextScaleScenario('Largest font', widget, textScaleFactor: 3.2);
  await tester.pumpWidgetBuilder(builder.build());
  await screenMatchesGolden(tester, 'weather_accessibility');
});

The output of this test will be this golden file: weather_accessibility.png:

example GoldenBuilder output showing different text scales

See tests for usage examples: golden_builder_test.dart

multiScreenGolden #

The multiScreenGolden assertion is used to capture multiple goldens of a single widget using different simulated device sizes & characteristics.

The typical use case is to validate that a UI is responsive for both phone & tablet.

testGoldens('Example of testing a responsive layout', (tester) async {
  await tester.pumpWidgetBuilder(WeatherForecast());
  await multiScreenGolden(tester, 'weather_forecast');
});

This will generate the following two goldens:

weather_forecast.phone.png

example widget captured with phone dimensions

weather_forecast.tablet_landscape.png

example widget captured with tablet dimensions

You can also specify the exact device configurations you are interested in:

await multiScreenGolden(
        tester,
        'weather_forecast',
        devices: [
          const Device(
            name: 'strange_device',
            size: Size(100, 600),
          ),
          const Device(
            name: 'accessibility',
            size: Size(375, 667),
            textScale: 2.5,
          )
        ],
      );

Getting Started #

A Note on Golden Testing:

Goldens aren't intended to be a replacement of typical behavioral widget testing that you should perform. What they provide is an automated way to provide regression testing for all of the visual details that can't be validated without manual verification.

The Golden assertions take longer to execute than traditional widget tests, so it is recommended to be intentional about when they are used. Additionally, they can have many reasons to change. Often, the primary reason a golden test will fail is becaue of an intentional change. Thankfully, Flutter makes it easy to regenerate new reference images.

The rule of thumb that the eBay Motors team has used is to minimize our "full-screen" goldens and reserve them for high-level visual integration tests. Often, we only have one multiScreenGolden() test per screen.

For testing variations, edge cases, permutations, etc, we tend to use the GoldenBuilder focused on smaller widgets that have fewer reasons to change.

Setup #

If you are new to Flutter's Golden testing, there are a few things you might want to do

Add the failures folder to .gitignore

When golden tests fail, artifacts are generated in a failures folder adjacent to your test. These are not intended to be tracked in source control.

# don't check in golden failure output
**/failures/*.png

Configure VS Code

If you use VSCode, we highly recommend adding this configuration to your .vscode/launch.json file in the root of your workspace.

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Golden",
      "request": "launch",
      "type": "dart",
      "template": "run-test",
      "args": ["--update-goldens"]
    }
  ]
}

This give you a context menu where you can easily regenerate a golden for a particular test directly from the IDE:

Screenshot of 'Golden' shortcut in VSCode

Loading Fonts #

By default, flutter test only uses a single "test" font called Ahem. This font is designed to show black spaces for every character and icon. This obviously makes goldens much less valuable if you are trying to verify that your app looks correct.

To make the goldens more useful, we have a utility to dynamically inject additional fonts into the flutter test engine so that we can get more human viewable output.

In order to inject your fonts, we have a helper method:

  await loadAppFonts();

This function will automatically load the Roboto font, and any fonts included from packages you depend on so that they are properly rendered during the test.

Material icons like Icons.battery will be rendered in goldens ONLY if your pubspec.yaml includes:

flutter:
  uses-material-design: true

Note, if you need Cupertino fonts, you will need to find a copy of .SF UI Display Text.ttf, and .SF UI Text.ttf to include in your package's yaml. These are not included in this package by default for licensing reasons.

The easiest and recommended way to invoke this, is to create a flutter_test_config.dart file in the root of your package's test directory with the following content:

import 'dart:async';

import 'package:golden_toolkit/golden_toolkit.dart';

Future<void> main(FutureOr<void> testMain()) async {
  await loadAppFonts();
  return testMain();
}

For more information on flutter_test_config.dart, see the Flutter Docs

testGoldens() #

It is possible to use golden assertions in any testWidgets() test. As the UI for a widget evolves, it is common to need to regenerate goldens to capture your new reference images. The easiest way to do this is via the command-line:

flutter test --update-goldens

By default, this will execute all tests in the package. In a package with a large number of non-golden widget tests, we found this to be sub-optimal. We would much rather run ONLY the golden tests when regenerating. Initially, we arrived at a convention of ensuring that the test descriptions included the word 'Golden'

flutter test --update-goldens -name=Golden

However, there wasn't a way to enforce that developers named their tests appropriately, and this was error-prone.

Ultimately, we ended up making this testGoldens() function to enforce the convention. It has the same signature as testWidgets but it will automatically structure the tests so that the above flutter test command can work.

Additionally, the following test assertions will fail if not executed within testGoldens:

multiScreenGolden()
screenMatchesGolden()

Pumping Widgets #

Flutter Test's WidgetTester already provides the ability to pump a widget for testing purposes.

However, in many cases it is common for the Widget under test to have a number of assumptions & dependencies about the widget tree it is included in. For example, it might require Material theming, or a particular Inherited Widget. Often this setup is common and shared across multiple widget tests.

For convenience, we've created an extension for [WidgetTester] with a function pumpWidgetBuilder to allow for easy configuration of the parent widget tree & device configuration to emulate.

pumpWidgetBuilder has optional parameters wrapper, surfaceSize, textScaleSize

This is entirely optional, but can help reduce boilerplate code duplication.

Example:

 await tester.pumpWidgetBuilder(yourWidget, surfaceSize: const Size(200, 200));

wrapper parameter is defaulted to materialAppWrapper, but you can use your own custom wrappers.

Important: materialAppWrapper allows your to inject specific platform, localizations, locales, theme and etc.

Example of injecting light Theme:

      await tester.pumpWidgetBuilder(
        yourWidget,
        wrapper: materialAppWrapper(
          theme: ThemeData.light(),
          platform: TargetPlatform.android,
        ),
      );

Note: you can create your own wrappers similar to materialAppWrapper

See more usage examples here: golden_builder_test.dart

License Information #

Copyright 2019-2020 eBay Inc.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of eBay Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

3rd Party Software Included or Modified in Project #

This software contains some 3rd party software licensed under open source license terms:

  1. Roboto Font File: Available at URL: https://github.com/google/fonts/tree/master/apache/roboto License: Available under Apache license at https://github.com/google/fonts/blob/master/apache/roboto/LICENSE.txt

  2. Icons at: Author: Adnen Kadri URL: https://www.iconfinder.com/iconsets/weather-281 License: Free for commercial use

  3. OpenSans Font File: Available at URL: https://github.com/googlefonts/opensans License: Available under Apache license at https://github.com/googlefonts/opensans/blob/master/LICENSE.txt

Changelog #

0.3.0 #

Add support for configuring safe area (to simulate a device notch) and platform brightness (light/dark mode) on a multiScreenGolden device.

0.2.3 #

Breaking: Removed Future return type from testGoldens.

0.2.2 #

Added deviceSetup to multiscreenGolden to allow pumping customization when device configuration (ex: screen size) changes

0.2.1 #

Dropping pre-release tag. Improved documentation.

0.2.1-dev #

Resolved an issue when using loadAppFonts() to unit test a non-application package which includes one or more custom fonts.

0.2.0-dev #

Improved the mechanism for loading font assets. Consumers no longer need to supply a directory to read the .ttf files from.

They can now simply call: await loadAppFonts() and the package will automatically load any font assets from their pubspec.yaml or from any packages they depend on.

0.1.0-dev #

Initial release. Includes utility methods for easily pumping complex widgets, loading real fonts, and for writing more advanced Golden-based tests.

Features

  • GoldenBuilder for widget integration tests that visually indicate different states of a widgets in a single Golden test.
  • multiScreenGolden for pumping a widget and performing multiple Golden assertions with different device characteristics (e.g. screen size, accessibility options, etc)

We expect to make breaking changes over the next few releases as we stabilize the API.

example/readme.md

Golden Toolkit Usage Example #

Please have look at golden_toolkit tests here:

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:


dependencies:
  golden_toolkit: ^0.3.0

2. Install it

You can install packages from the command line:

with Flutter:


$ flutter pub get

Alternatively, your editor might support flutter pub get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:


import 'package:golden_toolkit/golden_toolkit.dart';
  
Popularity:
Describes how popular the package is relative to other packages. [more]
56
Health:
Code health derived from static analysis. [more]
100
Maintenance:
Reflects how tidy and up-to-date the package is. [more]
100
Overall:
Weighted score of the above. [more]
78
Learn more about scoring.

We analyzed this package on Mar 31, 2020, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.7.1
  • pana: 0.13.6
  • Flutter: 1.12.13+hotfix.8

Dependencies

Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.7.0 <3.0.0
flutter 0.0.0
flutter_test 0.0.0
meta ^1.1.8 1.1.8
Transitive dependencies
archive 2.0.11 2.0.13
args 1.5.2 1.6.0
async 2.4.0 2.4.1
boolean_selector 1.0.5 2.0.0
charcode 1.1.2 1.1.3
collection 1.14.11 1.14.12
convert 2.1.1
crypto 2.1.3 2.1.4
image 2.1.4 2.1.12
matcher 0.12.6
path 1.6.4
petitparser 2.4.0 3.0.2
quiver 2.0.5 2.1.3
sky_engine 0.0.99
source_span 1.5.5 1.7.0
stack_trace 1.9.3
stream_channel 2.0.0
string_scanner 1.0.5
term_glyph 1.1.0
test_api 0.2.11 0.2.15
typed_data 1.1.6
vector_math 2.0.8
xml 3.5.0 4.1.0
Dev dependencies
pedantic ^1.8.0 1.8.0+1 1.9.0
sample_dependency