flutter-ci

Motivation

The coverage rate threshold on CI tools is a common approach to encourage developers to write tests. However, judging a pull request solely based on the coverage rate of the entire project is not always fair, particularly when it comes to significant refactor tasks that may naturally reduce the coverage rate.

This package adopts a different approach to analyze test coverage: it specifically focuses on checking the lines added in the pull request. The coverage rate is then calculated by dividing the number of uncovered new lines by the total number of new lines.

In addition, you have the option to set thresholds that will cause tests to fail on a CI tool. Furthermore, this package can also generate a report highlighting the uncovered code, making it easier to identify areas that require additional tests.

Installing

There are two ways to install the pull_request_coverage package. Since it is a binary package, you can activate it from the command line by using the command dart pub global activate pull_request_coverage. This will allow you to use it as a standalone program on your command-line interface (CLI). Alternatively, you can add it to the dev_dependencies section of your project's pubspec.yaml file.

ℹ️ Github Actions example

Usage

This package requires two pieces of information to analyze the test coverage:

  • An lcov.info file, which is generated by running the flutter test --coverage command
  • A diff between the current branch and the target branch, generated using the git diff command

Generating the lcov.info file

There is a known issue with the flutter test --coverage command. It may not report untested files. There is a workaround for it, described in this issue

Run the following command to generate the coverage/lcov.info file:

flutter test --coverage

If you want to analyze a Dart project rather than Flutter, use the coverage package

Running pull_request_coverage

To check the PR's code, pull_request_coverage needs a diff between its branch and the target one. The diff is read from the STDIN input.

You can pipe the STDIN to pull_request_coverage using bash's | operator, like this:

git diff repository/main | flutter pub run pull_request_coverage

If you activate pull_request_coverage using dart pub global activate, you can invoke it directly:

git diff repository/main | pull_request_coverage

See Example tab to check an output example out

Settings

There are two ways to configure the execution of pull_request_coverage: using CLI arguments or a YAML config file. Using a YAML file is recommended for large teams as the settings can be shared and used consistently across CI scripts. This eliminates the need for each developer to manually copy and paste the arguments to analyze coverage locally.

By default, pull_request_coverage looks for the ./pull_request_coverage.yaml file to load its settings. You can override this by using the config-file argument.

Both methods support the same settings, and if a setting is provided through both CLI arguments and the YAML file, the value from the CLI arguments will take precedence.

Examples

CLI args
git diff origin/main | flutter pub run pull_request_coverage --maximum-uncovered-lines 5 --ignore '/lib/di/**','**/gen.dart' --ignore-lines "^.*@override.*$"
YAML config file
maximum-uncovered-lines: 5
ignore:
  - /lib/di/**
  - "**/gen.dart"
ignore-lines:
  - "^.*@override.*$"

Available Options

Default values are indicated in parentheses.

Input

  • lcov-file (coverage/lcov.info): Path to the lcov.info file generated by the flutter test --coverage command.
  • config-file (pull_request_coverage.yaml): Path to the YAML settings file.

Threshold

  • minimum-coverage : Specifies the minimum coverage rate required for the tests to pass.

  • maximum-uncovered-lines : Specifies the maximum number of uncovered lines allowed before the tests fail.

  • approval-requirement (lines-and-rate): when both, minimum-coverage and maximum-uncovered-lines, are specified, the approval-requirement determines the conditions for passing the tests.

    • lines-and-rate: both conditions must be met to pass the tests.
    • lines-or-rate: only one of the conditions must be met to pass the tests.

Filters

  • ignore: list of files that should be ignored, using the widely-known Bash glob syntax. The cumulative count of ignored lines from these files will be shown in the report for statistical purposes only.

  • ignore-lines: A list of regular expressions used to filter lines in the source code. The cumulative count of ignored lines will be shown in the report for statistical purposes only.

  • ignore-known-generated-files (true) : Ignores file paths that end with .g.dart, .pb.dart, .pbenum.dart, .pbserver.dart or .pbjson.dart

  • add-to-known-generated-files: list of glob matchers to extend the list specified in ignore-known-generated-files. Unlike ignore, the lines matched by these patterns will be completely excluded from the report.

Presentation

Check example out to see how those params can change the output

  • output-mode (cli): Specifies the output format.

    • cli: this output format is designed to be read on the command-line interface (CLI).
    • markdown: this output is formatted using Markdown syntax, which is useful for displaying the output in a pull request comment, posted by a bot, for example.

  • markdown-mode (diff): Specifies the Markdown output format (see example page).

    • diff: Uses diff syntax to highlight the uncovered lines.
    • dart: Uses Dart syntax to display the code with an appropriate color scheme, adding a comment at the end of uncovered lines.

  • report-fully-covered-files (true): Prints the file path of each fully covered file as a celebratory message =)

  • show-uncovered-code (true): When set to true, the source code of the uncovered lines will be printed in red font color, making it easier to identify the missing tests. If this parameter is set to false, only the file path will be shown in the log.

  • use-colorful-output (true): By default, pull_request_coverage utilizes a colorful font to highlight uncovered lines. You can disable this feature by setting this parameter to false. Please note that this option is only available when the output mode is set to cli.

  • print-emojis (true) : Enable or disabled emojis in the output 🫣.

  • fraction-digits (2): Specifies the number of digits to display after the decimal point in the coverage rate.

  • fully-tested-message : Replace the final table report with a custom message when there are no untested lines

Under the hood

  • stdin-timeout (1): pull_request_coverage reads the diff from stdin. In certain cases, the stdin stream may never close, causing the analysis to become stuck. By default, if no data is received within one second, pull_request_coverage assumes that it has reached the EOF.

  • log-level (none): internal log level. Possible values are none, error, warning, info, debug and verbose.

Exit code

  • 0 - Tests passed.
  • 1 - Tests failed (only when thresholds are set).
  • 255 - Execution has failed and tests were not executed.

Libraries