sylph 0.4.0

pub package Build Status

Sylph

A sylph is a mythological invisible being of the air. Wikipedia

Sylph #

Sylph is a command line utility for running Flutter integration and end-to-end tests on pools of real iOS and Android devices in the cloud. Sylph runs on a developer mac or in a CI environment.

Installation #

pub global activate sylph

Usage #

sylph

or, if not using the default config file:

sylph -c <path to config file>

General usage:

usage: sylph [--help] [--config <config file>] [--devices <all|android|ios>]

sample usage: sylph

-c, --config=<sylph.yaml>          Path to config file.
                                   (defaults to "sylph.yaml")

-d, --devices=<all|android|ios>    List availabe devices.
                                   [all, android, ios]

    --help                         Display this help information.

Dependencies #

AWS CLI #

Install AWS Command Line Interface (AWS CLI)

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

For alternative install options see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

AWS CLI Credentials #

Configure the AWS CLI credentials:

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

For alternative configuration options see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

Configuration #

Configuration information is passed to Sylph using a configuration file. The default config file is called sylph.yaml:

# Config file for Flutter tests on real device pools.
# Auto-creates projects and device pools if needed.
# Configures android and ios test runs.
# Builds app, uploads and runs tests.
# Then monitors tests, returns final pass/fail result and downloads artifacts.
# Note: assumes the 'aws' command line utility is logged-in.
# Note: to build the debug iOS app, certain environment variables are required.

# sylph config
tmp_dir: /tmp/sylph
artifacts_dir: /tmp/sylph_artifacts
# local timeout per device farm run
sylph_timeout: 720 # seconds approx
# run on ios and android pools concurrently (for faster results)
concurrent_runs: true

# device farm config
project_name: test concurrent runs
default_job_timeout: 10 # minutes, set at project creation

device_pools:

  - pool_name: android pool 1
    pool_type: android
    devices:
      - name: Google Pixel 2
        model: Google Pixel 2
        os: 8.0.0

  - pool_name: ios pool 1
    pool_type: ios
    devices:
      - name: Apple iPhone X
        model: A1865
        os: 11.4

test_suites:

  - test_suite: example tests 1
    main: test_driver/main.dart
    tests:
      - test_driver/main_test.dart
    pool_names:
      - android pool 1
      - ios pool 1
    job_timeout: 15 # minutes, set per job, over-rides default job timeout above

Multiple test suites, consisting of multiple tests, can be run on each device in each device pool. The 'main' app must include a call to enableFlutterDriverExtension().

Device pools can consist of multiple devices. Devices in a device pool must be of the same type, iOS or Android.

Building an iOS debug app #

To build a testable iOS app locally, that can run on any real device in the cloud, the following environment variables must be present:

  • APP_IDENTIFIER
    This is the bundle identifier of your iOS app. For example, com.mycompany.myapp.
  • TEAM_ID
    This is the Developer Portal Team ID. It is of the form 'ABCDEFGHIJ'.

A check is made before the start of a run to confirm these environment variables are present.

Populating a device pool #

To add devices to a device pool, pick devices from the list provided by

sylph -d android
or
sylph -d ios

and add to the appropriate pool type in sylph.yaml. The listed devices are devices currently available on Device Farm.

Configuration Validation #

The sylph.yaml is validated to confirm the devices are available on Device Farm and tests are present before starting a run. Presence of the required environment variables for the iOS build are also confirmed.

Configuring a CI Environment for Sylph

iOS builds #

Special handling for building iOS apps is required for running tests on remote real devices. In particular, provisioning profiles and certificates must be installed on the build machine. To install the dependencies needed to complete the iOS build, Fastlane's match is used. Sylph will detect it is running in a CI environment (using the CI environment variable), and will install fastlane files that in turn will install the dependencies needed to build the iOS app using Fastlane's match. The iOS build can then complete as normal.

The following environment variables are required by a CI build to build the iOS app:

  • PUBLISHING_MATCH_CERTIFICATE_REPO
    This is the location of the private match repo. It expects an ssh-based url. For example, ssh://git@private.mycompany.com/private_repos/match.git
  • MATCH_PASSWORD
    This is the password that was used to encrypt the git repo's contents during match setup.

For details on how to configure Match see:
https://docs.fastlane.tools/actions/match/

The following are required by sylph in a CI environment to connect to the match host. The match host is running the ssh server that connects to the git server which serves the match repo. This configuration is required so that PUBLISHING_MATCH_CERTIFICATE_REPO will work via ssh:

  • MATCH_HOST
    This is used to configure the CI's ssh client to find the match host. For example, private.mycompany.com.
  • MATCH_PORT
    This is used to configure the CI's ssh client to find the match host's ssh port. For example, 22.

The following environment variables are also required by Sylph in a CI environment to configure the Fastlane Appfile and exportOptions.plist correctly:

  • APP_IDENTIFIER
    This is the bundle identifier of your iOS app. For example, com.mycompany.myapp.
  • TEAM_ID
    This is the Developer Portal Team ID. It is of the form 'ABCDEFGHIJ'.

AWS CLI Credentials for CI #

The following AWS CLI credentials are required:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

For details on other credentials see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

Note: the Travis-CI build uses pre-configured AWS CLI values in .aws/config

Sample environment variables for Travis-CI #

For example, when Sylph is run on Travis-CI the following environment variables are used:

secret variables

Upgrade #

To upgrade, simply re-issue the install command

$ pub global activate sylph

To check the version of Sylph currently installed:

pub global list

Live demo #

To see Sylph in action in a CI environment, a demo of the example app is available.

The log of the live run is here:
https://travis-ci.com/mmcc007/sylph

The resulting artifacts are here:
https://github.com/mmcc007/sylph/releases
(includes a video of test running on device)

Contributing #

When contributing to this repository, please feel free to discuss via issue or pull request.

Issues and pull requests are welcome.

Your feedback is used to guide where development effort is focused. So feel free to create as many issues and pull requests as you want. You should expect a timely and considered response.

0.4.0 #

  • Fixed multiple tests not running on android devices. #31
  • Added feature to support configuring a device pool. #35
    Enables looking-up devices from the command line.
  • Refactoring. #37
  • Added support for configurable iOS build. #30
    Requires adding environment variables.

0.3.0 #

  • Removed dependency on forked flutter #13
  • Added support for running sylph jobs in parallel #14
  • Added support for downloading artifacts per device #15
  • Added per job timeouts #17
  • Added reporting of test progress per device in pool #19
  • Added config validator at start of run #24 #25
  • Resolved potential security vulnerability in fastlane #27
  • Hid test_spec.yaml from user and allowed multiple tests to run on each device #20 #28

0.2.0 #

  • Added support for iOS real devices
    (now supports both iOS and Android real devices)

0.1.1 #

  • Downloads artifacts from AWS Device Farm
    (including video and log of running test)

0.1.0 #

  • Initial version

example/README.md

example #

A new Flutter project.

Getting Started #

This project is a starting point for a Flutter application.

A few resources to get you started if this is your first Flutter project:

For help getting started with Flutter, view our online documentation, which offers tutorials, samples, guidance on mobile development, and a full API reference.

Use this package as an executable

1. Install it

You can install the package from the command line:


$ pub global activate sylph

2. Use it

The package has the following executables:


$ sylph

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:


dependencies:
  sylph: ^0.4.0

2. Install it

You can install packages from the command line:

with pub:


$ pub get

with Flutter:


$ flutter pub get

Alternatively, your editor might support pub get or flutter pub get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:


import 'package:sylph/sylph.dart';
  
Popularity:
Describes how popular the package is relative to other packages. [more]
0
Health:
Code health derived from static analysis. [more]
100
Maintenance:
Reflects how tidy and up-to-date the package is. [more]
100
Overall:
Weighted score of the above. [more]
50
Learn more about scoring.

We analyzed this package on Aug 21, 2019, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.4.0
  • pana: 0.12.19

Platforms

Detected platforms: Flutter, other

Primary library: package:sylph/sylph.dart with components: io, isolate.

Dependencies

Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.0.0 <3.0.0
args ^1.5.1 1.5.2
duration ^2.0.8 2.0.8
isolate ^2.0.2 2.0.2
path ^1.6.2 1.6.4
resource ^2.1.5 2.1.6
sprintf ^4.0.2 4.0.2
version ^1.0.3 1.0.3
yaml ^2.1.15 2.1.16
Transitive dependencies
charcode 1.1.2
collection 1.14.12
meta 1.1.7
source_span 1.5.5
string_scanner 1.0.5
term_glyph 1.1.0
typed_data 1.1.6
Dev dependencies
pedantic ^1.8.0+1
test ^1.0.0