sylph 0.3.0

A utility for running Flutter integration tests on pools of real devices in cloud. Works with AWS Device Farms.

A sylph is a mythological invisible being of the air. Wikipedia

Sylph

Sylph is a command line utility for running Flutter integration and end-to-end tests on pools of real iOS and Android devices in the cloud. Sylph runs on a developer mac or in a CI environment.

Installation

pub global activate sylph

Usage

sylph

or, if not using the default config file:

sylph -c <path to config file>

Configuration

All configuration information is passed to Sylph using a configuration file. The default config file is called sylph.yaml:

project_name: flutter tests
default_job_timeout: 5 # minutes

tmp_dir: /tmp/sylph
artifacts_dir: /tmp/sylph_artifacts

# local timeout per device farm run
sylph_timeout: 720 # seconds approx
# run on ios and android pools concurrently (for faster results)
concurrent_runs: true

device_pools:

  - pool_name: android pool 1
    pool_type: android
    devices:
      - name: Samsung Galaxy S9 (Unlocked)
        model: SM-G960U1
        os: '8.0.0'

  - pool_name: ios pool 1
    pool_type: ios
    devices:
      - name: Apple iPhone X
        model: A1865
        os: '12.0'

test_suites:

  - test_suite: example tests 1
    main: test_driver/main1.dart
    tests:
      - test_driver/main1_test1.dart
      - test_driver/main1_test2.dart
    device_pools:
      - android pool 1
      - ios pool 1
    job_timeout: 5 # minutes per each device run
    
  - test_suite: example tests 2
    main: test_driver/main2.dart
    tests:
      - test_driver/main2_test1.dart
      - test_driver/main2_test2.dart
    pool_names:
      - android pool 1
      - ios pool 1
    job_timeout: 5 # minutes per each device run

Dependencies

AWS CLI

Install AWS Command Line Interface (AWS CLI)

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

For alternative install options see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

AWS CLI Credentials

Configure the AWS CLI credentials:

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

For alternative configuration options see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

Configuring a CI Environment for Sylph

iOS builds

Special handling for building iOS apps is required for running tests on remote real devices. In particular, provisioning profiles and certificates must be installed on the build machine. To install the dependencies needed to complete the iOS build, Fastlane's match is used. Sylph will detect it is running in a CI environment (using the CI environment variable), and will install fastlane files that in turn will install the dependencies needed to build the iOS app using Fastlane's match. The iOS build can then complete as normal.

The following environment variables are required by a CI build to build the iOS app:

• PUBLISHING_MATCH_CERTIFICATE_REPO
  This is the location of the private match repo. It expects an ssh-based url. For example, ssh://[e-mail]/private_repos/match.git
• MATCH_PASSWORD
  This is the password that was used to encrypt the git repo's contents during match setup.

For details on how to configure Match see:
https://docs.fastlane.tools/actions/match/

The following are required by sylph in a CI environment to connect to the match host. The match host is running the ssh server that connects to the git server which serves the match repo. This configuration is required so that PUBLISHING_MATCH_CERTIFICATE_REPO will work via ssh:

• MATCH_HOST
  This is used to configure the CI's ssh client to find the match host. For example, private.mycompany.com.
• MATCH_PORT
  This is used to configure the CI's ssh client to find the match host's ssh port. For example, 22.

AWS CLI Credentials for CI

The following AWS CLI credentials are required:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

For details on other credentials see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

Note: the Travis-CI build uses pre-configured AWS CLI values in .aws/config

Example secrets for Travis-CI

Sylph runs on Travis-CI and expects the following environment variables:

Live demo

To see Sylph in action in a CI environment, a demo of the example app is available.

The log of the live run is here:
https://travis-ci.com/mmcc007/sylph

The resulting artifacts are here:
https://github.com/mmcc007/sylph/releases
(includes a video of test running on device)

Contributing

When contributing to this repository, please feel free to discuss via issue or pull request.

Issues and pull requests are welcome.

Your feedback is used to guide where development effort is focused. So feel free to create as many issues and pull requests as you want. You should expect a timely and considered response.