dust 1.0.0-beta.9

  • Readme
  • Changelog
  • Installing
  • 38

Dust #

A coverage-guided fuzz tester for Dart.

inspired by libFuzzer and AFL etc.

Usage #

Simply write a dart program with a main function, and use the first argument as your input:

void main(List<String> args) {
  final input = args[0];

  /* use input */
}

The fuzz tester will look for crashes and report the failure output by randomly generating strings, passing them to your program, and adapting them in search of new code paths to maximize the exploration of your code for bugs. You can fuzz for all kinds of properties in your code by throwing exceptions when you wish.

To fuzz your script, simply run:

pub global activate dust
pub global run dust path/to/script.dart

Note: it is highly recommended to snapshot your script before running for better performance.

There are some special options you can see with pub global run dust --help to configure how exactly the fuzzer runs.

The Corpus #

By default, when you run dust on some script.dart, it will create a directory named script.dart.corpus that contains interesting fuzz samples used to explore your program (this is how coverage-guided fuzzing works). This means you can stop fuzzing your program and restart without losing progress.

Manual seeding #

You can specify manual seeds in one of two ways. You can either pass in seeds on the command-line, ie, --seed foo --seed bar, or you can pass in a seed directory (which may function much like a unit test directory) with --seed_dir.

If you do not specify manual seeds, the corpus begins as a single seed that is the empty string "".

When manually specifying seeds, they will only be added to the corpus if the coverage tool finds them interesting. Once interesting cases have been added to the corpus, you don't need to pass in those seeds again until your program perhaps changes in a meaningful way.

Minifying #

A corpus can be reduced using manual seeding, by simply providing the flags --seed_dir=old.corpus --corpus_dir=new.corpus. Optionally you may provide --count 0 to stop when the minimization is done.

Merging two corpuses #

You can also merge two corpuses by simply manually seeding one corpus into another, ie, --seed_dir=corpus1 --corpus-dir=corpus2.

Simplifying Cases #

You can simplify failing or non failing cases according to a simple character deletion search, and custom constraints. These constraints affect which simplifications are considered valid.

pub global run dust simplify path/to/script.dart input

There are useful constraints which default to off such as that no new paths are executed by the simplification, or that the error output from the case does not change.

By default, it is assumed you are simplifying a failure, and --constraint_failed is therefore on by default. However, it may be disabled to simplify non-failing cases as well.

Custom Mutators #

The default mutators will add, remove, or flip a random character in your seeds in attempt to search for new seeds. To specify custom behavior, you can write a script that can be spawned as an isolate by the main process:

import 'dart:isolate';
import 'package:dust/custom_mutator_helper.dart';

main(args, SendPort sendPort) => customMutatorHelper(sendPort, (str) {
  return ...; // mutate the string
});

To use this script, provide the flag --mutator_script=script.dart.

By default, each mutator (including the three default ones) have equal probability. However, you may set a weight on custom scripts by appending a : and then a double value, ie, --mutator_script=script.dart:2.0. The default mutators each have a weight of 1.0, and may be disabled entirely by passing --no_default_mutators.

Design #

Fuzz testing is often an excellent supplemental testing tool to add to programs where you need high stability.

The problem with black box fuzz testing is that the odds of striking a bad input are often easily demonstrably exceedingly low. Take this code:

if (x == 0) {
  if (y == 1) {
    if (z == 2) {
      throw "bet your fuzzer won't catch this!");
    }
  }
}

If x, y, and z are randomly chosen numbers, there is only a 1 in 2^32^3 chance of randomly getting through this code path.

This was first solved by inventing "white box" fuzz testing, which reads input code and uses it to generate constraints that it solves to generate test cases. This however is very challenging to do in a way that gives high coverage, as many constraints are hard to solve, and it usually involves code generation which is likely to be extremely complex.

White box fuzz testing was successful enough, however, to prompt the invention of grey box fuzz testing.

Grey box fuzz testing combines black box fuzz testing with code coverage instrumentation to guide the creation of a corpus of distinctly interesting fuzz cases. Those fuzz cases are then seeds to create new cases, and if those new cases provoke new code paths then they are added to the pool.

Going back to our code example, the first fuzz case to pass the first check (x == 0) will be saved and mutated until a case is found which also passes the second check, and so forth. While the odds of choosing the magical values 0, 1, and 2 may still be low, the chance of choosing all three together are greatly increased, and no special knowledge of the codes working is required. We only need to check code coverage of test cases.

We can do this in dart, too, using the VM service protocol.

Processes #

The fuzzer works like so:

  • User invokes fuzz's binary, passing in the location of a script to fuzz.
  • The fuzzer generates a basic seed (perhaps an empty string).
  • A seed is randomly chosen based on a fitness algorithm that values smaller seeds over shorter seeds, seeds that execute more paths over seeds that execute fewer, seeds that execute quicker vs seeds that take longer, and seeds that execute paths which are more unique relative to other seeds which execute paths that are more common.
  • That seed is then mutated n times, where we will attempt to concurrently run n fuzz tests at once.
  • n dart VMs are then started with debugging enabled, with a main script which knows the location of the target script to fuzz.
  • Each of the n mutations are passed to one of the n dart VMs, which execute that script in an isolate, which pauses on exit.
  • The main fuzz binary connects to the service protocol of the n VMs, and watches for the isolate completion events.
  • When the fuzz script isolates complete, the main fuzz binary will get coverage information for the fuzz isolates before closing them down, and recording whether they passed or failed and how long it took.
  • The coverage information for the new cases is compared to the old ones. If they executed new code paths, they are added to the pool of seeds.

TODO #

  • [ ] explore reusing isolates for better JITing. Locations will be cumulative rather than unique. When a fuzz test hits a new Location, rerun it in a fresh isolate.
  • [ ] investigate adding support for coverage in AOT apps, which will speed up running fuzz cases
  • [ ] improve error handling for cases where the dart VM crashes etc
  • [ ] provide coverage report in standardized format
  • [ ] some renames in the code: Library/Corpus
  • [ ] targeted scoring for paths through certain files/packages/etc
  • [ ] entropy based simplifier algorithm
  • [ ] automatic detection of simpler seeds during fuzzing?
  • [ ] store fuzzing options in script file (such as custom mutators, timeouts)
  • [ ] use locality sensitive hashing to dedupe failures with different messages (or in the case of timeouts, the same messages) by jaccard index of their code coverage sets. Perhaps from: https://arxiv.org/pdf/1811.04633
  • [ ] customizable limits & timeouts for simplifier?
  • [ ] special value recording via service extensions + kernel transformer?
  • [ ] break apart string comparisons with kernel transformer?
  • [ ] other service extensions?
  • [ ] other kernel transformer?
  • [ ] semantically-valid-dart transformer
  • [ ] generations of seeds?
  • [ ] way to exclude or score seeds when found?
  • [ ] change scoring of failed seeds?

etc.

  • 1.0.0-beta.9
  • enable asserts in fuzz target VMs
  • Added mutators to merge two existing seeds
  • Stack 1-3 changes rather than only doing 1 mutation.
  • Simplifier improvements: bugfixes, faster fixed-point search.
  • Fixed issue treating failures as executions (printing . & double counting)
  • New simplifier constraint "contains output"
  • Better error when port is already taken and observatory fails to start
  • Auto-snapshot scripts for users
  • Better error when script is erroneous (though erroneous snapshots may not give a good error message)
  • switch from vm_service_lib to vm_service
  • Coverage stats by path as well as files that have not been executed.
  • 1.0.0-beta.8
  • Fix critical bug that broke fuzzer when no limit was set.
  • 1.0.0-beta.7
  • Run manual seeds first, corpus seeds second, seed dir third, for better performance when the seed dir requires minimization.
  • 1.0.0-beta.6
  • Specifiable fuzz count limit, and exits 1 if any failures detected.
  • Time printouts during simplification
  • Warning if simplifier changed output / "same output" constraint no longer defaults to ON.
  • Run the simplifier to a fixed point when called from CLI.
  • Optimized optimizer for long input sequences.
  • 1.0.0-beta.5
  • Automatic simplification of seeds
  • Better simplification algorithm
  • Optimized profile collection. crash_on_bad.dart now runs 6x faster, though programs which do more meaningful work will see less gain.
  • Optimized simplifier. Will not collect coverage unless necessary.
  • 1.0.0-beta.4
  • Switch seed loading/persistence to more of a libfuzzer in/out/merge model.
  • Added option to persist failures to a directory
  • Changed compress_locations abbreviation from c to o.
  • 1.0.0-beta.3
  • Added a configurable timeout per case.
  • Changed default stats interval to 120s
  • Added seed loading/persistence to a directory
  • 1.0.0-beta.2
  • Added stat collection / printouts.
  • 1.0.0-beta.1
  • Added support for custom mutators.
  • 1.0.0-beta.0

Initial release

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:


dependencies:
  dust: ^1.0.0-beta.9

2. Install it

You can install packages from the command line:

with pub:


$ pub get

with Flutter:


$ flutter pub get

Alternatively, your editor might support pub get or flutter pub get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:


import 'package:dust/dust.dart';
  
Popularity:
Describes how popular the package is relative to other packages. [more]
0
Health:
Code health derived from static analysis. [more]
96
Maintenance:
Reflects how tidy and up-to-date the package is. [more]
45
Overall:
Weighted score of the above. [more]
38
Learn more about scoring.

We analyzed this package on Oct 18, 2019, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.5.1
  • pana: 0.12.21

Platforms

Detected platforms: Flutter, other

Primary library: package:dust/dust.dart with components: io.

Health suggestions

Fix lib/src/cli.dart. (-1 points)

Analysis of lib/src/cli.dart reported 2 hints:

line 28 col 8: Duplicate import.

line 35 col 8: Unused import: 'package:pedantic/pedantic.dart'.

Fix lib/src/driver.dart. (-0.50 points)

Analysis of lib/src/driver.dart reported 1 hint:

line 12 col 8: Unused import: 'package:dust/src/mutators.dart'.

Fix lib/src/failure_library.dart. (-0.50 points)

Analysis of lib/src/failure_library.dart reported 1 hint:

line 5 col 8: Unused import: 'dart:collection'.

Fix additional 4 files with analysis or formatting issues. (-2 points)

Additional issues in the following files:

  • lib/src/path_canonicalizer.dart (1 hint)
  • lib/src/path_scorer.dart (1 hint)
  • lib/src/simplify/simplifier.dart (1 hint)
  • lib/src/vm_controller.dart (1 hint)

Maintenance issues and suggestions

Homepage URL doesn't exist. (-20 points)

At the time of the analysis the homepage field https://github.com/MichaelRFairhurst/dust.dart was unreachable.

Support latest dependencies. (-10 points)

The version constraint in pubspec.yaml does not support the latest published versions for 1 dependency (vm_service).

The package description is too short. (-20 points)

Add more detail to the description field of pubspec.yaml. Use 60 to 180 characters to describe the package, what it does, and its target use case.

Package is pre-release. (-5 points)

Pre-release versions should be used with caution; their API can change in breaking ways.

Maintain an example.

None of the files in the package's example/ directory matches known example patterns.

Common filename patterns include main.dart, example.dart, and dust.dart. Packages with multiple examples should provide example/README.md.

For more information see the pub package layout conventions.

Dependencies

Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.2.2 <3.0.0
args ^1.5.2 1.5.2
collection ^1.14.11 1.14.12
crypto ^2.0.6 2.1.3
meta ^1.1.7 1.1.7
mockito ^4.1.0 4.1.1
path ^1.6.2 1.6.4
pedantic ^1.8.0+1 1.8.0+1
quiver ^2.0.3 2.0.5
test ^1.6.5 1.9.2
vm_service ^1.2.0 1.2.0 2.1.1
Transitive dependencies
analyzer 0.38.5
async 2.4.0
boolean_selector 1.0.5
charcode 1.1.2
convert 2.1.1
coverage 0.13.3
csslib 0.16.1
front_end 0.1.27
glob 1.2.0
html 0.14.0+3
http 0.12.0+2
http_multi_server 2.1.0
http_parser 3.1.3
io 0.3.3
js 0.6.1+1
kernel 0.3.27
logging 0.11.3+2
matcher 0.12.5
mime 0.9.6+3
multi_server_socket 1.0.2
node_interop 1.0.3
node_io 1.0.1+2
node_preamble 1.4.8
package_config 1.1.0
package_resolver 1.0.10
pool 1.4.0
pub_semver 1.4.2
shelf 0.7.5
shelf_packages_handler 1.0.4
shelf_static 0.2.8
shelf_web_socket 0.2.3
source_map_stack_trace 1.1.5
source_maps 0.10.8
source_span 1.5.5
stack_trace 1.9.3
stream_channel 2.0.0
string_scanner 1.0.5
term_glyph 1.1.0
test_api 0.2.9
test_core 0.2.13
typed_data 1.1.6
watcher 0.9.7+12
web_socket_channel 1.1.0
yaml 2.2.0