petitparser 3.0.1

  • Readme
  • Changelog
  • Example
  • Installing
  • 99

PetitParser for Dart #

Pub Package Build Status Coverage Status GitHub Issues GitHub Forks GitHub Stars GitHub License

Grammars for programming languages are traditionally specified statically. They are hard to compose and reuse due to ambiguities that inevitably arise. PetitParser combines ideas from scannnerless parsing, parser combinators, parsing expression grammars and packrat parsers to model grammars and parsers as objects that can be reconfigured dynamically.

This library is open source, stable and well tested. Development happens on GitHub. Feel free to report issues or create a pull-request there. General questions are best asked on StackOverflow.

The package is hosted on dart packages. Up-to-date API documentation is created with every release.

Tutorial #

Below are step by step instructions of how to write your first parser. More elaborate examples (JSON parser, LISP parser and evaluator, Prolog parser and evaluator, etc.) are included in the example repository.

Installation #

Follow the installation instructions on dart packages.

Import the package into your Dart code using:

import 'package:petitparser/petitparser.dart';

Writing a Simple Grammar #

Writing grammars with PetitParser is simple as writing Dart code. For example, to write a grammar that can parse identifiers that start with a letter followed by zero or more letter or digits is defined as follows:

final id = letter() & (letter() | digit()).star();

If you look at the object id in the debugger, you'll notice that the code above builds a tree of parser objects:

  • SequenceParser: This parser accepts a sequence of parsers.
    • CharacterParser: This parser accepts a single letter.
    • PossessiveRepeatingParser: This parser accepts zero or more times another parser.
      • ChoiceParser: This parser accepts a single word character.
        • CharacterParser: This parser accepts a single letter.
        • CharacterParser: This parser accepts a single digit.

The operators & and | are overloaded and create a sequence and a choice parser respectively. In some contexts it might be more convenient to use chained function calls:

final id = letter().seq(letter().or(digit()).star());

Parsing Some Input #

To actually parse a String (or List) we can use the method Parser.parse:

final id1 = id.parse('yeah');
final id2 = id.parse('f12');

The method Parser.parse returns a parse Result, which is either an instance of Success or Failure. In both examples above we are successful and can retrieve the parse result using Success.value:

print(id1.value);                   // ['y', ['e', 'a', 'h']]
print(id2.value);                   // ['f', ['1', '2']]

While it seems odd to get these nested arrays with characters as a return value, this is the default decomposition of the input into a parse tree. We'll see in a while how that can be customized.

If we try to parse something invalid we get an instance of Failure as an answer and we can retrieve a descriptive error message using Failure.message:

final id3 = id.parse('123');
print(id3.message);                 // 'letter expected'
print(id3.position);                // 0

Trying to retrieve the parse result by calling Failure.value would throw the exception ParserError. Context.isSuccess and Context.isFailure can be used to decide if the parse was successful.

If you are only interested if a given string matches or not you can use the helper method Parser.accept:

print(id.accept('foo'));            // true
print(id.accept('123'));            // false

Different Kinds of Parsers #

PetitParser provide a large set of ready-made parser that you can compose to consume and transform arbitrarily complex languages. The terminal parsers are the most simple ones. We've already seen a few of those:

  • char('a') parses the character a.
  • string('abc') parses the string abc.
  • any() parses any character.
  • digit() parses any digit from 0 to 9.
  • letter() parses any letter from a to z and A to Z.
  • word() parses any letter or digit.

So instead of using the letter and digit predicate, we could have written our identifier parser like this:

final id = letter() & word().star();

The next set of parsers are used to combine other parsers together:

  • p1 & p2 and p1.seq(p2) parse p1 followed by p2 (sequence).
  • p1 | p2 and p1.or(p2) parse p1, if that doesn't work parse p2 (ordered choice).
  • parses p zero or more times.
  • parses p one or more times.
  • p.optional() parses p, if possible.
  • p.and() parses p, but does not consume its input.
  • p.not() parses p and succeed when p fails, but does not consume its input.
  • p.end() parses p and succeed at the end of the input.

To attach an action or transformation to a parser we can use the following methods:

  • => ...) performs the transformation given the function.
  • p.pick(n) returns the n-th element of the list p returns.
  • p.flatten() creates a string from the result of p.
  • p.token() creates a token from the result of p.
  • p.trim() trims whitespaces before and after p.

To return a string of the parsed identifier, we can modify our parser like this:

final id = (letter() & word().star()).flatten();

To conveniently find all matches in a given input string you can use Parser.matchesSkipping:

final matches = id.matchesSkipping('foo 123 bar4');
print(matches);                     // ['foo', 'bar4']

These are the basic elements to build parsers. There are a few more well documented and tested factory methods in the Parser class. If you want browse their documentation and tests.

Writing a More Complicated Grammar #

Now we are able to write a more complicated grammar for evaluating simple arithmetic expressions. Within a file we start with the grammar for a number (actually an integer):

final number = digit().plus().flatten().trim().map(int.parse);

Then we define the productions for addition and multiplication in order of precedence. Note that we instantiate the productions with undefined parsers upfront, because they recursively refer to each other. Later on we can resolve this recursion by setting their reference:

final term = undefined();
final prod = undefined();
final prim = undefined();

term.set(prod.seq(char('+').trim()).seq(term).map((values) {
  return values[0] + values[2];
prod.set(prim.seq(char('*').trim()).seq(prod).map((values) {
  return values[0] * values[2];
prim.set(char('(').trim().seq(term).seq(char(')'.trim())).map((values) {
  return values[1];

To make sure that our parser consumes all input we wrap it with the end() parser into the start production:

final parser = term.end();

That's it, now we can test our parser and evaluator:

parser.parse('1 + 2 * 3');     // 7
parser.parse('(1 + 2) * 3');   // 9

Using the Expression Builder #

Writing such expression parsers is pretty common and can be quite tricky to get right. To simplify things, PetitParser comes with a builder that can help you to define such grammars easily. It supports the definition of operator precedence; and prefix, postfix, left- and right-associative operators.

The following code creates the empty expression builder:

final builder = ExpressionBuilder();

Then we define the operator-groups in descending precedence. The highest precedence are the literal numbers themselves. This time we accept floating point numbers, not just integers. In the same group we add support for parenthesis:
      .map((a) => num.tryParse(a)))
  ..wrapper(char('(').trim(), char(')').trim(), (l, a, r) => a);

Then come the normal arithmetic operators. Note, that the action blocks receive both, the terms and the parsed operator in the order they appear in the parsed input:

// negation is a prefix operator
  ..prefix(char('-').trim(), (op, a) => -a);

// power is right-associative
  ..right(char('^').trim(), (a, op, b) => math.pow(a, b));

// multiplication and addition are left-associative
  ..left(char('*').trim(), (a, op, b) => a * b)
  ..left(char('/').trim(), (a, op, b) => a / b);
  ..left(char('+').trim(), (a, op, b) => a + b)
  ..left(char('-').trim(), (a, op, b) => a - b);

Finally we can build the parser:

final parser =;

After executing the above code we get an efficient parser that correctly evaluates expressions like:

parser.parse('-8');      // -8
parser.parse('1+2*3');   // 7
parser.parse('1*2+3');   // 5
parser.parse('8/4/2');   // 1
parser.parse('2^2^3');   // 256

Misc #

Examples #

The package comes with a large collection of example grammars and language experiments ready to explore:

  • example/lib/dart contains an experimental Dart grammar.
  • example/lib/json contains a complete JSON grammar and parser.
  • example/lib/lisp contains a complete LISP grammar, parser and evaluator.
  • example/lib/prolog contains a basic Prolog grammar, parser and evaluator.
  • example/lib/smalltalk contains a complete Smalltalk grammar.

Furthermore, there are numerous open source projects using PetitParser:

  • badger is an experimental programming language.
  • expression_language is a library for parsing and evaluating expressions.
  • intl_translation provides internationalization and localization support to Dart.
  • pem encodes and decodes textual cryptographic keys.
  • powerconfig is a power config implementation.
  • query implements search queries with support for boolean groups, field scopes, ranges, etc.
  • rythm is a rich featured, high performance template engine.
  • xml is a lightweight library for parsing, traversing, and querying XML documents.

History #

PetitParser was originally implemented in Smalltalk. Later on, as a mean to learn these languages, I reimplemented PetitParser in Java and Dart. The implementations are very similar in their API and the supported features. If possible, the implementations adopt best practises of the target language.

Implementations #

License #

The MIT License, see LICENSE.

Changelog #

3.0.0 #

  • Dart 2.7 compatibility and requirement (extension methods).
  • New features:
    • String.toParser() enables creating efficient string and character parsers more easily.
    • Iterable.toChoiceParser() and Iterable.toSequenceParser() enables creating parsers from collections more easily.
    • Parser.callCC(Function) enables capturing a parse continuation more easily.
  • Restructure the internal code to be more modular:
    • The Parser class now only defines a few core methods, everything else is an extension method.
    • As long as you continue to import package:petitparser/petitparser.dart none of the changes should affect existing code.
    • Parser implementations have been moved to package:petitparser/parser.dart.
    • Helpers to parse and extract data has been moved to package:petitparser/matcher.dart.
    • The expression builder has been moved to package:petitparser/expression.dart.
    • The grammar builder has been moved topackage:petitparser/definition.dart.
  • Breaking changes:
    • Parser is no longer a Pattern, but can be converted to one with toPattern.
    • anyIn has been removed in favor of the already existing and equivalent anyOf parser.
    • pick and permute are defined on Parser<List>, thus they won't be available on the more generic Parser<dynamic> any longer. Prefix the operators with a castList operator.

2.5.0 #

  • Made ParserError a FormatException to follow typical Dart exception style.

2.4.0 #

  • Dart 2.4 compatibility and requirement.
  • More tight typing, more strict linter rules.
  • Documentation improvements.

2.3.0 #

  • Dart 2.3 compatibility and requirement.
  • The expression builder supports building expression with parens.
  • Improved the documentation on greedy and lazy parsers.
  • Add a prolog parser and interpreter example.
  • Numerous optimizations and improvements.

2.2.0 #

  • Dart 2.2 compatibility and requirement.
  • Parser implements the Pattern interface.
  • Add an example of the expression builder to the tutorial.
  • Introduce a fast-parse mode that avoids unnecessary memory allocations during parsing.

2.1.0 #

  • Rename ParserError to ParserException, and make it an Exception.
  • Simplify the EndOfInputParser and the ListParser.
  • Add a PositionParser that produces the current input position.
  • Constructor assertions across the stack.

2.0.0 #

  • Make parsers fully typed, where it makes sense.
    • In most cases this should have no effect on existing code, but sometimes can point out actual bugs.
    • In rare cases, it might be necessary to insert cast<R> or castList<R> at the appropriate places.
  • Move examples into their own example package.

1.8.0 #

  • Drop Dart 1.0 compatibility.

1.7.6 #

  • More Dart 2 strong mode fixes.

1.7.5 #

  • Dart 2.0 strong mode compatibility.
  • Removed deprecated code, and empty beta package.
  • Reformatted all code using dartfmt.

1.7.0 #

  • Dart 2.0 compatibility.
  • Fixed numerous analyzer warnings.
  • Generate better default error messages.
  • Moved example grammars to examples.

1.6.1 #

  • Fix bug with duplicated package name.
  • Update documentation.

1.6.0 #

  • Migrate to micro libraries.
  • Move Smalltalk, Json, Dart and Lisp grammars to examples.

1.5.5 #

  • Strict typing fixes.

1.5.4 #

  • Fix analyzer warnings.
  • Fix package dependencies.

1.5.3 #

  • Dev compiler support.

1.5.2 #

  • Enable strong mode.

1.5.1 #

  • Improve the Dart parser and add more tests.

1.5.0 #

  • Update documentation to match the style guide.
  • Change library names.
  • Add optimizations and tests for the Dart language grammar.
  • Improve comments.
  • Better error-handling and primitives for Lisp command line app.
  • Fix unicode parsing in the JSON parser.
  • Add browser back to dev_dependencies.

1.4.3 #

  • Restore the CompositeParser class.
  • Add more references to open source projects using PetitParser.

1.4.2 #

  • Integrate the tutorial into the README.
  • Improve formatting of README code blocks.

1.4.1 #

  • Improve test coverage.
  • Bump minimum SDK to 1.8.0.
  • Remove deprecated CompositeParser class.

1.4.0 #

  • Migrate from unittest to test.
  • Setup Travis.
  • Allow for const GrammarDefinitions.
  • Fix typo in docs.
  • Clean up the JSON grammar.
  • Format the benchmarks.

1.3.7 #

  • Cleanup dependencies:
    • browser is now >=0.10.0 <0.11.0.
    • unittest is now >=0.11.0 <0.12.0.
    • Remove explicit dependency on matcher.
  • Make the JSON parser twice as fast.
  • Reformat tests.

1.3.6 #

  • Add a benchmark for JSON native vs PetitParser.

1.3.5 #

  • Change hasEqualProperties to gracefully handle parsers of inconsistent types.

1.3.4 #

  • Format source code.
  • Add missing documentation.

1.3.3 #

  • Performance optimizations


PetitParser Examples #

For more complicated examples see the official GitHub repository.

This directory contains the command-line calculator that is described in the introductory tutorial:

dart petitparser/example/calc.dart "1 + 2 * 3"

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:

  petitparser: ^3.0.1

2. Install it

You can install packages from the command line:

with pub:

$ pub get

with Flutter:

$ flutter pub get

Alternatively, your editor might support pub get or flutter pub get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:

import 'package:petitparser/petitparser.dart';
Describes how popular the package is relative to other packages. [more]
Code health derived from static analysis. [more]
Reflects how tidy and up-to-date the package is. [more]
Weighted score of the above. [more]
Learn more about scoring.

We analyzed this package on Feb 29, 2020, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.7.1
  • pana: 0.13.5


Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.7.0 <3.0.0
meta ^1.1.0 1.1.8
Dev dependencies
test ^1.9.0