tokenize method

List<Token> tokenize()

Lexes the whole input (without the final EOF token).

Implementation

List<Token> tokenize() {
  final tokens = <Token>[];
  while (true) {
    final t = next();
    if (t.isEof) return tokens;
    tokens.add(t);
  }
}