tokenize method

List<List<Token>> tokenize(
  1. String code
)

Tokenizes a whole document, split with splitLines.

Implementation

List<List<Token>> tokenize(String code) {
  final result = <List<Token>>[];
  StateStack? state;
  for (final line in splitLines(code)) {
    final lineTokens = tokenizeLine(line, state);
    result.add(lineTokens.tokens);
    state = lineTokens.endState;
  }
  return result;
}