tokenize method

List<Token> tokenize(
  1. String source
)

Tokenizes the source into a list list Tokens. Each Token contains information about where it appears in the source and a TokenType.

Note that the list might be tokens that should be Token.invisibleToParser, if you're passing them to a Parser directly, you need to filter them. When using the methods in this class, this will be taken care of automatically.

Implementation

List<Token> tokenize(String source) {
  final scanner =
      Scanner(source, scanDriftTokens: options.useDriftExtensions);
  final tokens = scanner.scanTokens();

  return tokens;
}