tokenize method

@override List<CodeToken> tokenize(String code)

Tokenizes code into a flat list of tokens covering the full input.

Implementation

@override
List<CodeToken> tokenize(String code) => _tokenize(
  code,
  keywords: _keywords,
  lineComment: '#',
  blockComment: null,
);