tokenizeLine method

LineTokens tokenizeLine(String line, [ StateStack? state ])

Tokenizes one line, which must not contain a line break. Pass the previous line's LineTokens.endState as state, or null for the first line of a document.

Throws ArgumentError if state came from a different Grammar.

Implementation

LineTokens tokenizeLine(String line, [StateStack? state]) {
  if (state != null && !identical(state._grammar, this)) {
    throw ArgumentError.value(
      state,
      'state',
      'Produced by a different Grammar. States are only valid for the '
          'grammar that returned them',
    );
  }
  final isFirstLine = state == null;
  final start = state ?? _initialState;
  if (maxLineLength > 0 && line.length > maxLineLength) {
    return LineTokens([
      Token(0, line.length, start._contentNameScopes),
    ], start);
  }
  final text = '$line\n';
  final tokens = _LineTokenBuilder();
  final end = _tokenizeString(text, isFirstLine, 0, start, tokens, true);
  return LineTokens(tokens.finish(line.length), end._reset());
}