Lexer class final

Splits TeX source into Tokens.

Characters are grapheme clusters (UAX #29), so a base letter and its combining marks — Latin accents, Arabic harakat, Indic vowel signs — stay one token. Comments (% to end of line) are skipped. Whitespace runs collapse into one TokenKind.space token; spaces after a control word are dropped, as in TeX.

Constructors

Lexer(String input, {int spanOffset = 0, bool paragraphs = false, bool undoubleBackslashes = false})
Creates a lexer over input. spanOffset is added to every span, for lexing macro bodies that do not come from the user's source.

Properties

atEnd → bool
Whether all input has been consumed.
no setter
hashCode → int
The hash code for this object.
no setterinherited
input → String
The source being lexed.
final
paragraphs → bool
Whether an empty line is a \par token.
final
position → int
Current offset in input.
no setter
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
spanOffset → int
Added to every span.
final
undoubleBackslashes → bool
Whether doubled backslashes are read as single ones.
final

Methods

next() → Token
Returns the next token.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
tokenize() → List<Token>
Lexes the whole input (without the final EOF token).
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited