Token class

A token that was scanned from the input. Each token knows which tokens precede and follow it, acting as a link in a doubly linked list of tokens.

Clients may not extend, implement or mix-in this class.

Implemented types

Constructors

Token(TokenType type, int offset, [CommentToken preceedingComment])
Initialize a newly created token to have the given type and offset.
factory
Token.eof(int offset, [CommentToken precedingComments])
Initialize a newly created end-of-file token to have the given offset.
factory

Properties

beforeSynthetic Token
The token before this synthetic token, or null if this is not a synthetic ), ], }, or > token.
read / write
charCount int
The number of characters parsed by this token.
read-only
charEnd int
The character offset of the end of this token within the source text.
read-only
charOffset int
The character offset of the start of this token within the source text.
read-only
end int
Return the offset from the beginning of the file to the character after the last character of the syntactic entity.
read-only, override
endGroup Token
The token that corresponds to this token, or null if this token is not the first of a pair of matching tokens (such as parentheses).
read-only
hashCode int
The hash code for this object. [...]
read-only, inherited
isEof bool
Return true if this token represents an end of file.
read-only
isIdentifier bool
True if this token is an identifier. Some keywords allowed as identifiers, see implementation in KeywordToken.
read-only
isKeyword bool
True if this token is a keyword. Some keywords allowed as identifiers, see implementation in KeywordToken.
read-only
isKeywordOrIdentifier bool
True if this token is a keyword or an identifier.
read-only
isModifier bool
Return true if this token is a modifier such as abstract or const.
read-only
isOperator bool
Return true if this token represents an operator.
read-only
isSynthetic bool
Return true if this token is a synthetic token. A synthetic token is a token that was introduced by the parser in order to recover from an error in the code.
read-only
isTopLevelKeyword bool
Return true if this token is a keyword starting a top level declaration such as class, enum, import, etc.
read-only
isUserDefinableOperator bool
Return true if this token represents an operator that can be defined by users.
read-only
keyword Keyword
Return the keyword, if a keyword token, or null otherwise.
read-only
kind int
The kind enum of this token as determined by its type.
read-only
length int
Return the number of characters in the syntactic entity's source range.
read-only, override
lexeme String
Return the lexeme that represents this token. [...]
read-only
next Token
Return the next token in the token stream.
read / write
offset int
Return the offset from the beginning of the file to the first character in the syntactic entity.
read / write, override-getter
precedingComments → CommentToken
Return the first comment in the list of comments that precede this token, or null if there are no comments preceding this token. Additional comments can be reached by following the token stream using next until null is returned. [...]
read-only
previous Token
Return the previous token in the token stream.
read / write
runtimeType Type
A representation of the runtime type of the object.
read-only, inherited
stringValue String
For symbol and keyword tokens, returns the string value represented by this token. For StringTokens this method returns null. [...]
read-only
type TokenType
Return the type of the token.
read-only

Methods

copy() Token
Return a newly created token that is a copy of this tokens including any preceedingComment tokens, but that is not a part of any token stream.
copyComments(Token token) Token
Copy a linked list of comment tokens identical to the given comment tokens.
matchesAny(List<TokenType> types) bool
Return true if this token has any one of the given types.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a non-existent method or property is accessed. [...]
inherited
setNext(Token token) Token
Set the next token in the token stream to the given token. This has the side-effect of setting this token to be the previous token for the given token. Return the token that was passed in.
setNextWithoutSettingPrevious(Token token) Token
Set the next token in the token stream to the given token without changing which token is the previous token for the given token. Return the token that was passed in.
toString() String
Returns a textual representation of this token to be used for debugging purposes. The resulting string might contain information about the structure of the token, for example 'StringToken(foo)' for the identifier token 'foo'. [...]
override
value() Object
Return the value of this token. For keyword tokens, this is the keyword associated with the token, for other tokens it is the lexeme associated with the token.

Operators

operator ==(Object other) bool
The equality operator. [...]
inherited

Static Methods

lexicallyFirst(List<Token> tokens) Token
Compare the given tokens to find the token that appears first in the source being parsed. That is, return the left-most of all of the tokens. The list must be non-null, but the elements of the list are allowed to be null. Return the token with the smallest offset, or null if the list is empty or if all of the elements of the list are null.