CommonTokenStream class

This class extends BufferedTokenStream with functionality to filter token streams to tokens on a particular channel (tokens where {@link Token#getChannel} returns a particular value).

This token stream provides access to all tokens by index or when calling methods like {@link #getText}. The channel filtering is only used for code accessing tokens via the lookahead methods {@link #LA}, {@link #LT}, and {@link #LB}.

By default, tokens are placed on the default channel ({@link Token#DEFAULT_CHANNEL}), but may be reassigned by using the {@code ->channel(HIDDEN)} lexer command, or by using an embedded action to call {@link Lexer#setChannel}.

Note: lexer rules which use the {@code ->skip} lexer command or call {@link Lexer#skip} do not produce tokens at all, so input text matched by such a rule will not be available as part of the token stream, regardless of channel.

we
Inheritance

Constructors

CommonTokenStream(TokenSource tokenSource, [int channel = Token.DEFAULT_CHANNEL])
Constructs a new CommonTokenStream using the specified token source and filtering tokens to the specified channel. Only tokens whose {@link Token#getChannel} matches channel or have the {@link Token#getType} equal to {@link Token#EOF} will be returned by the token stream lookahead methods.

Properties

channel ↔ int
Specifies the channel to use for filtering tokens.
getter/setter pair
fetchedEOF ↔ bool
Indicates whether the Token.EOF token has been fetched from tokenSource and added to tokens. This field improves performance for the following cases:
getter/setter pairinherited
hashCode → int
The hash code for this object.
no setterinherited
index → int
Return the index into the stream of the input symbol referred to by {@code LA(1)}.
no setterinherited
numberOfOnChannelTokens → int
Count EOF just once.
no setter
p ↔ int
The index into tokens of the current token (next token to consume). p should be LT(1).
getter/setter pairinherited
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
size → int
Returns the total number of symbols in the stream, including a single EOF symbol.
no setterinherited
sourceName → String
Gets the name of the underlying symbol source. This method returns a non-null, non-empty string. If such a name is not known, this method returns {@link #UNKNOWN_SOURCE_NAME}.
no setterinherited
text → String
no setterinherited
tokens ↔ List<Token>
A collection of all tokens fetched from the token source. The list is considered a complete view of the input once {@link #fetchedEOF} is set to true.
getter/setter pairinherited
tokenSource ↔ TokenSource
Gets the underlying TokenSource which provides tokens for this stream.
getter/setter pairinherited

Methods

adjustSeekIndex(int i) → int
Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i. If an exception is thrown in this method, the current stream index should not be changed.
override
consume() → void
Consumes the current symbol in the stream. This method has the following effects:
inherited
fetch(int n) → int
Add n elements to buffer.
inherited
fill() → void
Get all tokens from lexer until EOF */
inherited
filterForChannel(int from, int to, int channel) → List<Token>?
inherited
get(int i) → Token
Gets the Token at the specified index in the stream. When the preconditions of this method are met, the return value is non-null.
inherited
getHiddenTokensToLeft(int tokenIndex, [int channel = -1]) → List<Token>?
Collect all tokens on specified channel to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL. If channel is -1, find any non default channel token.
inherited
getHiddenTokensToRight(int tokenIndex, [int channel = -1]) → List<Token>?
Collect all tokens on specified channel to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF. If channel is -1, find any non default channel token.
inherited
getRange(int start, [int? stop]) → List<Token>?
Get all tokens from start..stop inclusively */
inherited
getText([Interval? interval]) → String
Return the text of all tokens within the specified interval. This method behaves like the following code (including potential exceptions for violating preconditions of {@link #get}, but may be optimized by the specific implementation.
inherited
getTextFromCtx(RuleContext ctx) → String
Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to {@link #getText(Interval)}, but may be optimized by the specific implementation.
inherited
getTextRange(Token? start, Token? stop) → String
Return the text of all tokens in this stream between start and stop (inclusive).
inherited
getTokens([int? start, int? stop, Set<int>? types]) → List<Token>?
Given a start and stop index, return a List of all tokens in the token type BitSet. Return null if no tokens were found. This method looks at both on and off channel tokens.
inherited
LA(int i) → int?
Gets the value of the symbol at offset i from the current position. When {@code i==1}, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When {@code i==-1}, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with {@code i==0}, but the specific behavior is unspecified because this method is frequently called from performance-critical code.
inherited
lazyInit() → void
inherited
LB(int k) → Token?
override
LT(int k) → Token?
Get the Token instance associated with the value returned by LA. This method has the same pre- and post-conditions as IntStream.LA. In addition, when the preconditions of this method are met, the return value is non-null and the value of LT(k).getType()==LA(k).
override
mark() → int
A mark provides a guarantee that {@link #seek seek()} operations will be valid over a "marked range" extending from the index where {@code mark()} was called to the current {@link #index index()}. This allows the use of streaming input sources by specifying the minimum buffering requirements to support arbitrary lookahead during prediction.
inherited
nextTokenOnChannel(int i, int channel) → int
Given a starting index, return the index of the next token on channel. Return i if {@code tokensi} is on channel. Return the index of the EOF token if there are no tokens on channel between i and EOF.
inherited
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
previousTokenOnChannel(int i, int channel) → int
Given a starting index, return the index of the previous token on channel. Return i if {@code tokensi} is on channel. Return -1 if there are no tokens on channel between i and 0.
inherited
release(int marker) → void
This method releases a marked range created by a call to {@link #mark mark()}. Calls to {@code release()} must appear in the reverse order of the corresponding calls to {@code mark()}. If a mark is released twice, or if marks are not released in reverse order of the corresponding calls to {@code mark()}, the behavior is unspecified.
inherited
seek(int index) → void
Set the input cursor to the position indicated by index. If the specified index lies past the end of the stream, the operation behaves as though index was the index of the EOF symbol. After this method returns without throwing an exception, then at least one of the following will be true.
inherited
setup() → void
inherited
sync(int i) → bool
Make sure index i in tokens has a token.
inherited
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited