BufferedTokenStream class

This implementation of TokenStream loads tokens from a TokenSource on-demand, and places the tokens in a buffer to provide access to any previous token by index.

This token stream ignores the value of {@link Token#getChannel}. If your parser requires the token stream filter tokens to only those on a particular channel, such as {@link Token#DEFAULT_CHANNEL} or {@link Token#HIDDEN_CHANNEL}, use a filtering token stream such a [CommonTokenStream].

Implemented types
Implementers

Constructors

BufferedTokenStream(TokenSource _tokenSource)

Properties

fetchedEOF ↔ bool
Indicates whether the Token.EOF token has been fetched from tokenSource and added to tokens. This field improves performance for the following cases:
getter/setter pair
hashCode → int
The hash code for this object.
no setterinherited
index → int
Return the index into the stream of the input symbol referred to by {@code LA(1)}.
no setteroverride
p ↔ int
The index into tokens of the current token (next token to consume). p should be LT(1).
getter/setter pair
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
size → int
Returns the total number of symbols in the stream, including a single EOF symbol.
no setteroverride
sourceName → String
Gets the name of the underlying symbol source. This method returns a non-null, non-empty string. If such a name is not known, this method returns {@link #UNKNOWN_SOURCE_NAME}.
no setteroverride
text → String
no setteroverride
tokens ↔ List<Token>
A collection of all tokens fetched from the token source. The list is considered a complete view of the input once {@link #fetchedEOF} is set to true.
getter/setter pair
tokenSource ↔ TokenSource
Gets the underlying TokenSource which provides tokens for this stream.
getter/setter pairoverride-getter

Methods

adjustSeekIndex(int i) → int
Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i. If an exception is thrown in this method, the current stream index should not be changed.
consume() → void
Consumes the current symbol in the stream. This method has the following effects:
override
fetch(int n) → int
Add n elements to buffer.
fill() → void
Get all tokens from lexer until EOF */
filterForChannel(int from, int to, int channel) → List<Token>?
get(int i) → Token
Gets the Token at the specified index in the stream. When the preconditions of this method are met, the return value is non-null.
override
getHiddenTokensToLeft(int tokenIndex, [int channel = -1]) → List<Token>?
Collect all tokens on specified channel to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL. If channel is -1, find any non default channel token.
getHiddenTokensToRight(int tokenIndex, [int channel = -1]) → List<Token>?
Collect all tokens on specified channel to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF. If channel is -1, find any non default channel token.
getRange(int start, [int? stop]) → List<Token>?
Get all tokens from start..stop inclusively */
getText([Interval? interval]) → String
Return the text of all tokens within the specified interval. This method behaves like the following code (including potential exceptions for violating preconditions of {@link #get}, but may be optimized by the specific implementation.
override
getTextFromCtx(RuleContext ctx) → String
Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to {@link #getText(Interval)}, but may be optimized by the specific implementation.
override
getTextRange(Token? start, Token? stop) → String
Return the text of all tokens in this stream between start and stop (inclusive).
override
getTokens([int? start, int? stop, Set<int>? types]) → List<Token>?
Given a start and stop index, return a List of all tokens in the token type BitSet. Return null if no tokens were found. This method looks at both on and off channel tokens.
LA(int i) → int?
Gets the value of the symbol at offset i from the current position. When {@code i==1}, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When {@code i==-1}, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with {@code i==0}, but the specific behavior is unspecified because this method is frequently called from performance-critical code.
override
lazyInit() → void
LB(int k) → Token?
LT(int k) → Token?
Get the Token instance associated with the value returned by LA. This method has the same pre- and post-conditions as IntStream.LA. In addition, when the preconditions of this method are met, the return value is non-null and the value of LT(k).getType()==LA(k).
override
mark() → int
A mark provides a guarantee that {@link #seek seek()} operations will be valid over a "marked range" extending from the index where {@code mark()} was called to the current {@link #index index()}. This allows the use of streaming input sources by specifying the minimum buffering requirements to support arbitrary lookahead during prediction.
override
nextTokenOnChannel(int i, int channel) → int
Given a starting index, return the index of the next token on channel. Return i if {@code tokensi} is on channel. Return the index of the EOF token if there are no tokens on channel between i and EOF.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
previousTokenOnChannel(int i, int channel) → int
Given a starting index, return the index of the previous token on channel. Return i if {@code tokensi} is on channel. Return -1 if there are no tokens on channel between i and 0.
release(int marker) → void
This method releases a marked range created by a call to {@link #mark mark()}. Calls to {@code release()} must appear in the reverse order of the corresponding calls to {@code mark()}. If a mark is released twice, or if marks are not released in reverse order of the corresponding calls to {@code mark()}, the behavior is unspecified.
override
seek(int index) → void
Set the input cursor to the position indicated by index. If the specified index lies past the end of the stream, the operation behaves as though index was the index of the EOF symbol. After this method returns without throwing an exception, then at least one of the following will be true.
override
setup() → void
sync(int i) → bool
Make sure index i in tokens has a token.
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited