CommonTokenStream class
This class extends BufferedTokenStream with functionality to filter token streams to tokens on a particular channel (tokens where {@link Token#getChannel} returns a particular value).
This token stream provides access to all tokens by index or when calling methods like {@link #getText}. The channel filtering is only used for code accessing tokens via the lookahead methods {@link #LA}, {@link #LT}, and {@link #LB}.
By default, tokens are placed on the default channel ({@link Token#DEFAULT_CHANNEL}), but may be reassigned by using the {@code ->channel(HIDDEN)} lexer command, or by using an embedded action to call {@link Lexer#setChannel}.
Note: lexer rules which use the {@code ->skip} lexer command or call {@link Lexer#skip} do not produce tokens at all, so input text matched by such a rule will not be available as part of the token stream, regardless of channel.
we- Inheritance
-
- Object
- BufferedTokenStream
- CommonTokenStream
Constructors
- CommonTokenStream(TokenSource tokenSource, [int channel = Token.DEFAULT_CHANNEL])
-
Constructs a new CommonTokenStream using the specified token
source and filtering tokens to the specified channel. Only tokens whose
{@link Token#getChannel} matches
channel
or have the {@link Token#getType} equal to {@link Token#EOF} will be returned by the token stream lookahead methods.
Properties
- channel ↔ int
-
Specifies the channel to use for filtering tokens.
getter/setter pair
- fetchedEOF ↔ bool
-
Indicates whether the Token.EOF token has been fetched from
tokenSource and added to tokens. This field improves
performance for the following cases:
getter/setter pairinherited
- hashCode → int
-
The hash code for this object.
no setterinherited
- index → int
-
Return the index into the stream of the input symbol referred to by
{@code LA(1)}.
no setterinherited
- numberOfOnChannelTokens → int
-
Count EOF just once.
no setter
- p ↔ int
-
The index into tokens of the current token (next token to consume).
p should be LT(1).
getter/setter pairinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- size → int
-
Returns the total number of symbols in the stream, including a single EOF
symbol.
no setterinherited
- sourceName → String
-
Gets the name of the underlying symbol source. This method returns a
non-null, non-empty string. If such a name is not known, this method
returns {@link #UNKNOWN_SOURCE_NAME}.
no setterinherited
- text → String
-
no setterinherited
-
tokens
↔ List<
Token> -
A collection of all tokens fetched from the token source. The list is
considered a complete view of the input once {@link #fetchedEOF} is set
to
true
.getter/setter pairinherited - tokenSource ↔ TokenSource
-
Gets the underlying TokenSource which provides tokens for this
stream.
getter/setter pairinherited
Methods
-
adjustSeekIndex(
int i) → int -
Allowed derived classes to modify the behavior of operations which change
the current stream position by adjusting the target token index of a seek
operation. The default implementation simply returns
i
. If an exception is thrown in this method, the current stream index should not be changed.override -
consume(
) → void -
Consumes the current symbol in the stream. This method has the following
effects:
inherited
-
fetch(
int n) → int -
Add
n
elements to buffer.inherited -
fill(
) → void -
Get all tokens from lexer until EOF */
inherited
-
filterForChannel(
int from, int to, int channel) → List< Token> ? -
inherited
-
get(
int i) → Token -
Gets the Token at the specified
index
in the stream. When the preconditions of this method are met, the return value is non-null.inherited -
getHiddenTokensToLeft(
int tokenIndex, [int channel = -1]) → List< Token> ? -
Collect all tokens on specified channel to the left of
the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
If channel is -1, find any non default channel token.
inherited
-
getHiddenTokensToRight(
int tokenIndex, [int channel = -1]) → List< Token> ? -
Collect all tokens on specified channel to the right of
the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or
EOF. If channel is -1, find any non default channel token.
inherited
-
getRange(
int start, [int? stop]) → List< Token> ? -
Get all tokens from start..stop inclusively */
inherited
-
getText(
[Interval? interval]) → String -
Return the text of all tokens within the specified
interval
. This method behaves like the following code (including potential exceptions for violating preconditions of {@link #get}, but may be optimized by the specific implementation.inherited -
getTextFromCtx(
RuleContext ctx) → String -
Return the text of all tokens in the source interval of the specified
context. This method behaves like the following code, including potential
exceptions from the call to {@link #getText(Interval)}, but may be
optimized by the specific implementation.
inherited
-
getTextRange(
Token? start, Token? stop) → String -
Return the text of all tokens in this stream between
start
andstop
(inclusive).inherited -
getTokens(
[int? start, int? stop, Set< int> ? types]) → List<Token> ? -
Given a start and stop index, return a List of all tokens in
the token type BitSet. Return null if no tokens were found. This
method looks at both on and off channel tokens.
inherited
-
LA(
int i) → int? -
Gets the value of the symbol at offset
i
from the current position. When {@code i==1}, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When {@code i==-1}, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with {@code i==0}, but the specific behavior is unspecified because this method is frequently called from performance-critical code.inherited -
lazyInit(
) → void -
inherited
-
LB(
int k) → Token? -
override
-
LT(
int k) → Token? -
Get the Token instance associated with the value returned by
LA. This method has the same pre- and post-conditions as
IntStream.LA. In addition, when the preconditions of this method
are met, the return value is non-null and the value of
LT(k).getType()==LA(k)
.override -
mark(
) → int -
A mark provides a guarantee that {@link #seek seek()} operations will be
valid over a "marked range" extending from the index where {@code mark()}
was called to the current {@link #index index()}. This allows the use of
streaming input sources by specifying the minimum buffering requirements
to support arbitrary lookahead during prediction.
inherited
-
nextTokenOnChannel(
int i, int channel) → int -
Given a starting index, return the index of the next token on channel.
Return
i
if {@code tokensi
} is on channel. Return the index of the EOF token if there are no tokens on channel betweeni
and EOF.inherited -
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
previousTokenOnChannel(
int i, int channel) → int -
Given a starting index, return the index of the previous token on
channel. Return
i
if {@code tokensi
} is on channel. Return -1 if there are no tokens on channel betweeni
and 0.inherited -
release(
int marker) → void -
This method releases a marked range created by a call to
{@link #mark mark()}. Calls to {@code release()} must appear in the
reverse order of the corresponding calls to {@code mark()}. If a mark is
released twice, or if marks are not released in reverse order of the
corresponding calls to {@code mark()}, the behavior is unspecified.
inherited
-
seek(
int index) → void -
Set the input cursor to the position indicated by
index
. If the specified index lies past the end of the stream, the operation behaves as thoughindex
was the index of the EOF symbol. After this method returns without throwing an exception, then at least one of the following will be true.inherited -
setup(
) → void -
inherited
-
sync(
int i) → bool -
Make sure index
i
in tokens has a token.inherited -
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited