CharacterTextSplitter class
Implementation of TextSplitter that looks at characters.
- How the text is split: by single character (by default "\n\n").
- How the chunk size is measured: by number of characters.
- Inheritance
-
- Object
- Runnable<
List< Document> , BaseLangChainOptions, List<Document> > - BaseDocumentTransformer
- TextSplitter
- CharacterTextSplitter
- Available Extensions
Constructors
- CharacterTextSplitter({String separator = '\n\n', int chunkSize = 4000, int chunkOverlap = 200, int lengthFunction(String) = defaultLengthFunction, bool keepSeparator = false, bool addStartIndex = false})
-
Implementation of TextSplitter that looks at characters.
const
Properties
- addStartIndex → bool
-
If
true
, includes chunk'sstart_index
in metadata.finalinherited - chunkOverlap → int
-
Overlap in characters between chunks.
finalinherited
- chunkSize → int
-
Maximum size of chunks to return.
finalinherited
- defaultOptions → BaseLangChainOptions
-
The default options to use when invoking the Runnable.
finalinherited
- hashCode → int
-
The hash code for this object.
no setterinherited
- keepSeparator → bool
-
Whether to keep the separator in the chunks.
finalinherited
- lengthFunction → int Function(String)
-
Function that measures the length of given chunks.
finalinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- separator → String
-
The separator that is used to split the text.
final
Methods
-
batch(
List< List< inputs, {List<Document> >BaseLangChainOptions> ? options}) → Future<List< List< >Document> > -
Batches the invocation of the Runnable on the given
inputs
.inherited -
bind(
BaseLangChainOptions options) → RunnableBinding< List< Document> , BaseLangChainOptions, List<Document> > -
Binds the Runnable to the given
options
.inherited -
close(
) → void -
Cleans up any resources associated with it the Runnable.
inherited
-
createDocuments(
List< String> texts, {List<String> ? ids, List<Map< ? metadatas}) → List<String, dynamic> >Document> -
Creates documents from a list of texts.
inherited
-
getCompatibleOptions(
RunnableOptions? options) → BaseLangChainOptions? -
Returns the given
options
if they are compatible with the Runnable, otherwise returnsnull
.inherited -
invoke(
List< Document> input, {BaseLangChainOptions? options}) → Future<List< Document> > -
Transform a list of documents.
inherited
-
mergeSplits(
List< String> splits, String separator) → List<String> -
Merges smaller pieces into medium size chunks to send to the LLM.
inherited
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
pipe<
NewRunOutput extends Object?, NewCallOptions extends RunnableOptions> (Runnable< List< next) → RunnableSequence<Document> , NewCallOptions, NewRunOutput>List< Document> , NewRunOutput> -
Pipes the output of this Runnable into another Runnable using a
RunnableSequence.
inherited
-
splitDocuments(
List< Document> documents) → List<Document> -
Splits documents.
inherited
-
splitText(
String text) → List< String> -
Split text into multiple components.
override
-
stream(
List< Document> input, {BaseLangChainOptions? options}) → Stream<List< Document> > -
Streams the output of invoking the Runnable on the given
input
.inherited -
streamFromInputStream(
Stream< List< inputStream, {BaseLangChainOptions? options}) → Stream<Document> >List< Document> > -
Streams the output of invoking the Runnable on the given
inputStream
.inherited -
toString(
) → String -
A string representation of this object.
inherited
-
transformDocuments(
List< Document> documents) → Future<List< Document> > -
Transform a list of documents.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited