transformer library
Transformations for messages or conversations.
Classes
- ConversationTransformer
- A StreamTransformer that operates on a List<Conversation>
- CsvInput
- Read from CSV
- CsvOutput
- Output message fields to CSV
- DsBuildOutput
- Output JSON data formatted for DsBuild
- Encoding
- Re-encodes a text using the specified codec. Ensures encoding compatibility.
- ExactReplace
- Replace matches with a substitution.
- FastChatInput
- Read from FastChat formatted JSON
- FastChatOutput
- Output JSON data formatted for FastChat
- FileConcatenate
- Concatenate lines into a single file
- FullMatch
- Drop messages that exactly matches the provided pattern.
- HtmlStrip
- Strip HTML
- Participants
- Drops conversations according to their participant count.
- RawOutput
- Raw message output
- RegexOutput
- Output messages matching a RegExp
- RegexReplace
- Replace regex pattern with provided substitution.
- RenameParticipants
- Rename participants according to the order of their appearance.
- StatsAddColMerge
- Merge CSV files and perform addition on the given columns for duplicate keys
- StatsCountOccurrences
- Count occurrences of patterns in the dataset. Reads patterns from csv in cache
- Trim
- Trim whitespace
Functions
-
defaultTransformers(
) → Map< String, ConversationTransformerBuilderFn> - Maps builder names to their builder functions.
Typedefs
- ConversationTransformerBuilderFn = ConversationTransformer Function(Map, ProgressBloc? progress, PackedDataCache? cache)
- A function that takes a ConversationTransformer.config Map and returns a ConversationTransformer