post_process method

PostProcessedOutput post_process(
  1. List<String>? tokens, [
  2. List<String>? tokens2,
  3. bool? add_special_tokens
])

Method to be implemented in subclass to apply post-processing on the given tokens.

@param {Array} tokens The input tokens to be post-processed. @param {...*} args Additional arguments required by the post-processing logic. @returns {PostProcessedOutput} The post-processed tokens. @throws {Error} If the method is not implemented in subclass.

Implementation

PostProcessedOutput post_process(List<String>? tokens, [List<String>? tokens2, bool? add_special_tokens]) {
  throw UnimplementedError('post_process should be implemented in subclass.');
}