llama_detokenize method

int llama_detokenize(
  1. Pointer<llama_model> model,
  2. Pointer<llama_token> tokens,
  3. int n_tokens,
  4. Pointer<Char> text,
  5. int text_len_max,
  6. bool remove_special,
  7. bool unparse_special,
)

@details Convert the provided tokens into text (inverse of llama_tokenize()). @param text The char pointer must be large enough to hold the resulting text. @return Returns the number of chars/bytes on success, no more than text_len_max. @return Returns a negative number on failure - the number of chars/bytes that would have been returned. @param remove_special Allow to remove BOS and EOS tokens if model is configured to do so. @param unparse_special If true, special tokens are rendered in the output.

Implementation

int llama_detokenize(
  ffi.Pointer<llama_model> model,
  ffi.Pointer<llama_token> tokens,
  int n_tokens,
  ffi.Pointer<ffi.Char> text,
  int text_len_max,
  bool remove_special,
  bool unparse_special,
) {
  return _llama_detokenize(
    model,
    tokens,
    n_tokens,
    text,
    text_len_max,
    remove_special,
    unparse_special,
  );
}