tiktoken 1.0.3 copy "tiktoken: ^1.0.3" to clipboard
tiktoken: ^1.0.3 copied to clipboard

Tiktoken is a BPE tokeniser for use with OpenAI's models. It exposes APIs used for processing text using tokens.

Use this package as a library

Depend on it

Run this command:

With Dart:

 $ dart pub add tiktoken
copied to clipboard

With Flutter:

 $ flutter pub add tiktoken
copied to clipboard

This will add a line like this to your package's pubspec.yaml (and run an implicit dart pub get):

dependencies:
  tiktoken: ^1.0.3
copied to clipboard

Alternatively, your editor might support dart pub get or flutter pub get. Check the docs for your editor to learn more.

Import it

Now in your Dart code, you can use:

import 'package:tiktoken/tiktoken.dart';
copied to clipboard
22
likes
160
points
1.3k
downloads

Publisher

verified publishermarinvitlov.com

Weekly Downloads

2024.09.13 - 2025.03.28

Tiktoken is a BPE tokeniser for use with OpenAI's models. It exposes APIs used for processing text using tokens.

Homepage
Repository (GitHub)
View/report issues

Documentation

API reference

License

MIT (license)

More

Packages that depend on tiktoken