flutter_tiktoken 0.0.1 copy "flutter_tiktoken: ^0.0.1" to clipboard
flutter_tiktoken: ^0.0.1 copied to clipboard

A flutter offline package for a fast BPE tokeniser for OpenAI models.

flutter_tiktoken #

flutter_tiktoken is a flutter offline package for a fast BPE tokeniser for OpenAI models.

flutter_tiktoken is a copy package in https://github.com/mvitlov/tiktoken and fixed the issue build spent many time (more than one hour) when build in release mode.

Open Ai Official Calcuate flutter_tiktoken
Snip20230621_3 Simulator Screenshot - iPhone 14 - 2023-06-21 at 18 43 10

Usage #

init flutter_tiktoken

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await TiktokenDataProcessCenter().initata();
  runApp(const MyApp());
}
copied to clipboard

calcuateTokens

static int _calcuateTokens(
      {required String text, String modelName = 'gpt-4'}) {
    final encoding = encodingForModel(modelName);
    final numTokens = encoding.encode(text).length;
    return numTokens;
  }
copied to clipboard
4
likes
150
points
122
downloads

Publisher

unverified uploader

Weekly Downloads

2024.07.08 - 2025.01.20

A flutter offline package for a fast BPE tokeniser for OpenAI models.

Repository (GitHub)

Documentation

API reference

License

MIT (license)

Dependencies

flutter

More

Packages that depend on flutter_tiktoken