langchain_tiktoken 1.0.1 
langchain_tiktoken: ^1.0.1 copied to clipboard
dragonx.cloudA BPE tokeniser for use with OpenAI's models. It exposes APIs used for processing text using tokens.
We analyzed this package 10 days ago, and awarded it 140 pub points (of a possible 160):
10/10 points: Provide a valid pubspec.yaml
Issue tracker URL doesn't exist.
At the time of the analysis https://github.com/orenagiv/langchain_tiktoken/issues?q=is%3Aissue+is%3Aopen+ was unreachable. Make sure that the website is reachable via HEAD requests.
5/5 points: Provide a valid README.md
5/5 points: Provide a valid CHANGELOG.md
10/10 points: Use an OSI-approved license
Detected license: MIT.
10/10 points: 20% or more of the public API has dartdoc comments
20 out of 28 API elements (71.4 %) have documentation comments.
Some symbols that are missing documentation: tiktoken.SpecialTokensSet.SpecialTokensSet.all, tiktoken.SpecialTokensSet.SpecialTokensSet.custom, tiktoken.SpecialTokensSet.SpecialTokensSet.empty, tiktoken.SpecialTokensSet.isAll, tiktoken.SpecialTokensSet.set.
10/10 points: Package has an example
20/20 points: Supports 6 of 6 possible platforms (iOS, Android, Web, Windows, macOS, Linux)
-
✓ Android
-
✓ iOS
-
✓ Windows
-
✓ Linux
-
✓ macOS
-
✓ Web
0/0 points: WASM compatibility
This package is compatible with runtime wasm, and will be rewarded additional points in a future version of the scoring model.
See https://dart.dev/web/wasm for details.
30/50 points: code has no errors, warnings, lints, or formatting issues
Found 5 issues. Showing the first 2:
WARNING: A value for optional parameter 'explicitNVocab' isn't ever given.
lib/src/core_bpe_constructor.dart:40:10
╷
40 │ this.explicitNVocab,
│ ^^^^^^^^^^^^^^
╵
To reproduce make sure you are using the lints_core and run dart analyze lib/src/core_bpe_constructor.dart
INFO: Dangling library doc comment.
lib/src/ranks/cl100k_base.tiktoken.dart:1:1
╷
1 │ ┌ /// Placeholder file
2 │ │ ///
3 │ │ /// Use `download_ranks.dart` script under the `\tools` directory
4 │ └ /// to download required files.
╵
To reproduce make sure you are using the lints_core and run dart analyze lib/src/ranks/cl100k_base.tiktoken.dart
10/10 points: All of the package dependencies are supported in the latest version
No dependencies.
To reproduce run dart pub outdated --no-dev-dependencies --up-to-date --no-dependency-overrides.
10/10 points: Package supports latest stable Dart and Flutter SDKs
20/20 points: Compatible with dependency constraint lower bounds
pub downgrade does not expose any static analysis error.
Analyzed with Pana 0.22.17, Dart 3.6.0.
Check the analysis log for details.
Metadata
Publisher
Weekly Downloads
Metadata
A BPE tokeniser for use with OpenAI's models. It exposes APIs used for processing text using tokens.
Repository (GitHub)
View/report issues
Documentation
License
MIT (license)
