tokenizer 0.0.1

  • Readme
  • Changelog
  • Example
  • Installing
  • 71

tokenizer #

lesnitsky.dev GitHub stars Twitter Follow

Converts a stream of strings into stream of tokens splitted by delimiters

Installation #

pubspec.yaml:

dependencies:
  tokenizer: ^0.0.1

or

pub global activate tokenizer

Example #

import 'dart:async';

import 'package:tokenizer/tokenizer.dart';

void main(List<String> args) async {
  final string = 'Hello, world';
  final tokenizer = Tokenizer({','});
  final c = StreamController<String>();

  c.add(string);
  c.close();

  final tokens = await c.stream.transform(tokenizer.streamTransformer).toList();
  print(tokens); // [Token("Hello"), Token(","), Token(" "), Token("world")]
}

License #

MIT

[0.0.1] #

  • Initial release

example/lib/main.dart

import 'dart:async';

import 'package:tokenizer/tokenizer.dart';

void main(List<String> args) async {
  final string = 'Hello, world';
  final tokenizer = Tokenizer({','});
  final c = StreamController<String>();

  c.add(string);
  c.close();

  final tokens = await c.stream.transform(tokenizer.streamTransformer).toList();
  print(tokens); // [Token("Hello"), Token(","), Token(" "), Token("world")]
}

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:


dependencies:
  tokenizer: ^0.0.1

2. Install it

You can install packages from the command line:

with Flutter:


$ flutter pub get

Alternatively, your editor might support flutter pub get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:


import 'package:tokenizer/tokenizer.dart';
  
Popularity:
Describes how popular the package is relative to other packages. [more]
46
Health:
Code health derived from static analysis. [more]
100
Maintenance:
Reflects how tidy and up-to-date the package is. [more]
90
Overall:
Weighted score of the above. [more]
71
Learn more about scoring.

We analyzed this package on Jul 2, 2020, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.8.4
  • pana: 0.13.13
  • Flutter: 1.17.5

Analysis suggestions

Package not compatible with SDK dart

because of import path [tokenizer] that is in a package requiring null.

Maintenance suggestions

Package is pre-v0.1 release. (-10 points)

While nothing is inherently wrong with versions of 0.0.*, it might mean that the author is still experimenting with the general direction of the API.

Dependencies

Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.7.0 <3.0.0
flutter 0.0.0
fs_extensions ^0.0.6 0.0.10
guard ^1.0.1 1.0.4
l_cli ^1.0.2 1.0.2
path ^1.6.4 1.7.0
Transitive dependencies
collection 1.14.12 1.14.13
meta 1.1.8
sky_engine 0.0.99
typed_data 1.1.6 1.2.0
vector_math 2.0.8
Dev dependencies
flutter_test