tokenizer 1.0.0 copy "tokenizer: ^1.0.0" to clipboard
tokenizer: ^1.0.0 copied to clipboard

Converts a stream of strings into a stream of tokens

tokenizer #

GitHub stars Twitter Follow

Converts a stream of strings into stream of tokens splitted by delimiters

Installation #

dart pub add tokenizer

or

flutter pub add tokenizer

Example #

import 'dart:async';

import 'package:tokenizer/tokenizer.dart';

void main(List<String> args) async {
  final string = 'Hello, world';
  final tokenizer = Tokenizer({',', ' '});
  final c = StreamController<String>();

  c.add(string);
  c.close();

  final tokens = await c.stream.transform(tokenizer.transformer).toList();
  print(tokens); // ["Hello", ",", " ", "world"]
}

License #

MIT

4
likes
110
pub points
73%
popularity

Publisher

verified publisherlesnitsky.dev

Converts a stream of strings into a stream of tokens

Repository (GitHub)
View/report issues

Documentation

API reference

License

MIT (LICENSE)

More

Packages that depend on tokenizer