tokenizer 0.0.1
tokenizer: ^0.0.1 copied to clipboard

Converts a stream of strings into stream of tokens splitted by delimiters

tokenizer #

lesnitsky.dev GitHub stars Twitter Follow

Converts a stream of strings into stream of tokens splitted by delimiters

Installation #

pubspec.yaml:

dependencies:
  tokenizer: ^0.0.1

or

pub global activate tokenizer

Example #

import 'dart:async';

import 'package:tokenizer/tokenizer.dart';

void main(List<String> args) async {
  final string = 'Hello, world';
  final tokenizer = Tokenizer({','});
  final c = StreamController<String>();

  c.add(string);
  c.close();

  final tokens = await c.stream.transform(tokenizer.streamTransformer).toList();
  print(tokens); // [Token("Hello"), Token(","), Token(" "), Token("world")]
}

License #

MIT

2
likes
100
pub points
65%
popularity

Publisher

lesnitsky.dev

Converts a stream of strings into stream of tokens splitted by delimiters

Repository (GitHub)
View/report issues

Documentation

API reference

License

MIT (LICENSE)

Dependencies

flutter, fs_extensions, guard, l_cli, path

More

Packages that depend on tokenizer