tokenizer 1.0.0

Converts a stream of strings into a stream of tokens

tokenizer

Converts a stream of strings into stream of tokens splitted by delimiters

Installation

dart pub add tokenizer

or

flutter pub add tokenizer

Example

import 'dart:async';

import 'package:tokenizer/tokenizer.dart';

void main(List<String> args) async {
  final string = 'Hello, world';
  final tokenizer = Tokenizer({',', ' '});
  final c = StreamController<String>();

  c.add(string);
  c.close();

  final tokens = await c.stream.transform(tokenizer.transformer).toList();
  print(tokens); // ["Hello", ",", " ", "world"]
}

License

MIT