zstd_neo 0.1.0

Pure Dart implementation of Zstandard (zstd): a complete decoder and a streaming encoder, running on the Dart VM, JavaScript and WebAssembly.

zstd_neo

A from-scratch, pure Dart implementation of the Zstandard compression format.

• Runs on the Dart VM, dart2js and dart2wasm — no FFI, no native libraries.
• Complete Zstandard decoder, including dictionaries, skippable frames, concatenated frames, content checksums and streaming.
• Streaming encoder with Converter / startChunkedConversion, plus a one-shot fast path.

Status: v0.1.0. The decoder targets the full format. The encoder implements levels 1–3 with fast/double-fast match finding, raw/RLE literals and the predefined FSE sequence tables; Huffman literal coding, custom FSE tables, repeat modes and levels 4–19 are on the roadmap. Expect the encoder to trade compression ratio for a small, auditable code base until then.

Usage

import 'dart:convert';

import 'package:zstd_neo/zstd_neo.dart';

void main() {
  final input = utf8.encode('hello hello hello hello');

  final codec = ZstdCodec(level: 3);
  final compressed = codec.encode(input);
  final restored = codec.decode(compressed);

  print(utf8.decode(restored)); // hello hello hello hello
}

Streaming

import 'dart:convert';

import 'package:zstd_neo/zstd_neo.dart';

void main() {
  final output = <int>[];
  final sink = ZstdDecoder().startChunkedConversion(
    ByteConversionSink.withCallback(output.addAll),
  );

  sink.add(compressedPart1);
  sink.add(compressedPart2);
  sink.close();

  print(output.length);
}

Bytes fast path

blit's Bytes can be passed without an intermediate copy:

final compressed = codec.compressBytes(inputBytes); // Bytes -> Bytes
final restored = codec.decompressBytes(compressed);

Dictionaries

final decoder = ZstdDecoder(dictionary: dictionaryBytes);
final data = decoder.decompress(frameWithDictionary);

Raw-content dictionaries and formatted zstd dictionaries (magic 0xEC30A437) are both supported for decoding.

Format support

Feature	Decode	Encode
Raw / RLE / compressed blocks	✓	✓
Raw / RLE literals	✓	✓
Huffman literals, 1 & 4 streams	✓	—
Treeless literals / repeat Huffman	✓	—
Predefined / RLE / FSE / repeat sequence tables	✓	predefined only
Repeat offsets	✓	✓
Content checksum (XXH64)	✓	optional
Frame content size / single segment	✓	one-shot
Skippable frames, concatenated frames	✓	—
Structured & raw dictionaries	✓	—
Streaming	✓	✓

The decoder rejects windows larger than maxWindowSize (64 MiB by default) and verifies checksums when present.

Performance

The card is rendered from benchmark_test timings (Dart AOT, JS and Wasm) against the Rust zstd crate on 256 KiB Silesia slices. It is generated and committed automatically by CI on the default branch; a missing series is omitted rather than faked.

The decoder currently reaches roughly 130 MiB/s on 64 KiB of English text (compressed, one-shot, JIT), 241 MiB/s on level-19 text, and 684 MiB/s on the 1 MiB RLE golden vector. The v0.1.0 encoder trades ratio for simplicity; the remaining gap is mostly Huffman literal coding (v0.2).

Development

dart pub get
dart analyze
dart test        # Windows / Linux / macOS VM tests

Web tests (CI): dart test -p chrome and dart test -p chrome -c dart2wasm.

Test fixtures are generated by the dev-only Rust harness:

cd benchmark/rust_ref
cargo run --release -- gen-fixtures --out ../../test/fixtures

Rust is only used for tests, fixtures and benchmarks; the published library is pure Dart.

License

MIT — see LICENSE. Test vectors derived from the reference zstd project and the Silesia corpus are covered by NOTICE.