zstd_neo 0.1.1

Pure Dart implementation of Zstandard (zstd): a complete decoder and a streaming encoder, running on the Dart VM, JavaScript and WebAssembly.

zstd_neo

A from-scratch, pure Dart implementation of the Zstandard compression format.

• Runs on the Dart VM, dart2js and dart2wasm — no FFI, no native libraries.
• Complete Zstandard decoder, including dictionaries, skippable frames, concatenated frames, content checksums and streaming.
• Streaming encoder with Converter / startChunkedConversion, plus a one-shot fast path.

Status: v0.1.1 (production). The decoder covers the full Zstandard format. The encoder now covers levels 1–19 with Huffman literals (1/4 streams, treeless repeat), custom FSE sequence tables, repeat/RLE modes and raw/formatted dictionary support. Levels 10–19 use a deeper-search lazy2 parser rather than the reference btopt, so they produce valid frames and competitive ratios but may trail libzstd at the very highest levels.

Usage

import 'dart:convert';

import 'package:zstd_neo/zstd_neo.dart';

void main() {
  final input = utf8.encode('hello hello hello hello');

  final codec = ZstdCodec(level: 3);
  final compressed = codec.encode(input);
  final restored = codec.decode(compressed);

  print(utf8.decode(restored)); // hello hello hello hello
}

Streaming

import 'dart:convert';

import 'package:zstd_neo/zstd_neo.dart';

void main() {
  final output = <int>[];
  final sink = ZstdDecoder().startChunkedConversion(
    ByteConversionSink.withCallback(output.addAll),
  );

  sink.add(compressedPart1);
  sink.add(compressedPart2);
  sink.close();

  print(output.length);
}

Bytes fast path

blit's Bytes can be passed without an intermediate copy:

final compressed = codec.compressBytes(inputBytes); // Bytes -> Bytes
final restored = codec.decompressBytes(compressed);

Dictionaries

final codec = ZstdCodec(level: 9, dictionary: dictionaryBytes);
final frame = codec.encode(payload);
final data = codec.decode(frame);

Raw-content dictionaries and formatted zstd dictionaries (magic 0xEC30A437, including its dictionary id) are supported by both the encoder and the decoder.

Output limits

final decoder = ZstdDecoder(maxOutputSize: 64 << 20); // 64 MiB

The default is unlimited; production services receiving untrusted input should set maxOutputSize (and optionally a smaller maxWindowSize).

Format support

Feature	Decode	Encode
Raw / RLE / compressed blocks	✓	✓
Raw / RLE literals	✓	✓
Huffman literals, 1 & 4 streams	✓	✓
Treeless literals / repeat Huffman	✓	✓
Predefined / RLE / FSE / repeat sequence tables	✓	✓
Repeat offsets	✓	✓
Content checksum (XXH64)	✓	optional
Frame content size / single segment	✓	one-shot
Skippable frames, concatenated frames	✓	—
Structured & raw dictionaries	✓	✓
Streaming	✓	✓

The decoder rejects windows larger than maxWindowSize (64 MiB by default) and verifies checksums when present.

Performance

The card compares Dart AOT, dart2js, dart2wasm and the native Rust zstd crate on 256 KiB Silesia slices at levels 3, 9 and 19, including ratios. It is generated and committed automatically by CI on the default branch; a missing series is omitted rather than faked.

On the current runners, Dart AOT reaches roughly 20 MiB/s compressing English text at level 3, 6.5 MiB/s at level 9, and 3.4 MiB/s at level 19 (about the same throughput as libzstd there). Decoding runs at roughly 117 MiB/s for level 3 and 122 MiB/s for level 19 frames. Compression ratios are close to libzstd at levels 3 and 9 (dickens: 0.386 vs 0.388 and 0.357 vs 0.355) and about 6% behind at level 19 (0.353 vs 0.332) because the high levels approximate the reference btopt parser. Native C remains faster; the chart makes the trade-off explicit.

Development

dart pub get
dart analyze
dart test        # Windows / Linux / macOS VM tests

Web tests (CI): dart test -p chrome and dart test -p chrome -c dart2wasm.

Test fixtures are generated by the dev-only Rust harness:

cd benchmark/rust_ref
cargo run --release -- gen-fixtures --out ../../test/fixtures

Rust is only used for tests, fixtures and benchmarks; the published library is pure Dart.

License

MIT — see LICENSE. Test vectors derived from the reference zstd project and the Silesia corpus are covered by NOTICE.