pure_zstd 1.1.0

A zstd compressor and decompressor written in plain Dart. Reads and writes RFC 8878 frames with no native library and no platform channel, verified against libzstd.

pure_zstd

A zstd compressor and decompressor in plain Dart, with no native dependency.

It exists because every zstd binding on pub.dev is a wrapper over libzstd, and an app that reads block-compressed data on a phone would rather not carry a native library per platform to do it. The decoder came first and is the reason the package exists; the encoder followed for feature completeness.

It decodes the real data it was written for, correctly and completely, and it is roughly four times slower than native zstd. The encoder's ratio lands between libzstd's level 1 and its level 19, often much nearer 19, at roughly a tenth to a twentieth of native speed. See Speed below before depending on either.

Using it

import 'package:pure_zstd/pure_zstd.dart';

final plain = zstdDecode(frame);

// Decoding many frames: hold the decoder, which keeps its entropy tables and
// scratch buffers between calls and then allocates only the output.
final decoder = ZstdDecoder();
for (final frame in frames) {
  use(decoder.decode(frame));
}

A frame that declares no content size needs expectedSize, because the output is allocated whole rather than grown.

final frame = zstdEncode(plain);

// Encoding many inputs: hold the encoder, which keeps its hash table and
// scratch buffers between calls.
final encoder = ZstdEncoder();
for (final plain in inputs) {
  send(encoder.encode(plain));
}

Every frame the encoder writes declares its content size and carries a checksum; checksum: false drops the four bytes. level is 1 to 5 and trades speed for ratio: 1 takes the first paying match greedily, 2, the default, also weighs a match starting one byte later, and 3 weighs starts one and two bytes later. Levels 4 and 5 walk a chain of earlier candidates, eight and thirty-two deep, and add three-byte matches, which is what structured binary data such as integer arrays wants; on the near-sorted integer corpus level 4 lands within two percent of zstd -19 at half the default level's speed.

What it implements

RFC 8878 is the specification. Of it:

Part of the format	State
Frame header, single segment and windowed	Decoded. The window descriptor is skipped, since the output is allocated from the content size.
Raw, RLE and compressed blocks	All three.
Multi-block frames	Yes, including a later block reusing an earlier one's tables and recent offsets.
Raw, RLE, compressed and treeless literals	All four, in one-stream and four-stream form.
Huffman weights	Both the direct nibble form and the FSE-coded form.
Sequence tables	Predefined, RLE, FSE-compressed and repeat, for all three of literal length, match length and offset.
Repeated offsets	Yes, including the no-literals shift.
Content checksum	Verified. XXH64 over the output, compared with the stored low 32 bits. Hashed only when the frame carries one.
Dictionaries	Refused.
Skippable frames, several frames in one buffer	Refused. Reading one frame is the whole API, and input left over after it is an error.
Streaming, or decoding part of a frame	Not offered.

The encoder writes one frame per call and chooses what to emit per 128 KiB block by what it costs:

Choice	What the encoder does
Frame header	Single segment with an explicit content size up to 128 MiB. Above that, a windowed header declaring a 128 MiB window, which every decoder accepts by default, and no match reaches further back than that.
Blocks	Raw, RLE or compressed, whichever is smallest.
Matches	Through a hash table plus the last repeated offset, four bytes minimum, taken only when the sequence costs fewer bits than coding the same bytes as literals. From level 2 up, a found match is deferred when a later start scores better; levels 4 and 5 walk a chain of earlier candidates and add three-byte matches.
Repeated offsets	Emitted whenever an offset matches one of the three recent ones, including the no-literals shift.
Sequence tables	Chosen per block and per alphabet: predefined, RLE, FSE built from this block's counts, or a repeat of the previous block's, whichever prices smallest.
Literals	Raw, RLE, Huffman, or treeless reusing the previous block's table, per block; one stream up to 1,023 literals and four streams above.
Huffman weights	Package-merge lengths capped at eleven bits, written in the direct nibble or the FSE-coded form, whichever is smaller.
Dictionaries	Not written.

What it leaves on the table against libzstd: optimal parsing (every match here is priced on its own rather than against the parse as a whole), binary-tree search, long-distance matching, and dictionaries. That is the remaining gap to the highest levels on long or repetitive inputs.

It targets native Dart and not the web: the literals header reads a five-byte field with a 32-bit shift, and the bit reader carries 56 bits in an int. Both are exact on the VM and on AOT, and neither is on JavaScript.

Correctness

The reference is pyzstd, which wraps libzstd. tool/oracle.py extracts every block of a .hgtz container as its own frame, decompresses it with pyzstd, hands it to tool/verify.dart, and compares byte for byte.

./tool/oracle.py <a .hgtz container>
./tool/oracle.py --every 1700 <a directory of them>

As of 2026-09-21 that is 32 real containers, 2,615 blocks and 0.3 GB of output, all identical, spanning both the one and three arc-second sets, every thinned column count from 3601 down to 361, and latitudes from 88 S to 80 N.

dart test needs no Python. It decodes the five blocks of the container in test/fixtures/, whose every sample is a known function of its position, so a wrong decode cannot pass.

The encoder is held to two oracles and no expectation is written by hand: every frame it emits must decode to its input through this package's own decoder and through the zstd CLI. The entropy stages are also tested alone, each encoding side against the decoding side the decoder already trusts. test/generic_corpus_test.dart runs both oracles over generic data shapes, prose, JSON logs, CSV, base64, hex dumps, a sorted word list, XML, binary time series, DNA, UUID lists, varint records, near-sorted integers, already-compressed data, and mixed compressible and incompressible segments, each with a ratio ceiling so an encoding path that quietly dies fails a test instead of shipping bigger frames. Everything in it is generated in the test from seeded generators or public boilerplate text; nothing is committed and nothing is downloaded.

Speed

Measured 2026-09-21 on this eight-core Linux box, dart compile exe, Dart 3.12.2. One 256 by 256 block is 131,072 bytes out. The frames are read into memory first, so only decoding is timed, and the figure includes allocating the output buffer because the API allocates it.

Block set	pure_zstd, AOT	pyzstd (libzstd)
Colorado, N39W105, 196 blocks, median	0.66 ms	0.18 ms
Alaska, N60W160, 98 blocks, median	0.94 ms	not measured
One block, repeated 1000 times	0.85 ms	0.22 ms
113 blocks, a 360 degree sweep	67 ms	19 ms

That is 140 to 200 MB/s of output. The JIT is about 25% slower than AOT, so measure with dart compile exe.

The cost is almost all in the sequence loop rather than in Huffman literals. A block of real terrain carries around 1,700 literals and 14,400 sequences, so the hot path is three FSE state updates, six bit reads and a nine-byte match copy, about 46 ns per sequence against native's dozen or so. The headroom left is bounds checks and the byte-at-a-time match copy; widening that copy to 64-bit words, the way libzstd does, is the next thing to try.

Run it yourself:

dart compile exe benchmark/bench.dart -o /tmp/bench && /tmp/bench

Encoding, measured 2026-10-25 on the same box, AOT, at the default level 2. The inputs are the seeded generators in test/corpus.dart:

Input	pure_zstd frame	zstd -1 frame	zstd -19 frame	encode speed
300 KB of 3-byte tokens, pool 200	171,062 B	287,498 B	134,539 B	21 MB/s
320 KB of 8-byte tokens, pool 2,000	90,597 B	125,955 B	89,949 B	32 MB/s
300 KB of a skewed alphabet	168,000 B	166,923 B	167,090 B	16 MB/s
300 KB of noise	307,222 B	307,222 B	307,222 B	26 MB/s
600 KB of mixed segments	181,936 B	205,997 B	173,646 B	35 MB/s
84 KB of Dart source, four copies	6,177 B	6,394 B	5,428 B	41 MB/s

Level 1 is 20 to 30% faster and a touch larger; levels 4 and 5 run at about half this speed and pull ahead on structured binary data. Native zstd -1 runs at a few hundred MB/s, so encoding is roughly ten to twenty times slower than native where decoding is four.

Acknowledgements

This is an independent implementation. It contains no code from libzstd, and nothing about its licence reaches this package. It would not exist without the following, and they are named here because that is where the debt actually is.

RFC 8878, "Zstandard Compression and the application/zstd Media Type", by Yann Collet and Murray Kucherawy, is the specification this was written from. Every table in lib/src/tables.dart was checked against section 3.1.1.3.2 entry by entry, and the FSE and Huffman decoders follow the reference algorithms it describes. A format specification written clearly enough to implement from is a gift, and this one is.

Zstandard, by Yann Collet and contributors at Meta, is the format and the reference implementation. Its licence is BSD-3-Clause, and it obliges nothing here; it is named because without it there would be nothing to decode.

pyzstd wraps libzstd and is the oracle every test in this package is measured against. Correctness here means "agrees with libzstd, byte for byte", and pyzstd is how that is asked. The committed fixtures under test/fixtures/corpus/ are output of the zstd CLI and of pyzstd, which is tool output rather than anything derived from their source.

The Copernicus Programme. One committed fixture, test/fixtures/corpus/direct_nibble.zst, is a block of real terrain, kept because it is the only small input that exercises the direct-nibble Huffman weight form. Its licence requires this to travel with it:

Produced using Copernicus WorldDEM-30 © DLR e.V. 2010-2014 and © Airbus Defence and Space GmbH 2014-2018 provided under COPERNICUS by the European Union and ESA; all rights reserved.

The Dart team, for making int an exact 64-bit integer on the VM and on AOT. The bit reader carries 56 bits in one and XXH64's multiplies wrap exactly, neither of which would be true on a platform with doubles underneath.

Measured on the target device, 2026-09-21

Sixteen real 256x256 blocks lifted out of one elevation container, 131,072 bytes of output each, the same blocks in every row so the numbers are comparable. Median of 960 timed calls after 320 warm-up calls.

where	mode	median	MB/s	113-block sweep
Pixel 10 Pro XL	profile, AOT	0.414 ms	302	47 ms
Pixel 10 Pro XL	debug, JIT	0.540 ms	231	61 ms
this laptop	AOT	0.917 ms	136	104 ms
this laptop	JIT	1.125 ms	111	127 ms

The phone is about twice as fast as the laptop at this, which is why the desktop figures in the commit message read worse than reality.

Native, for scale: pyzstd on the laptop is 0.175 ms on comparable blocks. That is a Python C extension rather than Dart FFI, so take it as an indication. No native-on-phone figure was taken.

Do not benchmark against test/fixtures/N51E000.hgtz. Its samples are a synthetic pattern that compresses to almost no sequences, and decode cost tracks sequence count, so it reads about six times too fast. That mistake was made once already.

---

If this saved you some time, please consider sponsoring the work.