ES Compression: A Compression Framework for Dart

Description

Compression framework for Dart providing FFI implementations for Brotli, Lz4, Zstd (Zstandard) with ready-to-use prebuilt binaries for Win/Linux/Mac.

This work is an inspired port of the Unified Compression Framework from the VAST Platform (VA Smalltalk) language and development environment.

See the Design Document for detailed information on how this package was designed and implemented.

Below is a simple example of what an encode/decode would look like:

import 'dart:convert';
import 'package:es_compression_silicon_support/brotli.dart';
import 'package:es_compression_silicon_support/lz4.dart';
import 'package:es_compression_silicon_support/zstd.dart';

/// Brotli, Lz4, Zstd usage example
void main() {
  final bytes = utf8.encode('Hello Dart');
  for (final codec in [brotli, lz4, zstd]) {
    final encoded = codec.encode(bytes);
    final decoded = codec.decode(encoded);
    print(utf8.decode(decoded));
  }
}

Executables

In the bin subdirectory, the following executables are provided.

Executable	Source	Description
escompress	es_compress.dart	Encode/Decode files using brotli, gzip, lz4 and zstd

If you want to use escompress on the command line, install it using pub global activate:

> pub global activate es_compression

escompress

escompress is a program that will encode/decode files using brotli, gzip, lz4 or zstd.

The user provides the input and output file names from the command line. By default, the file extension of either the input or output file name is used to determine which algorithm and encode/decode mode to use. The user can provide additional command line arguments to make these decisions explicitly. Some examples are provided below.

Examples

Encode input.txt to output.lz4 using Lz4 compression:

> escompress -i"input.txt" -o"output.lz4"

Encode input.txt to output.lz4 using Lz4 compression at compression level 3:

> escompress -l 3 -i"input.txt" -o"output.lz4"

Decode input.brotli to output.txt using Brotli compression:

> escompress -i"input.brotli" -o"output.txt"

Encode input.txt to output.compressed using Zstd compression:

> escompress -e -a zstd -i"input.txt" -o"output.compressed"

Decode input.compressed to output.txt using GZip compression:

> escompress -d -a gzip -i"input.compressed" -o"output.txt"

Print help:

> escompress -h

Examples

In the example subdirectory, the following examples are provided to demonstrate usage of the converters and the framework.

Example	Description
brotli_example.dart	Encode/Decode in one-shot and streaming modes using the Brotli FFI-based implementation
lz4_example.dart	Encode/Decode in one-shot and streaming modes using the Lz4 FFI-based implementation
rle_example.dart	A simple RLE compression example designed to show how build custom codecs
zstd_example.dart	Encode/Decode in one-shot and streaming modes using the Zstd FFI-based implementation

To run (brotli shown below):

> dart example/brotli_example.dart

Tests

In the test subdirectory, the following tests are provided for the compression framework and encoder/decoder implementations.

Test	Description
benchmarks_test.dart	Test benchmarks defined in benchmark subdirectory
brotli_test.dart	Test encoding/decode of the Brotli FFI-based implementation
buffer_test.dart	Test CodecBuffer and friends in the compression framework
escompress_test.dart	Test escompress binary in the bin subdirectory
examples_test.dart	Test examples defined in the example subdirectory
lz4_test.dart	Test encoding/decode of the Lz4 FFI-based implementation
zstd_test.dart	Test encoding/decode of the Zstd FFI-based implementation

To run test suite:

> pub run test

Benchmarks

In the benchmark subdirectory, the following benchmarks are provided to help understand encoder/decoder performance and tradeoffs involved with parameters like buffer sizing.

Benchmark	Description
brotli_benchmark.dart	Benchmark encoding/decode of the Brotli FFI-based implementation
gzip_benchmark.dart	Benchmark encoding/decode of the GZip implementation from the Dart SDK
lz4_benchmark.dart	Benchmark encoding/decode of the Lz4 FFI-based implementation
zstd_benchmark.dart	Benchmark encoding/decode of the Zstd FFI-based implementation

To run (lz4 shown below):

> dart benchmark/lz4_benchmark.dart

Deployment

FFI-based implementations will need access to the low-level shared libraries (i.e. .dll, .so, .dylib). This package offers a flexible library loader that can be customized for end-user deployment needs.

By default, the resolution order is:

• Environment Variable
• Package-Relative
• Script-Relative
• OS-Dependent

User Provided Resolution: The user can override the above resolution with a user provided library path. The different strategies for locating shared libraries are described below.

Environment Variable Resolution

An environment variable can be defined that provides the path to the shared library. This is either the path to the shared library file or the directory which should contain the filename of the form es{algo}_{os}{bitness}.{ext}. For example, the filename for lz4 on 64-bit windows would be eslz4_win64.dll.

Codec	Environment Variable
brotli	BROTLI_LIBRARY_PATH
lz4	LZ4_LIBRARY_PATH
zstd	ZSTD_LIBRARY_PATH

Package-Relative Resolution

Prebuilt shared libraries for Win/Linux/Mac are provided in the blob directory of each FFI codec implementation. These have been built by the package maintainer using the blob_builder tool.

The distributed shared libs for a codec named 'xxx' is expected to be located in lib/src/xxx/blobs by default.

Script-Relative Resolution

An attempt is made to find the shared library in the same directory as the running script. The name of the shared library is expected to be of the form es{algo}_{os}{bitness}.{ext}. For example, the filename for zstd on 64-bit linux would be eszstd_linux64.dll.

OS-Dependent Resolution

A call to DynamicLibrary.open() is made for the filename of the form es{algo}_{os}{bitness}.{ext} which will use the resolution rules for the operating system.

User Provided Resolution

Users of this package have the option to override the library path early in the program.

Provided FFI Codecs have static getters/setters for the libraryPath. Users should be sure set the libraryPath early before the first use. A StateError will be thrown if a user attempts to set the libraryPath more than once.

final codec = ZstdCodec.libraryPath = '/path/to/shared/library.so';

Code Signing

Windows

Provided dlls are digitally signed with an MS authenticode certificate owned by Instantiations, Inc.

Linux

N/A

Mac

Provided dylibs are not currently signed, and recent versions of OSX will refuse to load them unless you allow it from the Security & Privacy dialog.

The build scripts have been provided blob_builder and gives you access to build and sign them yourself, if desired.

Instantiations may sign the libraries in the future, and if so, it will be noted in the changelog and here.

Tools

In the tool subdirectory, the following tools are provided.

Blob Builder

blob_builder is a cmake-based build generator which builds all the prebuilt shared libraries and copies them to their appropriate location in the dart library.

The maintainers use this tool to update the prebuilt shared libraries. It can also be used to build custom versions of the various libraries by making adjustments to CMake variables that define the version level information.

Prebuilt shared libraries for Win/Linux/Mac are provided in the blobs directory for each FFI codec implementation. The distributed shared libs for a codec named xxx is expected to be located in lib/src/xxx/blobs.

There are instructions in the main CMakeLists.txt file that describe how to run the tool.

Framework

Initial codecs provided by this library use FFI-based implementations. However, the framework easily allows for pure dart implementations for use within a front-end web context. rle_example.dart shows a simple example of how to accomplish this.

The GZipCodec from dart:io served as a great starting point for understanding how to put the VAST Platform framework abstractions in terms of Dart codecs, converters, filters, sinks.

The major compression framework abstractions are:

• CodecConverter - Connects the compression framework to Converter in dart:convert.
• CodecFilter - Direct processing of byte data and provides low-level compression implementation and hooks.
• CodecSink - A type of ByteConversionSink for efficient transmission of byte data.
• CodecBuffer - A buffer with a streaming API that is backed by either native or dart heap bytes.

Features and bugs

Please file feature requests and bugs at the issue tracker.

They will be reviewed and addressed on a best-effort basis by Instantiations, Inc.

About Us

Since 1988, Instantiations has been building software to meet the diverse and evolutionary needs of our customers. We've now added Dart and Flutter to our toolbox.

For more information about our custom development or consulting services with Dart, Flutter, and other languages, please visit: https://www.instantiations.com/services/

Libraries

brotli

Brotli encoder and decoder using a fast FFI-based implementation.

framework

Core compression framework library.

framework_io

Compression framework for non-web contexts. This is required for FFI-based implementations.

lz4

Lz4 encoder and decoder using a fast FFI-based implementation.

zstd

Zstd (Zstandard) encoder and decoder using a fast FFI-based implementation.