parquet 0.1.0

Pure Dart Apache Parquet reader with VM and Web support.

parquet

Pure Dart Apache Parquet reader with VM and Web support.

parquet reads Parquet files without native libraries or FFI. It runs on the Dart VM, dart2js and dart2wasm, and works with in-memory bytes, native files, blit OPFS/IndexedDB files, browser Blob/File slices, and HTTP Range readers.

Status: v0.1.0 — usable read-only release. Writing, async streaming, page-index/bloom filters and isolate parallelism are planned for v0.1.1/v0.1.2. The format reader (flat and nested) is complete for the supported feature set below.

Features

• Pure Dart, no native dependencies.
• VM + Web (dart2js; dart2wasm target planned).
• All Parquet compression codecs: Uncompressed, Snappy, Gzip, LZO, Brotli, Zstandard, LZ4_RAW; legacy LZ4 (Hadoop framing) is read.
• Encodings: PLAIN, PLAIN_DICTIONARY/RLE_DICTIONARY, RLE, BIT_PACKED levels, DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, BYTE_STREAM_SPLIT.
• Data pages v1 and v2, dictionary pages, statistics metadata.
• Nested data: LIST, STRUCT and MAP (common shapes; see limitations).
• Typed columnar API plus a row view; zero-copy Bytes for binary values.
• I/O abstraction: ParquetSource / SyncParquetSource; no coupling to File.
• Exact 64-bit handling: Int64List on the VM, two-limb storage on the web, with BigInt accessors.

Quick start

import 'dart:io';

import 'package:parquet/parquet.dart';

void main() {
  final reader = ParquetReader.fromBytes(File('data.parquet').readAsBytesSync());
  print('${reader.numRows} rows, ${reader.numRowGroups} row groups');
  print(reader.schema.columns.map((c) => c.dotPath));

  final table = reader.readTableSync();
  final id = table['id'] as Int64Column;
  final name = table['name'] as BinaryColumn;
  print(id.getInt(0));
  print(name.getString(0));
}

Projection and row groups:

final table = reader.readTableSync(columns: ['id', 'name']);
final firstGroup = reader.readRowGroupSync(0);
final row = firstGroup.row(0);
print(row.toMap());

Asynchronous sources (native file, OPFS, IndexedDB, Blob, HTTP Range):

final source = await openSource('data.parquet'); // 'opfs://…' or 'idb://…'
final reader = await ParquetReader.open(source);
final table = await reader.readTable();
await source.close();

Web:

// Blob/File: blit's read-only backend caches small blobs and slices larger ones.
final source = await openBlobSource(blob); // a JS Blob or File
final reader = await ParquetReader.open(source);

// IndexedDB: explicit fallback when OPFS sync handles are unavailable.
final idb = await openSource('idb://data.parquet');

A Uint8List can always be read synchronously without any Web storage API:

final reader = ParquetReader.fromBytes(bytes);

Supported Parquet features (v0.1.0)

Feature	Status
Physical types (BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY)	yes
Logical types (string, decimal, date/time/timestamp, int/uint, enum, json, bson, uuid, float16, list/map)	yes
Data page v1/v2, dictionary pages	yes
Compression: uncompressed, Snappy, Gzip, LZO, Brotli, Zstd, LZ4_RAW	yes
Legacy LZ4 (Hadoop framing)	read
Encodings listed above	yes
Flat schemas	yes
LIST of leaf or struct; STRUCT; MAP leaf→leaf	yes
Nested projection	no (top-level projection only)
Writing	no (v0.1.1)
Page index, bloom filters, row-group statistics pruning	no (v0.1.1/v0.1.2)
Encryption, geospatial, Variant, ALP, CDC	no (post-1.0)

Known limitations

• Deeper nesting shapes (list of list, map inside list, struct containing a list) are reported as ParquetUnsupportedException in v0.1.0.
• LZO is a legacy/deprecated Parquet codec; both Hadoop BlockCompressorStream framing and raw LZO1X blocks are read.
• Encrypted footers (PARE) are rejected.
• maxRows truncation works for flat columns; nested truncation lands in v0.1.1.
• Isolate parallelism is planned for v0.1.1; v0.1.0 is single-isolate.
• Web Blob sources are read-only; IndexedDB is an explicit async fallback (idb://), not a replacement for OPFS on every browser.

I/O

ParquetReader accepts either a synchronous SyncParquetSource or an asynchronous ParquetSource:

final source = MemorySource.fromUint8List(bytes); // sync
final reader = ParquetReader.fromBytes(bytes);   // convenience

Adapters live in lib/src/io/:

• MemorySource
• native file (sync/async, dart:io, VM only)
• BlitSource (blit BytesFile: native files, browser OPFS, Blob, IndexedDB)
• BlobSliceSource / openBlobSource (browser Blob/File via blit 0.1.6)
• BlitSource.openIndexedDb / openIdbBlitSource (blit 0.1.6 IDB backend)
• HttpRangeSource (callback-based; bring your own HTTP client)
• openSource / openSourceSync with file://, opfs:// and idb:// support

Dependencies

• blit — zero-copy Bytes, OPFS, Blob and IndexedDB BytesFile backends
• snappy_neo
• zstd_neo
• brotli_neo
• lzo_neo
• dart_lz4
• just_deflate

Development

Local development runs Windows VM tests only:

dart analyze
dart format --set-exit-if-changed .
dart test

Linux/macOS, Chrome and Wasm tests run in GitHub Actions. Benchmarks are built with benchmark_test and rendered by benchcat in CI; the chart above is committed to assets/benchmarks.svg.

License

MIT. See LICENSE and NOTICE.