geo_osm_pbf 1.0.0

Streaming OpenStreetMap `.osm.pbf` reader and extract downloader in pure Dart. Hand-rolled protobuf decoding with no dependencies, delivering nodes, tagged nodes, ways and relations with their full tags.

geo_osm_pbf

A streaming OpenStreetMap .osm.pbf reader and extract downloader written in pure Dart. It hand-decodes the protobuf wire format and inflates blocks with dart:io codecs, so it has no protobuf or compression dependency.

This is the acquisition and decoding layer, and nothing more: it delivers nodes, tagged nodes, ways and relations with their full tags, and leaves every interpretation to its consumers. geo_route_finder turns the same stream into a routing graph; geo_tile_builder turns it into vector map tiles. Neither interpretation lives here.

Runs anywhere Dart runs (server, CLI, desktop, mobile, Flutter — any non-web target, since it uses dart:io for files and networking).

API Documentation

See the API Documentation for a full list of functions, classes and extensions.

Features

• Streaming and memory-bounded. One fileblock (~8k entities) at a time, so peak memory is a single decompressed block regardless of file size. Country extracts parse without ever loading the whole file.
• Nothing is decoded that nobody asked for. Node tags are read only when a tagged-node callback is given; relation members only when a relation callback is. A consumer pays nothing for what it does not use.
• All four element kinds. Dense nodes, sparse nodes, ways and relations — each with its complete, unfiltered tag map.
• Header without a scan. readHeader returns the declared bounding box and feature flags after a single blob read.
• Pluggable download sources. Region → URL mapping lives behind OsmDownloadSource, with Geofabrik, OSM France, BBBike and Planet built in, automatic mirror benchmarking, and resumable, cache-aware downloads.
• No runtime dependencies beyond path.

Architecture

OsmDownloader ──► .osm.pbf ──► OsmPbfParser ──┬─► OsmHeader     (bbox, features)
 (Geofabrik,                    (streaming)   ├─► GeoNode       (every node)
  OSM France,                                 ├─► GeoTaggedNode (places, POIs)
  BBBike, Planet)                             ├─► GeoWay        (+ full tags)
                                              └─► GeoRelation   (+ members)
                                                        │
                            ┌───────────────────────────┴──────────────┐
                            ▼                                          ▼
                     geo_route_finder                          geo_tile_builder
                     (routing graph)                            (vector tiles)

Getting started

dependencies:
  geo_osm_pbf: ^1.0.0

Usage

Read an extract

import 'package:geo_osm_pbf/geo_osm_pbf.dart';

const parser = OsmPbfParser();

await parser.parse(
  'santa-catarina.osm.pbf',
  onWay: (way) {
    if (way.tags['highway'] != null) {
      print('${way.id}: ${way.tags['name']} (${way.nodeIds.length} nodes)');
    }
  },
);

Read only what you need

Each entity type is decoded only when its callback is supplied, and readNodes/readWays/readRelations skip whole types on a given pass — which is how a two-pass builder learns which node ids it needs before reading any coordinates.

// Pass 1: ways only. Nodes are not even visited.
final needed = <int>{};
await parser.parse(
  path,
  readNodes: false,
  onWay: (way) => needed.addAll(way.nodeIds),
);

// Pass 2: only the coordinates that pass 1 asked for.
final coords = <int, GeoNode>{};
await parser.parse(
  path,
  readWays: false,
  onNode: (node) {
    if (needed.contains(node.id)) coords[node.id] = node;
  },
);

Places and points of interest

Most nodes are untagged shape points. onTaggedNode fires only for the ones that describe something, so a consumer looking for places never walks tens of millions of anonymous vertices.

await parser.parse(
  path,
  onTaggedNode: (node) {
    final place = node.tags['place'];
    if (place != null) {
      print('$place: ${node.tags['name']} @ ${node.coordinate}');
    }
  },
);

Relations

await parser.parse(
  path,
  readNodes: false,
  readWays: false,
  onRelation: (relation) {
    if (relation.isMultipolygon) {
      final outer = relation.membersWithRole('outer').map((m) => m.ref);
      print('${relation.tags['natural']}: outer rings $outer');
    }
  },
);

The header, cheaply

final header = await parser.readHeader(path);
print(header?.bbox);                 // extent, without scanning the file
print(header?.unsupportedFeatures);  // empty means safe to parse

Download an extract

final registry = OsmDownloadSourceRegistry.withDefaults(benchmarkByDefault: true);

registry.register(
  GeofabrikSource(baseUrls: ['https://maps.corp.example/geofabrik/']),
  priority: 0,
);
registry.setEnabled('planet', false); // opt out of the whole-planet fallback

final downloader = OsmDownloader(sourceResolver: registry);
final pbf = await downloader.downloadRegion(
  region: 'south-america/brazil/santa-catarina',
);

Implement OsmDownloadSource (or OsmMirroredSource) to add new providers, cloud buckets or local mirrors — no downloader changes.

How it works

Streaming

The file is a sequence of length-prefixed blobs. Each is read, inflated and decoded on its own, then discarded, so peak memory tracks the largest single block rather than the file. Nothing is buffered across blocks.

Dense nodes and their tag stream

Real files store nodes in a DenseNodes message where ids and coordinates are delta-encoded across the whole block, and tags live in one flat keys_vals array of string-table indices with a 0 terminating each node — including nodes that have no tags at all. Miss a terminator and every subsequent tag lands on the wrong node, which is why that alignment has a test of its own.

Relations

Member ids are delta-encoded against the previous member of the same relation, regardless of member type, and roles are string-table indices. Members are returned as ids rather than resolved elements: a regional extract routinely references ways it does not contain, so consumers must tolerate members they cannot resolve.

Performance targets

Parsing is bounded by inflate and varint decoding, and peak memory is one decompressed block (~8k entities) regardless of input size. A caller that supplies only onWay pays nothing to decode node tags or relation members.

Running the example, tests and benchmark

dart run example/geo_osm_pbf_example.dart
dart run example/geo_osm_pbf_example.dart ./maps/santa-catarina.osm.pbf
dart test

Source

The official source code is hosted @ GitHub:

• https://github.com/gmpassos/geo_osm_pbf

Features and bugs

Please file feature requests and bugs at the issue tracker.

Contribution

Any help from the open-source community is always welcome and needed:

• Found an issue?
  • Please fill a bug report with details.
• Wish a feature?
  • Open a feature request with use cases.
• Are you using and liking the project?
  • Promote the project: create an article, do a post or make a donation.
• Are you a developer?
  • Fix a bug and send a pull request.
  • Implement a new feature.
  • Improve the Unit Tests.
• Have you already helped in any way?
  • Many thanks from me, the contributors and everybody that uses this project!

If you donate 1 hour of your time, you can contribute a lot, because others will do the same, just be part and start with your 1 hour.

Author

Graciliano M. Passos: gmpassos@GitHub.

License

Apache License - Version 2.0