lmdb library

A high-level Dart interface for LMDB (Lightning Memory-Mapped Database).

LMDB is a fast key-value store that provides:

Memory-mapped file storage for optimal performance
ACID transactions (depending on given flags) with full CRUD operations
Multiple named databases within a single environment
Concurrent readers with a single writer
Zero-copy lookup for read operations

This Dart wrapper adds the following features:

Automatic transaction management for simple operations
UTF-8 string support for keys and values
Named databases with automatic handle management
Comprehensive statistics and analysis tools
Safe memory management through FFI

Basic Usage

Simple string operations with automatic transactions:

final db = LMDB();
await db.init('/path/to/db');

// Store and retrieve data
await db.putUtf8Auto('key', 'value');
final value = await db.getUtf8Auto('key');

db.close();

Named Databases

LMDB supports multiple named databases within a single environment:

// Initialize with support for multiple DBs
await db.init('/path/to/db',
  config: LMDBInitConfig(
    mapSize: 10 * 1024 * 1024,  // 10 MB
    maxDbs: 5,  // Support up to 5 named DBs
  )
);

// Store data in different named databases
await db.putUtf8Auto('key', 'value1', dbName: 'users');
await db.putUtf8Auto('key', 'value2', dbName: 'settings');

// Data is separated by database
print(await db.getUtf8Auto('key', dbName: 'users'));     // value1
print(await db.getUtf8Auto('key', dbName: 'settings'));  // value2

// List all named databases
final databases = await db.listDatabases();

Transaction Management

For better control and performance, use explicit transactions:

final txn = await db.txnStart();
try {
  // Multiple operations in one atomic transaction
  await db.putUtf8(txn, 'key1', 'value1', dbName: 'users');
  await db.putUtf8(txn, 'key2', 'value2', dbName: 'users');
  await db.txnCommit(txn);
} catch (e) {
  await db.txnAbort(txn);
  rethrow;
}

Read-only transactions for better concurrency:

final readTxn = await db.txnStart(
  flags: LMDBFlagSet()..add(MDB_RDONLY)
);

Performance Features

Optimize for specific use cases:

// High-performance mode (less durability)
await db.init('/path/to/db',
  flags: LMDBFlagSet()
    ..add(MDB_NOSYNC)      // Don't sync to disk immediately
    ..add(MDB_WRITEMAP)    // Use write-ahead mapping
);

// Configure map size for expected data volume
await db.init('/path/to/db',
  config: LMDBInitConfig.fromEstimate(
    expectedEntries: 1000000,
    averageKeySize: 16,
    averageValueSize: 64,
  )
);

LMDB MapSize Behavior

The MapSize in LMDB determines the maximum database size and behaves as follows:

Read-Only Access:

Databases can be opened with any MapSize (even smaller) in read-only mode
Perfect for use cases like dictionaries or lookups where only reading is required
The actual DB size can be determined using statsAuto()

Write Access:

MapSize must be at least as large as the current DB size
Write operations will fail with MDB_MAP_FULL when MapSize limit is reached
MapSize can only be set when opening the DB, not during runtime

Size Adjustment:

A DB can be reopened with larger MapSize to allow growth
It's recommended to reserve more MapSize than currently needed
Typical pattern: Open read-only to check size -> Close -> Reopen with proper MapSize

Example Usage:

    final db = LMDB();
    // Open with 100MB initial size
    await db.init(path, config: LMDBInitConfig(mapSize: 100 * 1024 * 1024));

Note: MapSize can be important for performance and resource management. Choose it based on:

Expected data growth
Available system resources
Application requirements
Even with small MapSize (e.g. 1MB for a 100MB DB), LMDB maintains very good performance !

Monitoring and Analysis

Track database health and performance:

// Get statistics for specific database
final stats = await db.statsAuto(dbName: 'users');
print('Entries: ${stats.entries}');
print('Tree depth: ${stats.depth}');

// Analyze database efficiency
final analysis = await db.analyzeUsage();
print(analysis);

// Monitor all databases
final allStats = await db.getAllDatabaseStats();
allStats.forEach((dbName, stats) {
  print('$dbName: ${stats.entries} entries');
});

Common Use Cases

Simple Key-Value Store:

Use automatic methods (putUtf8Auto, getUtf8Auto)
Perfect for configuration storage, caching

Multiple Data Types:

Use named databases to separate different data types
Each database can have its own configuration

High-Performance Requirements:

Use explicit transactions for batching
Configure flag sets for specific durability needs
Adjust map size based on data volume

Concurrent Access:

Multiple readers can access simultaneously
Use read-only transactions when possible
Single writer ensures data consistency

Best Practices

Always close the database when done
Use try-catch blocks with transactions
Configure map size appropriately
Monitor database statistics for optimization
Use named databases for data organization
Consider using read-only transactions for queries

Error Handling

The library provides specific error handling through LMDBException:

try {
  await db.putUtf8Auto('key', 'value');
} catch (e) {
  if (e is LMDBException) {
    print('LMDB Error: ${e.errorString}');
    print('Error code: ${e.errorCode}');
  }
}

Database Organization

Named databases can be used to organize different types of data:

// Initialize with multiple databases
await db.init('/path/to/db',
  config: LMDBInitConfig(maxDbs: 5)
);

// Users database
await db.putUtf8Auto(
  'user:123',
  jsonEncode({'name': 'John', 'age': 30}),
  dbName: 'users'
);

// Settings database
await db.putUtf8Auto(
  'theme',
  jsonEncode({'dark': true, 'fontSize': 14}),
  dbName: 'settings'
);

// Logs database
await db.putUtf8Auto(
  DateTime.now().toIso8601String(),
  'Application started',
  dbName: 'logs'
);

Database Maintenance

Regular maintenance tasks:

// Ensure data is synced to disk
await db.sync(true);

// Check database statistics
final stats = await db.getEnvironmentStats();
if (stats.overflowPages > 0) {
  print('Warning: Database has overflow pages');
}

// Monitor database growth
final efficiency = LMDBConfig.analyzeEfficiency(stats);
if (!efficiency.isWellBalanced) {
  print('Warning: B+ tree needs optimization');
}

Platform Specifics

LMDB behavior can vary by platform:

Windows: Some features like MDB_WRITEMAP might not work as expected
Linux/Unix: Full feature support, including file permissions
macOS: Similar to Linux/Unix with some performance differences

Configure accordingly:

if (Platform.isWindows) {
  await db.init('/path/to/db',
    flags: LMDBFlagSet()..add(MDB_NOSYNC)  // Skip MDB_WRITEMAP on Windows
  );
} else {
  await db.init('/path/to/db',
    flags: LMDBFlagSet()
      ..add(MDB_NOSYNC)
      ..add(MDB_WRITEMAP),
    config: LMDBInitConfig(mode: '644')  // Unix permissions
  );
}

Performance Optimization

Tips for optimal performance:

Use appropriate map sizes to avoid reallocations
Batch operations in transactions
Use read-only transactions for queries
Consider durability vs speed tradeoffs

Example of batch processing:

final txn = await db.txnStart();
try {
  for (var i = 0; i < 1000; i++) {
    await db.putUtf8(
      txn,
      'key$i',
      'value$i',
      dbName: 'batch_data'
    );
  }
  await db.txnCommit(txn);
} catch (e) {
  await db.txnAbort(txn);
  rethrow;
}

Memory Management

LMDB uses memory-mapped files, so consider:

Set appropriate map sizes for your data
Monitor system memory usage
Use efficient key/value sizes
Clean up with dispose() when done

Example of memory-conscious initialization:

await db.init('/path/to/db',
  config: LMDBInitConfig(
    mapSize: LMDBConfig.calculateMapSize(
      expectedEntries: 1000000,
      averageKeySize: 16,
      averageValueSize: 64,
      overheadFactor: 1.5
    )
  )
);

Classes

CursorEntry: Represents an entry returned by cursor operations
DatabaseEfficiency: Represents database efficiency metrics for analyzing LMDB performance.
DatabaseStats: Represents statistical information about an LMDB database.
LMDB: A high-level Dart interface for LMDB (Lightning Memory-Mapped Database).
LMDBConfig: Configuration and utility functions for LMDB (Lightning Memory-Mapped Database)
LMDBFlagSet: A class for managing LMDB flag combinations in a type-safe way.
LMDBInitConfig: Configuration class for LMDB initialization.
MDB_txn: An opaque handle for a database transaction.

Enums

CursorOp: Public cursor operation modes for LMDB

Constants

MDB_CORRUPTED → const int: MDB_CORRUPTED: Located page was wrong type. Indicates database corruption or structural integrity issues. Database may need recovery or rebuilding.
MDB_CREATE → const int: / Creates the named database if it doesn't exist.
MDB_DUPFIXED → const int: Specifies fixed-size duplicate data items.
MDB_DUPSORT → const int: Enables duplicate keys in the database.
MDB_FIXEDMAP → const int: Use fixed-size memory map.
MDB_INTEGERDUP → const int: MDB_INTEGERDUP: Used only with MDB_DUPFIXED databases. Indicates that duplicate data items are binary integers. This flag enables efficient storage and comparison of integer values similar to MDB_INTEGERKEY for keys.
MDB_INTEGERKEY → const int: Specifies that keys are binary integers in native byte order.
MDB_INVALID → const int: MDB_INVALID: File is not an LMDB file. Occurs when attempting to open a file that:
MDB_KEYEXIST → const int: MDB_KEYEXIST: Key/data pair already exists. Occurs during put operations when the key already exists and MDB_NOOVERWRITE was specified.
MDB_MAP_FULL → const int: MDB_MAP_FULL: Environment mapsize limit reached. Occurs when:
MDB_MAP_RESIZED → const int: MDB_MAP_RESIZED: Database was resized externally. Occurs when another process has increased the database size. Action required: Close and reopen the environment. Common scenario: Multi-process access with dynamic growth.
MDB_MAPASYNC → const int: Enables asynchronous flushes to disk when using MDB_WRITEMAP.
MDB_NOLOCK → const int: When used:
MDB_NOMEMINIT → const int: Skips initialization of malloc'd memory before writing.
MDB_NOMETASYNC → const int: Don't sync meta pages when committing transaction.
MDB_NOOVERWRITE → const int: Flag for put operations that prevents overwriting existing keys.
MDB_NORDAHEAD → const int: Disables read-ahead for random access patterns.
MDB_NOSUBDIR → const int: Opens a database file directly instead of using a directory.
MDB_NOSYNC → const int: Disable syncing of system buffers to disk on transaction commit.
MDB_NOTFOUND → const int: MDB_NOTFOUND: Key/data pair not found (EOF). Occurs when:
MDB_NOTLS → const int: Disables thread-local storage.
MDB_PAGE_NOTFOUND → const int: MDB_PAGE_NOTFOUND: Requested page not found. Internal error indicating database corruption or invalid page access.
MDB_PANIC → const int: MDB_PANIC: Update of meta page failed or environment had fatal error. Severe error condition requiring immediate attention. Database might be corrupted or system resources exhausted.
MDB_PREVSNAPSHOT → const int: Allows read-only access if write access is unavailable.
MDB_RDONLY → const int: Opens the environment in read-only mode.
MDB_REVERSEDUP → const int: Enables reverse string comparison for duplicate data items.
MDB_REVERSEKEY → const int: Stores key/data pairs in reverse byte order.
MDB_SUCCESS → const int: MDB_SUCCESS: Operation completed successfully. Return code: 0
MDB_VERSION_MISMATCH → const int: MDB_VERSION_MISMATCH: Environment version mismatch. Occurs when:
MDB_WRITEMAP → const int: Use a writeable memory map instead of malloc/msync for database operations.

Functions

fetchNativeLibraries({String? targetDir}) → Future<void>: Downloads native libraries from GitHub releases with manifest support

Exceptions / Errors

LMDBException: Exception class for LMDB-specific errors.

lmdb library

Basic Usage

Named Databases

Transaction Management

Performance Features

LMDB MapSize Behavior

Monitoring and Analysis

Common Use Cases

Best Practices

Error Handling

Database Organization

Database Maintenance

Platform Specifics

Performance Optimization

Memory Management

Classes

Enums

Constants

Functions

Exceptions / Errors

dart_lmdb2 package

lmdb library