ctls_logging 1.0.1

High-performance columnar log storage system with adaptive tiering, compression, and intelligent query optimization for Dart/Flutter apps.

CTLS Logging

A high-performance columnar log storage system for Dart/Flutter applications with adaptive tiering, compression, and intelligent query optimization.

Features

🚀 Core Capabilities

• Columnar Time-Series Log Storage (CTLS) - Ultra-fast write performance (<1μs per entry)
• Zero-Copy Reading - Direct memory access with no deserialization overhead
• Adaptive Tiering - Automatic hot/warm/cold data movement based on access patterns
• Intelligent Query Optimization - Selectivity-based execution planning
• Cross-Platform - Pure Dart implementation, works everywhere

📦 Compression

• Run-Length Encoding (RLE) - For repetitive log levels
• Bitmap Compression - Compact boolean flag storage
• Dictionary Encoding - Deduplicated tag storage
• ZSTD Compression - Heavy compression for cold tier archives

🔍 Indexing & Search

• Bloom Filters - Fast tag existence checks (<1% false positive rate)
• Inverted Index - Full-text message search
• Parallel Search - Multi-core concurrent query processing
• Progressive Streaming - Real-time result delivery

💾 Storage Tiers

• Hot Tier - Recent logs, no compression, fast access
• Warm Tier - Moderate compression, standard access
• Cold Tier - Heavy compression, archive storage

Installation

Add to your pubspec.yaml:

dependencies:
  ctls_logging: ^1.0.0

Or for development from source:

dependencies:
  ctls_logging:
    git:
      url: https://github.com/Hiteshdon/ctls_logging.git

Quick Start

import 'package:ctls_logging/ctls_logging.dart';

// Initialize
final config = CTLSConfig(logDirectory: '/path/to/logs');
final writer = CTLSWriter(config: config);
final reader = CTLSReader();

// Write logs (< 1μs per call)
writer.write(LogEntry(
  timestamp: DateTime.now(),
  level: LogLevel.info,
  tag: 'MyApp',
  message: 'User logged in',
));

// Flush to disk
await writer.flush();

// Search logs
final criteria = SearchCriteria(
  levels: {LogLevel.error, LogLevel.warning},
  tags: {'Auth', 'Database'},
  messageContains: 'timeout',
  startTime: DateTime.now().subtract(Duration(hours: 24)),
  limit: 100,
);

final results = await reader.search(blockFiles, criteria);

🚀 Advanced Features (Optional - Most Users Won't Need These!)

Note: The basic features above are all you need for 99% of apps! These advanced features are only for power users with special requirements. Feel free to skip this section if you're just getting started.

⚡ Parallel Search

When to use: You have millions of log entries and searches are taking too long. What it does: Uses all your CPU cores at once to search faster.

final parallelSearch = ParallelSearch();
final results = await parallelSearch.search(blockFiles, criteria);

📊 Progressive Streaming

When to use: You want to show results immediately as they're found, like a live feed. What it does: Returns results one-by-one instead of waiting for everything.

final streamingSearch = StreamingSearch();
await for (final entry in streamingSearch.searchStream(blockFiles, criteria)) {
  print('${entry.timestamp}: ${entry.message}');
}

🧠 Query Optimization (Debug Tool)

When to use: You're curious about how CTLS will execute your search query. What it does: Shows you the search strategy before running it - useful for debugging slow queries.

final planner = QueryPlanner();
final plan = planner.plan(blockFiles, criteria);
print(plan.strategy); // sequential, parallel, streaming, or batched
print(plan.estimatedResultCount);

💾 Adaptive Tiering

When to use: You need custom control over when logs get compressed (most apps don't need this). What it does: Manually move logs between fast storage (hot), compressed storage (warm), and heavily compressed storage (cold).

final policy = TieringPolicy.defaultPolicy();
final tracker = AccessTracker();
final manager = TierManager(policy: policy, accessTracker: tracker);

// Evaluate and move blocks between tiers
final movements = manager.evaluateTierMovements(blocks);
await manager.processTierMovements(movements);

Architecture

CTLS Format:
┌─────────────────────────────────────┐
│ Block Header (64 bytes)             │
│ - Magic number, version, flags      │
│ - Entry count, compression type     │
│ - Column offsets, checksums         │
├─────────────────────────────────────┤
│ Timestamp Column (delta-encoded)    │
│ - Base timestamp + deltas (varint)  │
├─────────────────────────────────────┤
│ Level Column (RLE compressed)       │
│ - Run-length encoded log levels     │
├─────────────────────────────────────┤
│ Tag Column (dictionary)             │
│ - Tag IDs + string dictionary       │
├─────────────────────────────────────┤
│ Message Column (optional compress)  │
│ - Raw or GZip compressed messages   │
└─────────────────────────────────────┘

Performance

• Write: <1μs per entry (columnar buffering)
• Flush: ~50ms for 10,000 entries
• Read: Zero-copy, direct memory access
• Search: Parallel processing across cores
• Compression: 70%+ reduction for text columns
• Bloom Filter: <1% false positive rate

Example Application

Check out the included example Flutter app that demonstrates all features:

cd example
flutter run

📱 App Screenshots

Dashboard Storage efficiency & performance metrics	Search Advanced filtering with 13K+ results in 4.5s	Export Export 569K logs to TXT/JSON/CSV

The example app showcases:

• Dashboard - Storage efficiency metrics (75%+ compression, 60MB saved) and real-time write performance (64.7K logs/sec)
• Search - Advanced filtering by level, tag, date range with millisecond response times
• Export - Bulk export to TXT, JSON, or CSV formats with preview
• Log Rotation - Automatic cleanup after 30 days
• File Browser - View all log files with age tracking

Usage Details

🛠️ Basic Setup

First, let's get your logging system up and running. Think of this as setting up where your logs will live and how often they'll be saved.

import 'package:ctls_logging/ctls_logging.dart';
import 'package:path_provider/path_provider.dart';

// Get app directory
final appDir = await getApplicationDocumentsDirectory();

// Create config with log rotation (30 days retention)
final config = CTLSConfig(
  logDirectory: '${appDir.path}/logs',
  retentionDays: 30,        // Auto-delete logs older than 30 days
  bufferCapacity: 10000,    // Flush after 10K entries
  flushInterval: Duration(minutes: 15),  // Or flush every 15 min
);

// Initialize writer and reader
final writer = CTLSWriter(config: config);
final reader = CTLSReader();

✍️ Writing Logs

Writing logs is super simple - just create a LogEntry and write it. CTLS buffers everything in memory for blazing fast performance, then saves to disk automatically.

// Simple logging
writer.write(LogEntry(
  timestamp: DateTime.now(),
  level: LogLevel.info,
  tag: 'Auth',
  message: 'User login successful',
));

// Log levels: debug, info, warning, error, critical
writer.write(LogEntry(
  timestamp: DateTime.now(),
  level: LogLevel.error,
  tag: 'Database',
  message: 'Connection timeout after 30s',
));

// Automatic flush when buffer is full (10K entries)
// Or manual flush:
await writer.flush();  // Writes to disk + runs log rotation cleanup

🔍 Reading & Searching Logs

Need to find something in your logs? CTLS makes it easy to search by level, tag, message text, or date range - and it's lightning fast thanks to smart indexing!

// STEP 1: Get the reader you created earlier in Basic Setup
// (Remember: final reader = CTLSReader(); from above)

// STEP 2: Get all your log files from the log directory
final logDir = Directory(config.logDirectory);
final blockFiles = logDir
    .listSync()
    .whereType<File>()
    .where((f) => f.path.endsWith('.ctls'))  // Only .ctls files
    .toList();

// STEP 3: Search with filters - find exactly what you need!
final criteria = SearchCriteria(
  levels: {LogLevel.error, LogLevel.critical},  // Only errors & critical
  tags: {'Database', 'Network'},                // Only these components
  messageContains: 'timeout',                   // Text must include "timeout"
  startTime: DateTime.now().subtract(Duration(days: 7)),  // Last 7 days
  endTime: DateTime.now(),
  limit: 1000,                                  // Max 1000 results
);

// STEP 4: Execute search and get results
final results = await reader.search(blockFiles, criteria);
print('Found ${results.length} matching logs');

// STEP 5: Use the results!
for (final log in results) {
  print('${log.level} | ${log.tag} | ${log.message}');
}

🗑️ Log Rotation & Retention

Don't let old logs fill up your disk! CTLS automatically deletes logs older than your specified retention period. Perfect for mobile apps with limited storage.

// Configure retention in CTLSConfig
final config = CTLSConfig(
  logDirectory: '/path/to/logs',
  retentionDays: 30,  // Keep logs for 30 days only
);

// Rotation runs automatically on every flush()
await writer.flush();  // Old files deleted if > 30 days

// Disable rotation by setting retentionDays to 0
final noRotation = CTLSConfig(
  logDirectory: '/path/to/logs',
  retentionDays: 0,  // Never delete old logs
);

📤 Exporting Logs

Need to share logs with your team or analyze them in other tools? Export to TXT, JSON, or CSV formats with a single method call.

import 'package:ctls_logging/src/export/log_exporter.dart';

final exporter = LogExporter();

// Export to TXT
final txtContent = await exporter.exportToString(
  blockFiles,
  ExportOptions.txt(),
);

// Export to JSON
final jsonContent = await exporter.exportToString(
  blockFiles,
  ExportOptions.json(pretty: true),
);

// Export to CSV
final csvContent = await exporter.exportToString(
  blockFiles,
  ExportOptions.csv(),
);

// Save to file
await File('/path/to/export.json').writeAsString(jsonContent);

🐛 Query Optimization (Debug Tool - Optional)

What is this for? This is a debugging tool that shows you HOW CTLS will search your logs BEFORE actually searching. Think of it like "Explain Query" in SQL databases.

Do I need this? NO! Only use this if your searches are slow and you want to understand why.

Real Example: Let's say you have 1 million logs and search is taking 10 seconds. The query planner will tell you:

• "Using sequential scan" = Your search is slow because it's checking one file at a time
• "Using parallel scan with 8 threads" = Your search is fast because it's using all 8 CPU cores
• "Using streaming" = Your search will show results as it finds them instead of waiting

import 'package:ctls_logging/src/query/query_planner.dart';

// STEP 1: Create your search criteria (same as normal search)
final criteria = SearchCriteria(
  levels: {LogLevel.error},
  startTime: DateTime.now().subtract(Duration(days: 30)),
);

// STEP 2: Ask the planner "How would you search this?"
final planner = QueryPlanner();
final plan = planner.plan(blockFiles, criteria);

// STEP 3: See what strategy it chose
print('Strategy: ${plan.strategy}');
print('Estimated results: ${plan.estimatedResultCount}');
print('Estimated time: ${plan.estimatedDuration}');

// Example output:
// Strategy: QueryStrategy.parallel
// Estimated results: ~5000 logs
// Estimated time: 0.3 seconds

// STEP 4: If it says "sequential" but you have many files, 
// you might want to use ParallelSearch instead for better performance!

Bottom line: Skip this unless you're debugging slow searches!

API Reference

LogEntry

class LogEntry {
  final DateTime timestamp;
  final LogLevel level;     // debug, info, warning, error, critical
  final String tag;         // Component/module name
  final String message;     // Log message
}

CTLSConfig

class CTLSConfig {
  final String logDirectory;          // Where to store .ctls files
  final int retentionDays;             // Auto-delete after N days (0 = never)
  final int bufferCapacity;            // Entries before auto-flush (default: 10000)
  final Duration flushInterval;        // Time-based flush (default: 15 min)
}

SearchCriteria

class SearchCriteria {
  final Set<LogLevel>? levels;         // Filter by log levels
  final Set<String>? tags;             // Filter by tags
  final String? messageContains;       // Text search in messages
  final DateTime? startTime;           // Start of time range
  final DateTime? endTime;             // End of time range
  final int? limit;                    // Max results to return
}

License

MIT License - See LICENSE file for details.

Changelog

See CHANGELOG.md for version history.