toonizasion 1.0.1

A Dart implementation of Token-Oriented Object Notation (TOON), a compact and LLM-friendly data format.

TOON (Token-Oriented Object Notation) for Dart

Token-Oriented Object Notation is a compact, human-readable encoding of the JSON data model optimized for LLM input. This package is the official Dart implementation, providing a lossless, drop-in replacement for traditional JSON that significantly reduces token overhead.

TOON combines YAML's indentation-based structure for nested objects with a CSV-style tabular layout for uniform arrays, achieving CSV-like compactness while providing explicit structure that helps LLMs parse and validate data reliably.

Key Features

• 📊 Token Efficiency: Reduces token count by up to 40% compared to JSON.
• 🔁 Lossless Round-trips: Full fidelity conversion between Dart objects (Map, List, Primitives) and TOON.
• 🛤️ Explicit Guardrails: Uses [N] length markers and {fields} headers to improve LLM parsing accuracy.
• 📐 Minimal Syntax: Eliminates redundant braces, quotes, and commas.
• 🧺 Tabular Arrays: Collapses uniform lists of objects into streaming row values.

Installation

Add toonizasion to your pubspec.yaml:

dependencies:
  toonizasion: ^1.0.0

Then run:

dart pub get

Quick Start

Basic Usage

import 'package:toonizasion/toonizasion.dart';

void main() {
  final data = {
    'user': {
      'name': 'Alice',
      'role': 'admin',
      'active': true
    },
    'tags': ['dart', 'llm', 'compact']
  };

  // 1. Encode to TOON
  final encoded = toToon(data);
  print(encoded);
  
  // 2. Decode back to Dart
  final decoded = fromToon(encoded);
}

Tabular Arrays (The TOON Advantage)

TOON excels at uniform arrays of objects. It declares keys once and streams row values, making it extremely token-efficient for large datasets.

final data = {
  'hikes': [
    { 'id': 1, 'name': 'Blue Lake', 'km': 7.5 },
    { 'id': 2, 'name': 'Ridge Trail', 'km': 9.2 }
  ]
};

print(toToon(data));
// Output:
// hikes[2]{id,name,km}:
//   1,Blue Lake,7.5
//   2,Ridge Trail,9.2

Formatting Overview

• Objects: Represented by indentation (2 spaces) and key: value pairs.
• Arrays:
  • Simple lists: key[N]: val1,val2,val3
  • Tabular lists: key[N]{field1,field2}: followed by CSV-like rows.
• Primitives: Numbers, booleans, and strings match JSON values but with minimal quoting.

Benchmarks

TOON reaches 74% accuracy (vs JSON's 70%) while using ~40% fewer tokens in mixed-structure benchmarks across major LLM models (GPT-4, Claude 3, Gemini Pro).

License

MIT License © 2025-PRESENT