Token-Oriented Object Notation (TOON)

Token-Oriented Object Notation is a compact, human-readable format designed for passing structured data to Large Language Models with significantly reduced token usage. It's intended for LLM input, not output.

TOON's sweet spot is uniform arrays of objects – multiple fields per row, same structure across items. It borrows YAML's indentation-based structure for nested objects and CSV's tabular format for uniform data rows, then optimizes both for token efficiency in LLM contexts. For deeply nested or non-uniform data, JSON may be more efficient.

Why TOON?

AI is becoming cheaper and more accessible, but larger context windows allow for larger data inputs as well. LLM tokens still cost money – and standard JSON is verbose and token-expensive:

{
  "users": [
    { "id": 1, "name": "Alice", "role": "admin" },
    { "id": 2, "name": "Bob", "role": "user" }
  ]
}

TOON conveys the same information with fewer tokens:

users[2]{id,name,role}:
  1,Alice,admin
  2,Bob,user

Key Features

• 💸 Token-efficient: typically 30–60% fewer tokens than JSON
• 🤿 LLM-friendly guardrails: explicit lengths and field lists help models validate output
• 🍱 Minimal syntax: removes redundant punctuation (braces, brackets, most quotes)
• 📐 Indentation-based structure: replaces braces with whitespace for better readability
• 🧺 Tabular arrays: declare keys once, then stream rows without repetition

Benchmarks

The benchmarks test datasets that favor TOON's strengths (uniform tabular data). Real-world performance depends heavily on your data structure. For info about benchmarking https://github.com/johannschopplich/toon

Token Efficiency

⭐ GitHub Repositories       ██████████████░░░░░░░░░░░    8,745 tokens
                             vs JSON (-42.3%)           15,145
                             vs JSON compact (-23.7%)   11,455
                             vs YAML (-33.4%)           13,129
                             vs XML (-48.8%)            17,095

📈 Daily Analytics           ██████████░░░░░░░░░░░░░░░    4,507 tokens
                             vs JSON (-58.9%)           10,977
                             vs JSON compact (-35.7%)    7,013
                             vs YAML (-48.8%)            8,810
                             vs XML (-65.7%)            13,128

🛒 E-Commerce Order          ████████████████░░░░░░░░░      166 tokens
                             vs JSON (-35.4%)              257
                             vs JSON compact (-2.9%)       171
                             vs YAML (-15.7%)              197
                             vs XML (-38.7%)               271

─────────────────────────────────────────────────────────────────────
Total                        ██████████████░░░░░░░░░░░   13,418 tokens
                             vs JSON (-49.1%)           26,379
                             vs JSON compact (-28.0%)   18,639
                             vs YAML (-39.4%)           22,136
                             vs XML (-56.0%)            30,494

{
  "repositories": [
    {
      "id": 28457823,
      "name": "freeCodeCamp",
      "repo": "freeCodeCamp/freeCodeCamp",
      "description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…",
      "createdAt": "2014-12-24T17:49:19Z",
      "updatedAt": "2025-10-28T11:58:08Z",
      "pushedAt": "2025-10-28T10:17:16Z",
      "stars": 430886,
      "watchers": 8583,
      "forks": 42146,
      "defaultBranch": "main"
    },
    {
      "id": 132750724,
      "name": "build-your-own-x",
      "repo": "codecrafters-io/build-your-own-x",
      "description": "Master programming by recreating your favorite technologies from scratch.",
      "createdAt": "2018-05-09T12:03:18Z",
      "updatedAt": "2025-10-28T12:37:11Z",
      "pushedAt": "2025-10-10T18:45:01Z",
      "stars": 430877,
      "watchers": 6332,
      "forks": 40453,
      "defaultBranch": "master"
    },
    {
      "id": 21737465,
      "name": "awesome",
      "repo": "sindresorhus/awesome",
      "description": "😎 Awesome lists about all kinds of interesting topics",
      "createdAt": "2014-07-11T13:42:37Z",
      "updatedAt": "2025-10-28T12:40:21Z",
      "pushedAt": "2025-10-27T17:57:31Z",
      "stars": 410052,
      "watchers": 8017,
      "forks": 32029,
      "defaultBranch": "main"
    }
  ]
}

repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
  28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…","2014-12-24T17:49:19Z","2025-10-28T11:58:08Z","2025-10-28T10:17:16Z",430886,8583,42146,main
  132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-28T12:37:11Z","2025-10-10T18:45:01Z",430877,6332,40453,master
  21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-28T12:40:21Z","2025-10-27T17:57:31Z",410052,8017,32029,main

{
  "metrics": [
    {
      "date": "2025-01-01",
      "views": 6890,
      "clicks": 401,
      "conversions": 23,
      "revenue": 6015.59,
      "bounceRate": 0.63
    },
    {
      "date": "2025-01-02",
      "views": 6940,
      "clicks": 323,
      "conversions": 37,
      "revenue": 9086.44,
      "bounceRate": 0.36
    },
    {
      "date": "2025-01-03",
      "views": 4390,
      "clicks": 346,
      "conversions": 26,
      "revenue": 6360.75,
      "bounceRate": 0.48
    },
    {
      "date": "2025-01-04",
      "views": 3429,
      "clicks": 231,
      "conversions": 13,
      "revenue": 2360.96,
      "bounceRate": 0.65
    },
    {
      "date": "2025-01-05",
      "views": 5804,
      "clicks": 186,
      "conversions": 22,
      "revenue": 2535.96,
      "bounceRate": 0.37
    }
  ]
}

metrics[5]{date,views,clicks,conversions,revenue,bounceRate}:
  2025-01-01,6890,401,23,6015.59,0.63
  2025-01-02,6940,323,37,9086.44,0.36
  2025-01-03,4390,346,26,6360.75,0.48
  2025-01-04,3429,231,13,2360.96,0.65
  2025-01-05,5804,186,22,2535.96,0.37

Retrieval Accuracy

Accuracy across 4 LLMs on 154 data retrieval questions:

gpt-5-nano
→ TOON           ███████████████████░    96.1% (148/154)
  CSV            ██████████████████░░    91.6% (141/154)
  YAML           ██████████████████░░    91.6% (141/154)
  JSON compact   ██████████████████░░    91.6% (141/154)
  XML            █████████████████░░░    87.0% (134/154)
  JSON           █████████████████░░░    86.4% (133/154)

claude-haiku-4-5-20251001
  JSON           ██████████░░░░░░░░░░    50.0% (77/154)
  YAML           ██████████░░░░░░░░░░    49.4% (76/154)
→ TOON           ██████████░░░░░░░░░░    48.7% (75/154)
  XML            ██████████░░░░░░░░░░    48.1% (74/154)
  CSV            █████████░░░░░░░░░░░    47.4% (73/154)
  JSON compact   █████████░░░░░░░░░░░    44.2% (68/154)

gemini-2.5-flash
  CSV            ██████████████████░░    87.7% (135/154)
  XML            ██████████████████░░    87.7% (135/154)
→ TOON           █████████████████░░░    86.4% (133/154)
  YAML           ████████████████░░░░    79.9% (123/154)
  JSON compact   ████████████████░░░░    79.9% (123/154)
  JSON           ███████████████░░░░░    76.6% (118/154)

grok-4-fast-non-reasoning
→ TOON           ██████████░░░░░░░░░░    49.4% (76/154)
  JSON           ██████████░░░░░░░░░░    48.7% (75/154)
  XML            █████████░░░░░░░░░░░    46.1% (71/154)
  YAML           █████████░░░░░░░░░░░    46.1% (71/154)
  JSON compact   █████████░░░░░░░░░░░    45.5% (70/154)
  CSV            █████████░░░░░░░░░░░    44.2% (68/154)

Key tradeoff: TOON achieves 70.1% accuracy (vs JSON's 65.4%) while using 46.3% fewer tokens on these datasets.

Installation

Add this to your pubspec.yaml:

dependencies:
  toon: ^1.0.0

Then run:

dart pub get

CLI

Command-line tool for converting between JSON and TOON formats.

Usage

dart run toon <command> [input]

Commands

Examples

# Encode JSON from stdin
echo '{"name": "Alice", "age": 30}' | dart run toon encode

# Decode TOON from stdin
echo "name: Alice\nage: 30" | dart run toon decode

# Process files (auto-detect based on extension)
dart run toon data.json    # .json → encode to TOON
dart run toon data.toon    # .toon → decode to JSON

Quick Start

import 'package:toon_dart/toon_dart.dart';

void main() {
  final data = {
    'user': {
      'id': 123,
      'name': 'Ada',
      'tags': ['reading', 'gaming'],
      'active': true,
      'preferences': []
    }
  };

  print(toonEncode(data));
}

Output:

user:
  id: 123
  name: Ada
  tags[2]: reading,gaming
  active: true
  preferences[0]:

Format Overview

Objects

Simple objects with primitive values:

toonEncode({
  'id': 123,
  'name': 'Ada',
  'active': true
});

id: 123
name: Ada
active: true

Nested objects:

toonEncode({
  'user': {
    'id': 123,
    'name': 'Ada'
  }
});

user:
  id: 123
  name: Ada

Arrays

Primitive Arrays (Inline)

toonEncode({
  'tags': ['admin', 'ops', 'dev']
});

tags[3]: admin,ops,dev

Arrays of Objects (Tabular)

When all objects share the same primitive fields, TOON uses an efficient tabular format:

toonEncode({
  'items': [
    {'sku': 'A1', 'qty': 2, 'price': 9.99},
    {'sku': 'B2', 'qty': 1, 'price': 14.5}
  ]
});

items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.5

Tabular formatting applies recursively: nested arrays of objects (whether as object properties or inside list items) also use tabular format if they meet the same requirements.

toonEncode({
  'items': [
    {
      'users': [
        {'id': 1, 'name': 'Ada'},
        {'id': 2, 'name': 'Bob'}
      ],
      'status': 'active'
    }
  ]
});

items[1]:
  - users[2]{id,name}:
    1,Ada
    2,Bob
    status: active

Mixed and Non-Uniform Arrays

Arrays that don't meet the tabular requirements use list format:

items[3]:
  - 1
  - a: 1
  - text

When objects appear in list format, the first field is placed on the hyphen line:

items[2]:
  - id: 1
    name: First
  - id: 2
    name: Second
    extra: true

Arrays of Arrays

When you have arrays containing primitive inner arrays:

toonEncode({
  'pairs': [
    [1, 2],
    [3, 4]
  ]
});

pairs[2]:
  - [2]: 1,2
  - [2]: 3,4

Empty Arrays and Objects

Empty containers have special representations:

toonEncode({'items': []}); // items[0]:
toonEncode([]); // [0]:
toonEncode({}); // (empty output)
toonEncode({'config': {}}); // config:

Quoting Rules

TOON quotes strings only when necessary to maximize token efficiency:

• Inner spaces are allowed; leading or trailing spaces force quotes.
• Unicode and emoji are safe unquoted.
• Quotes and control characters are escaped with backslash.

Object Keys and Field Names

Keys are unquoted if they match the identifier pattern: start with a letter or underscore, followed by letters, digits, underscores, or dots (e.g., id, userName, user_name, user.name, _private). All other keys must be quoted (e.g., "user name", "order-id", "123", "order:id", "").

String Values

String values are quoted when any of the following is true:

Examples of unquoted strings: Unicode and emoji are safe (hello 👋 world), as are strings with inner spaces (hello world).

Type Conversions

Some non-JSON types are automatically normalized for LLM-safe output:

API

toonEncode(value, {options})

Converts any JSON-serializable value to TOON format.

Parameters:

• value – Any JSON-serializable value (object, array, primitive, or nested structure). Non-JSON-serializable values (non-finite numbers) are converted to null. Dates are converted to ISO strings, and BigInts are emitted as decimal integers (no quotes).
• options – Optional encoding options:
  • indent – Number of spaces per indentation level (default: 2)
  • delimiter – Delimiter for array values and tabular rows (default: ',')
  • lengthMarker – Optional marker to prefix array lengths (default: false)

Returns:

A TOON-formatted string with no trailing newline or spaces.

Example:

import 'package:toon_dart/toon_dart.dart';

final items = [
  {'sku': 'A1', 'qty': 2, 'price': 9.99},
  {'sku': 'B2', 'qty': 1, 'price': 14.5}
];

toonEncode({'items': items});

Output:

items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.5

Delimiter Options

The delimiter option allows you to choose between comma (default), tab, or pipe delimiters for array values and tabular rows. Alternative delimiters can provide additional token savings in specific contexts.

Tab Delimiter (\t)

Using tab delimiters instead of commas can reduce token count further, especially for tabular data:

final data = {
  'items': [
    {'sku': 'A1', 'name': 'Widget', 'qty': 2, 'price': 9.99},
    {'sku': 'B2', 'name': 'Gadget', 'qty': 1, 'price': 14.5}
  ]
};

toonEncode(data, options: EncodeOptions(delimiter: '\t'));

Output:

items[2	]{sku	name	qty	price}:
  A1	Widget	2	9.99
  B2	Gadget	1	14.5

Benefits:

• Tabs are single characters and often tokenize more efficiently than commas.
• Tabs rarely appear in natural text, reducing the need for quote-escaping.
• The delimiter is explicitly encoded in the array header, making it self-descriptive.

Considerations:

• Some terminals and editors may collapse or expand tabs visually.
• String values containing tabs will still require quoting.

Pipe Delimiter (|)

Pipe delimiters offer a middle ground between commas and tabs:

toonEncode(data, options: EncodeOptions(delimiter: '|'));

Output:

items[2|]{sku|name|qty|price}:
  A1|Widget|2|9.99
  B2|Gadget|1|14.5

Length Marker Option

The lengthMarker option adds an optional hash (#) prefix to array lengths to emphasize that the bracketed value represents a count, not an index:

final data = {
  'tags': ['reading', 'gaming', 'coding'],
  'items': [
    {'sku': 'A1', 'qty': 2, 'price': 9.99},
    {'sku': 'B2', 'qty': 1, 'price': 14.5},
  ],
};

toonEncode(data, options: EncodeOptions(lengthMarker: '#'));
// tags[#3]: reading,gaming,coding
// items[#2]{sku,qty,price}:
//   A1,2,9.99
//   B2,1,14.5

// Works with custom delimiters
toonEncode(data, options: EncodeOptions(lengthMarker: '#', delimiter: '|'));
// tags[#3|]: reading|gaming|coding
// items[#2|]{sku|qty|price}:
//   A1|2|9.99
//   B2|1|14.5

toonDecode(input, {options})

Converts a TOON-formatted string back to Dart values.

Parameters:

• input – A TOON-formatted string to parse
• options – Optional decoding options:
  • indent – Expected number of spaces per indentation level (default: 2)
  • strict – Enable strict validation (default: true)

Returns:

A Dart value (object, array, or primitive) representing the parsed TOON data.

Example:

import 'package:toon_dart/toon_dart.dart';

final toon = '''
items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.5
''';

final data = toonDecode(toon);
// {
//   'items': [
//     {'sku': 'A1', 'qty': 2, 'price': 9.99},
//     {'sku': 'B2', 'qty': 1, 'price': 14.5}
//   ]
// }

Strict Mode:

By default, the decoder validates input strictly:

• Invalid escape sequences: Throws on "\x", unterminated strings.
• Syntax errors: Throws on missing colons, malformed headers.
• Array length mismatches: Throws when declared length doesn't match actual count.
• Delimiter mismatches: Throws when row delimiters don't match header.

Notes and Limitations

• Format familiarity and structure matter as much as token count. TOON's tabular format requires arrays of objects with identical keys and primitive values only. When this doesn't hold (due to mixed types, non-uniform objects, or nested structures), TOON switches to list format where JSON can be more efficient at scale.
  • TOON excels at: Uniform arrays of objects (same fields, primitive values), especially large datasets with consistent structure.
  • JSON is better for: Non-uniform data, deeply nested structures, and objects with varying field sets.
• Token counts vary by tokenizer and model. Benchmarks use a GPT-style tokenizer (cl100k/o200k); actual savings will differ with other models (e.g., SentencePiece).
• TOON is designed for LLM input where human readability and token efficiency matter. It's not a drop-in replacement for JSON in APIs or storage.

Using TOON in LLM Prompts

TOON works best when you show the format instead of describing it. The structure is self-documenting – models parse it naturally once they see the pattern.

Sending TOON to LLMs (Input)

Wrap your encoded data in a fenced code block (label it ```toon for clarity). The indentation and headers are usually enough – models treat it like familiar YAML or CSV. The explicit length markers ([N]) and field headers ({field1,field2}) help the model track structure, especially for large tables.

Generating TOON from LLMs (Output)

For output, be more explicit. When you want the model to generate TOON:

• Show the expected header (users[N]{id,name,role}:). The model fills rows instead of repeating keys, reducing generation errors.
• State the rules: 2-space indent, no trailing spaces, [N] matches row count.

Here's a prompt that works for both reading and generating:

Data is in TOON format (2-space indent, arrays show length and fields).

```toon
users[3]{id,name,role,lastLogin}:
  1,Alice,admin,2025-01-15T10:30:00Z
  2,Bob,user,2025-01-14T15:22:00Z
  3,Charlie,user,2025-01-13T09:45:00Z
```

Task: Return only users with role "user" as TOON. Use the same header. Set [N] to match the row count. Output only the code block.

Quick Reference

// Object
{ id: 1, name: 'Ada' }          → id: 1
                                  name: Ada

// Nested object
{ user: { id: 1 } }             → user:
                                    id: 1

// Primitive array (inline)
{ tags: ['foo', 'bar'] }        → tags[2]: foo,bar

// Tabular array (uniform objects)
{ items: [                      → items[2]{id,qty}:
  { id: 1, qty: 5 },                1,5
  { id: 2, qty: 3 }                 2,3
]}

// Mixed / non-uniform (list)
{ items: [1, { a: 1 }, 'x'] }   → items[3]:
                                    - 1
                                    - a: 1
                                    - x

// Array of arrays
{ pairs: [[1, 2], [3, 4]] }     → pairs[2]:
                                    - [2]: 1,2
                                    - [2]: 3,4

// Root array
['x', 'y']                      → [2]: x,y

// Empty containers
{}                              → (empty output)
{ items: [] }                   → items[0]:

// Special quoting
{ note: 'hello, world' }        → note: "hello, world"
{ items: ['true', true] }       → items[2]: "true",true

Ports in Other Languages

• Typescript toon (orginal)
• Dart: toon (this package)
• Elixir: toon_ex
• PHP: toon-php
• Python: python-toon or pytoon
• Ruby: toon-ruby
• Java: JToon
• .NET: toon.NET
• Swift: TOONEncoder
• Go: gotoon
• Rust: toon-rs

Special Thanks to Johann Schopplich see his repo

https://github.com/johannschopplich/toon

License

MIT License © 2025-PRESENT Wisam idris