chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,586 @@
|
||||
---
|
||||
description: Retrieval accuracy and token efficiency results for TOON across mixed-structure and flat-only tracks.
|
||||
---
|
||||
|
||||
# Benchmarks
|
||||
|
||||
The benchmarks on this page measure TOON's performance across two key dimensions:
|
||||
|
||||
- **Retrieval Accuracy**: How well LLMs understand and extract information from different input formats.
|
||||
- **Token Efficiency**: How many tokens each format requires to represent the same data.
|
||||
|
||||
Benchmarks are organized into two tracks to ensure fair comparisons:
|
||||
|
||||
- **Mixed-Structure Track**: Datasets with nested or semi-uniform structures (TOON vs JSON, YAML, XML). CSV excluded as it cannot properly represent these structures.
|
||||
- **Flat-Only Track**: Datasets with flat tabular structures where CSV is applicable (CSV vs TOON vs JSON, YAML, XML).
|
||||
|
||||
## Retrieval Accuracy
|
||||
|
||||
<!-- automd:file src="../../benchmarks/results/retrieval-accuracy.md" -->
|
||||
|
||||
Benchmarks test LLM comprehension across different input formats using 209 data retrieval questions on 4 models.
|
||||
|
||||
<details>
|
||||
<summary><strong>Show Dataset Catalog</strong></summary>
|
||||
|
||||
#### Dataset Catalog
|
||||
|
||||
| Dataset | Rows | Structure | CSV Support | Eligibility |
|
||||
| ------- | ---- | --------- | ----------- | ----------- |
|
||||
| Uniform employee records | 100 | uniform | ✓ | 100% |
|
||||
| E-commerce orders with nested structures | 50 | nested | ✗ | 33% |
|
||||
| Time-series analytics data | 60 | uniform | ✓ | 100% |
|
||||
| Top 100 GitHub repositories | 100 | uniform | ✓ | 100% |
|
||||
| Semi-uniform event logs | 75 | semi-uniform | ✗ | 50% |
|
||||
| Deeply nested configuration | 11 | deep | ✗ | 0% |
|
||||
| Valid complete dataset (control) | 20 | uniform | ✓ | 100% |
|
||||
| Array truncated: 3 rows removed from end | 17 | uniform | ✓ | 100% |
|
||||
| Extra rows added beyond declared length | 23 | uniform | ✓ | 100% |
|
||||
| Inconsistent field count (missing salary in row 10) | 20 | uniform | ✓ | 100% |
|
||||
| Missing required fields (no email in multiple rows) | 20 | uniform | ✓ | 100% |
|
||||
|
||||
**Structure classes:**
|
||||
- **uniform**: All objects have identical fields with primitive values
|
||||
- **semi-uniform**: Mix of uniform and non-uniform structures
|
||||
- **nested**: Objects with nested structures (nested objects or arrays)
|
||||
- **deep**: Highly nested with minimal tabular eligibility
|
||||
|
||||
**CSV Support:** ✓ (supported), ✗ (not supported – would require lossy flattening)
|
||||
|
||||
**Eligibility:** Percentage of arrays that qualify for TOON's tabular format (uniform objects with primitive values)
|
||||
|
||||
</details>
|
||||
|
||||
#### Efficiency Ranking (Accuracy per 1K Tokens)
|
||||
|
||||
Each format ranked by efficiency (accuracy percentage per 1,000 tokens):
|
||||
|
||||
```
|
||||
TOON ████████████████████ 27.7 acc%/1K tok │ 76.4% acc │ 2,759 tokens
|
||||
JSON compact █████████████████░░░ 23.7 acc%/1K tok │ 73.7% acc │ 3,104 tokens
|
||||
YAML ██████████████░░░░░░ 19.9 acc%/1K tok │ 74.5% acc │ 3,749 tokens
|
||||
JSON ████████████░░░░░░░░ 16.4 acc%/1K tok │ 75.0% acc │ 4,587 tokens
|
||||
XML ██████████░░░░░░░░░░ 13.8 acc%/1K tok │ 72.1% acc │ 5,221 tokens
|
||||
```
|
||||
|
||||
*Efficiency score = (Accuracy % ÷ Tokens) × 1,000. Higher is better.*
|
||||
|
||||
> [!TIP]
|
||||
> TOON achieves **76.4%** accuracy (vs JSON's 75.0%) while using **39.9% fewer tokens**.
|
||||
|
||||
**Note on CSV:** Excluded from ranking as it only supports 109 of 209 questions (flat tabular data only). While CSV is highly token-efficient for simple tabular data, it cannot represent nested structures that other formats handle.
|
||||
|
||||
#### Per-Model Accuracy
|
||||
|
||||
Accuracy across 4 LLMs on 209 data retrieval questions:
|
||||
|
||||
```
|
||||
claude-haiku-4-5-20251001
|
||||
→ TOON ████████████░░░░░░░░ 59.8% (125/209)
|
||||
JSON ███████████░░░░░░░░░ 57.4% (120/209)
|
||||
YAML ███████████░░░░░░░░░ 56.0% (117/209)
|
||||
XML ███████████░░░░░░░░░ 55.5% (116/209)
|
||||
JSON compact ███████████░░░░░░░░░ 55.0% (115/209)
|
||||
CSV ██████████░░░░░░░░░░ 50.5% (55/109)
|
||||
|
||||
gemini-3-flash-preview
|
||||
XML ████████████████████ 98.1% (205/209)
|
||||
JSON ███████████████████░ 97.1% (203/209)
|
||||
YAML ███████████████████░ 97.1% (203/209)
|
||||
→ TOON ███████████████████░ 96.7% (202/209)
|
||||
JSON compact ███████████████████░ 96.7% (202/209)
|
||||
CSV ███████████████████░ 96.3% (105/109)
|
||||
|
||||
gpt-5-nano
|
||||
→ TOON ██████████████████░░ 90.9% (190/209)
|
||||
JSON compact ██████████████████░░ 90.9% (190/209)
|
||||
JSON ██████████████████░░ 89.0% (186/209)
|
||||
CSV ██████████████████░░ 89.0% (97/109)
|
||||
YAML █████████████████░░░ 87.1% (182/209)
|
||||
XML ████████████████░░░░ 80.9% (169/209)
|
||||
|
||||
grok-4-1-fast-non-reasoning
|
||||
→ TOON ████████████░░░░░░░░ 58.4% (122/209)
|
||||
YAML ████████████░░░░░░░░ 57.9% (121/209)
|
||||
JSON ███████████░░░░░░░░░ 56.5% (118/209)
|
||||
XML ███████████░░░░░░░░░ 54.1% (113/209)
|
||||
JSON compact ██████████░░░░░░░░░░ 52.2% (109/209)
|
||||
CSV ██████████░░░░░░░░░░ 51.4% (56/109)
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> TOON achieves **76.4% accuracy** (vs JSON's 75.0%) while using **39.9% fewer tokens** on these datasets.
|
||||
|
||||
<details>
|
||||
<summary><strong>Performance by dataset, model, and question type</strong></summary>
|
||||
|
||||
#### Performance by Question Type
|
||||
|
||||
| Question Type | TOON | JSON | YAML | JSON compact | XML | CSV |
|
||||
| ------------- | ---- | ---- | ---- | ---- | ---- | ---- |
|
||||
| Field Retrieval | 99.6% | 99.3% | 98.5% | 98.5% | 98.9% | 100.0% |
|
||||
| Aggregation | 61.9% | 61.9% | 59.9% | 58.3% | 54.4% | 50.9% |
|
||||
| Filtering | 56.8% | 53.1% | 56.3% | 55.2% | 51.6% | 50.9% |
|
||||
| Structure Awareness | 89.0% | 87.0% | 84.0% | 84.0% | 81.0% | 85.9% |
|
||||
| Structural Validation | 70.0% | 60.0% | 60.0% | 55.0% | 85.0% | 80.0% |
|
||||
|
||||
#### Performance by Dataset
|
||||
|
||||
##### Uniform employee records
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `csv` | 73.2% | 2,334 | 120/164 |
|
||||
| `toon` | 73.2% | 2,498 | 120/164 |
|
||||
| `json-compact` | 73.8% | 3,924 | 121/164 |
|
||||
| `yaml` | 73.8% | 4,959 | 121/164 |
|
||||
| `json-pretty` | 73.8% | 6,331 | 121/164 |
|
||||
| `xml` | 74.4% | 7,296 | 122/164 |
|
||||
|
||||
##### E-commerce orders with nested structures
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `toon` | 82.3% | 7,458 | 135/164 |
|
||||
| `json-compact` | 78.7% | 7,110 | 129/164 |
|
||||
| `yaml` | 79.9% | 8,755 | 131/164 |
|
||||
| `json-pretty` | 79.3% | 11,234 | 130/164 |
|
||||
| `xml` | 77.4% | 12,649 | 127/164 |
|
||||
|
||||
##### Time-series analytics data
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `csv` | 75.0% | 1,411 | 90/120 |
|
||||
| `toon` | 78.3% | 1,553 | 94/120 |
|
||||
| `json-compact` | 74.2% | 2,354 | 89/120 |
|
||||
| `yaml` | 75.8% | 2,954 | 91/120 |
|
||||
| `json-pretty` | 75.0% | 3,681 | 90/120 |
|
||||
| `xml` | 72.5% | 4,389 | 87/120 |
|
||||
|
||||
##### Top 100 GitHub repositories
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `csv` | 65.9% | 8,527 | 87/132 |
|
||||
| `toon` | 66.7% | 8,779 | 88/132 |
|
||||
| `yaml` | 65.2% | 13,141 | 86/132 |
|
||||
| `json-compact` | 59.8% | 11,464 | 79/132 |
|
||||
| `json-pretty` | 63.6% | 15,157 | 84/132 |
|
||||
| `xml` | 56.1% | 17,105 | 74/132 |
|
||||
|
||||
##### Semi-uniform event logs
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `json-compact` | 68.3% | 4,839 | 82/120 |
|
||||
| `toon` | 65.0% | 5,819 | 78/120 |
|
||||
| `json-pretty` | 69.2% | 6,817 | 83/120 |
|
||||
| `yaml` | 61.7% | 5,847 | 74/120 |
|
||||
| `xml` | 58.3% | 7,729 | 70/120 |
|
||||
|
||||
##### Deeply nested configuration
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `json-compact` | 90.5% | 568 | 105/116 |
|
||||
| `toon` | 94.8% | 655 | 110/116 |
|
||||
| `yaml` | 93.1% | 675 | 108/116 |
|
||||
| `json-pretty` | 92.2% | 924 | 107/116 |
|
||||
| `xml` | 91.4% | 1,013 | 106/116 |
|
||||
|
||||
##### Valid complete dataset (control)
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `toon` | 100.0% | 535 | 4/4 |
|
||||
| `json-compact` | 100.0% | 787 | 4/4 |
|
||||
| `yaml` | 100.0% | 992 | 4/4 |
|
||||
| `json-pretty` | 100.0% | 1,274 | 4/4 |
|
||||
| `xml` | 25.0% | 1,462 | 1/4 |
|
||||
| `csv` | 0.0% | 483 | 0/4 |
|
||||
|
||||
##### Array truncated: 3 rows removed from end
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `csv` | 100.0% | 413 | 4/4 |
|
||||
| `xml` | 100.0% | 1,243 | 4/4 |
|
||||
| `toon` | 0.0% | 462 | 0/4 |
|
||||
| `json-pretty` | 0.0% | 1,085 | 0/4 |
|
||||
| `yaml` | 0.0% | 843 | 0/4 |
|
||||
| `json-compact` | 0.0% | 670 | 0/4 |
|
||||
|
||||
##### Extra rows added beyond declared length
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `csv` | 100.0% | 550 | 4/4 |
|
||||
| `toon` | 75.0% | 605 | 3/4 |
|
||||
| `json-compact` | 75.0% | 901 | 3/4 |
|
||||
| `xml` | 100.0% | 1,678 | 4/4 |
|
||||
| `yaml` | 75.0% | 1,138 | 3/4 |
|
||||
| `json-pretty` | 50.0% | 1,460 | 2/4 |
|
||||
|
||||
##### Inconsistent field count (missing salary in row 10)
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `csv` | 100.0% | 480 | 4/4 |
|
||||
| `json-compact` | 100.0% | 782 | 4/4 |
|
||||
| `yaml` | 100.0% | 985 | 4/4 |
|
||||
| `toon` | 100.0% | 1,008 | 4/4 |
|
||||
| `json-pretty` | 100.0% | 1,266 | 4/4 |
|
||||
| `xml` | 100.0% | 1,453 | 4/4 |
|
||||
|
||||
##### Missing required fields (no email in multiple rows)
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `csv` | 100.0% | 340 | 4/4 |
|
||||
| `xml` | 100.0% | 1,409 | 4/4 |
|
||||
| `toon` | 75.0% | 974 | 3/4 |
|
||||
| `json-pretty` | 50.0% | 1,225 | 2/4 |
|
||||
| `yaml` | 25.0% | 951 | 1/4 |
|
||||
| `json-compact` | 0.0% | 750 | 0/4 |
|
||||
|
||||
#### Performance by Model
|
||||
|
||||
##### claude-haiku-4-5-20251001
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
| ------ | -------- | ------------- |
|
||||
| `toon` | 59.8% | 125/209 |
|
||||
| `json-pretty` | 57.4% | 120/209 |
|
||||
| `yaml` | 56.0% | 117/209 |
|
||||
| `xml` | 55.5% | 116/209 |
|
||||
| `json-compact` | 55.0% | 115/209 |
|
||||
| `csv` | 50.5% | 55/109 |
|
||||
|
||||
##### gemini-3-flash-preview
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
| ------ | -------- | ------------- |
|
||||
| `xml` | 98.1% | 205/209 |
|
||||
| `json-pretty` | 97.1% | 203/209 |
|
||||
| `yaml` | 97.1% | 203/209 |
|
||||
| `toon` | 96.7% | 202/209 |
|
||||
| `json-compact` | 96.7% | 202/209 |
|
||||
| `csv` | 96.3% | 105/109 |
|
||||
|
||||
##### gpt-5-nano
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
| ------ | -------- | ------------- |
|
||||
| `toon` | 90.9% | 190/209 |
|
||||
| `json-compact` | 90.9% | 190/209 |
|
||||
| `json-pretty` | 89.0% | 186/209 |
|
||||
| `csv` | 89.0% | 97/109 |
|
||||
| `yaml` | 87.1% | 182/209 |
|
||||
| `xml` | 80.9% | 169/209 |
|
||||
|
||||
##### grok-4-1-fast-non-reasoning
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
| ------ | -------- | ------------- |
|
||||
| `toon` | 58.4% | 122/209 |
|
||||
| `yaml` | 57.9% | 121/209 |
|
||||
| `json-pretty` | 56.5% | 118/209 |
|
||||
| `xml` | 54.1% | 113/209 |
|
||||
| `json-compact` | 52.2% | 109/209 |
|
||||
| `csv` | 51.4% | 56/109 |
|
||||
|
||||
</details>
|
||||
|
||||
#### What's Being Measured
|
||||
|
||||
This benchmark tests **LLM comprehension and data retrieval accuracy** across different input formats. Each LLM receives formatted data and must answer questions about it. This does **not** test the model's ability to generate TOON output – only to read and understand it.
|
||||
|
||||
#### Datasets Tested
|
||||
|
||||
Eleven datasets designed to test different structural patterns and validation capabilities:
|
||||
|
||||
**Primary datasets:**
|
||||
|
||||
1. **Tabular** (100 employee records): Uniform objects with identical fields – optimal for TOON's tabular format.
|
||||
2. **Nested** (50 e-commerce orders): Complex structures with nested customer objects and item arrays.
|
||||
3. **Analytics** (60 days of metrics): Time-series data with dates and numeric values.
|
||||
4. **GitHub** (100 repositories): Real-world data from top GitHub repos by stars.
|
||||
5. **Event Logs** (75 logs): Semi-uniform data with ~50% flat logs and ~50% with nested error objects.
|
||||
6. **Nested Config** (1 configuration): Deeply nested configuration with minimal tabular eligibility.
|
||||
|
||||
**Structural validation datasets:**
|
||||
|
||||
7. **Control**: Valid complete dataset (baseline for validation)
|
||||
8. **Truncated**: Array with 3 rows removed from end (tests `[N]` length detection)
|
||||
9. **Extra rows**: Array with 3 additional rows beyond declared length
|
||||
10. **Width mismatch**: Inconsistent field count (missing salary in row 10)
|
||||
11. **Missing fields**: Systematic field omissions (no email in multiple rows)
|
||||
|
||||
#### Question Types
|
||||
|
||||
209 questions are generated dynamically across five categories:
|
||||
|
||||
- **Field retrieval (33%)**: Direct value lookups or values that can be read straight off a record (including booleans and simple counts such as array lengths)
|
||||
- Example: "What is Alice's salary?" → `75000`
|
||||
- Example: "How many items are in order ORD-0042?" → `3`
|
||||
- Example: "What is the customer name for order ORD-0042?" → `John Doe`
|
||||
|
||||
- **Aggregation (30%)**: Dataset-level totals and averages plus single-condition filters (counts, sums, min/max comparisons)
|
||||
- Example: "How many employees work in Engineering?" → `17`
|
||||
- Example: "What is the total revenue across all orders?" → `45123.50`
|
||||
- Example: "How many employees have salary > 80000?" → `23`
|
||||
|
||||
- **Filtering (23%)**: Multi-condition queries requiring compound logic (AND constraints across fields)
|
||||
- Example: "How many employees in Sales have salary > 80000?" → `5`
|
||||
- Example: "How many active employees have more than 10 years of experience?" → `8`
|
||||
|
||||
- **Structure awareness (12%)**: Tests format-native structural affordances (TOON's `[N]` count and `{fields}`, CSV's header row)
|
||||
- Example: "How many employees are in the dataset?" → `100`
|
||||
- Example: "List the field names for employees" → `id, name, email, department, salary, yearsExperience, active`
|
||||
- Example: "What is the department of the last employee?" → `Sales`
|
||||
|
||||
- **Structural validation (2%)**: Tests ability to detect incomplete, truncated, or corrupted data using structural metadata
|
||||
- Example: "Is this data complete and valid?" → `YES` (control dataset) or `NO` (corrupted datasets)
|
||||
- Tests TOON's `[N]` length validation and `{fields}` consistency checking
|
||||
- Demonstrates CSV's lack of structural validation capabilities
|
||||
|
||||
#### Evaluation Process
|
||||
|
||||
1. **Format conversion**: Each dataset is converted to all 6 formats (TOON, JSON, YAML, JSON compact, XML, CSV).
|
||||
2. **Query LLM**: Each model receives formatted data + question in a prompt and extracts the answer.
|
||||
3. **Validate deterministically**: Answers are validated using type-aware comparison (e.g., `50000` = `$50,000`, `Engineering` = `engineering`, `2025-01-01` = `January 1, 2025`) without requiring an LLM judge.
|
||||
|
||||
#### Models & Configuration
|
||||
|
||||
- **Models tested**: `claude-haiku-4-5-20251001`, `gemini-3-flash-preview`, `gpt-5-nano`, `grok-4-1-fast-non-reasoning`
|
||||
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding (GPT-5 tokenizer)
|
||||
- **Temperature**: Not set (models use their defaults)
|
||||
- **Total evaluations**: 209 questions × 6 formats × 4 models = 5,016 LLM calls
|
||||
|
||||
<!-- /automd -->
|
||||
|
||||
## Token Efficiency
|
||||
|
||||
Token counts are measured using the GPT-5 `o200k_base` tokenizer via [`gpt-tokenizer`](https://github.com/niieani/gpt-tokenizer). Savings are calculated against formatted JSON (2-space indentation) as the primary baseline, with additional comparisons to compact JSON (minified), YAML, and XML. Actual savings vary by model and tokenizer.
|
||||
|
||||
The benchmarks test datasets across different structural patterns (uniform, semi-uniform, nested, deeply nested) to show where TOON excels and where other formats may be better.
|
||||
|
||||
<!-- automd:file src="../../benchmarks/results/token-efficiency.md" -->
|
||||
|
||||
#### Mixed-Structure Track
|
||||
|
||||
Datasets with nested or semi-uniform structures. CSV excluded as it cannot properly represent these structures.
|
||||
|
||||
```
|
||||
🛒 E-commerce orders with nested structures ┊ Tabular: 33%
|
||||
│
|
||||
TOON █████████████░░░░░░░ 73,126 tokens
|
||||
├─ vs JSON (−33.3%) 109,599 tokens
|
||||
├─ vs JSON compact (+5.3%) 69,459 tokens
|
||||
├─ vs YAML (−14.4%) 85,415 tokens
|
||||
└─ vs XML (−40.7%) 123,344 tokens
|
||||
|
||||
🧾 Semi-uniform event logs ┊ Tabular: 50%
|
||||
│
|
||||
TOON █████████████████░░░ 154,084 tokens
|
||||
├─ vs JSON (−15.0%) 181,201 tokens
|
||||
├─ vs JSON compact (+19.9%) 128,529 tokens
|
||||
├─ vs YAML (−0.8%) 155,397 tokens
|
||||
└─ vs XML (−25.2%) 205,859 tokens
|
||||
|
||||
🧩 Deeply nested configuration ┊ Tabular: 0%
|
||||
│
|
||||
TOON ██████████████░░░░░░ 620 tokens
|
||||
├─ vs JSON (−31.9%) 911 tokens
|
||||
├─ vs JSON compact (+11.1%) 558 tokens
|
||||
├─ vs YAML (−6.3%) 662 tokens
|
||||
└─ vs XML (−38.2%) 1,003 tokens
|
||||
|
||||
──────────────────────────────────── Total ────────────────────────────────────
|
||||
TOON ████████████████░░░░ 227,830 tokens
|
||||
├─ vs JSON (−21.9%) 291,711 tokens
|
||||
├─ vs JSON compact (+14.7%) 198,546 tokens
|
||||
├─ vs YAML (−5.7%) 241,474 tokens
|
||||
└─ vs XML (−31.0%) 330,206 tokens
|
||||
```
|
||||
|
||||
#### Flat-Only Track
|
||||
|
||||
Datasets with flat tabular structures where CSV is applicable.
|
||||
|
||||
```
|
||||
👥 Uniform employee records ┊ Tabular: 100%
|
||||
│
|
||||
CSV ███████████████████░ 47,102 tokens
|
||||
TOON ████████████████████ 49,919 tokens (+6.0% vs CSV)
|
||||
├─ vs JSON (−60.7%) 127,063 tokens
|
||||
├─ vs JSON compact (−36.9%) 79,059 tokens
|
||||
├─ vs YAML (−50.1%) 100,011 tokens
|
||||
└─ vs XML (−65.9%) 146,579 tokens
|
||||
|
||||
📈 Time-series analytics data ┊ Tabular: 100%
|
||||
│
|
||||
CSV ██████████████████░░ 8,383 tokens
|
||||
TOON ████████████████████ 9,115 tokens (+8.7% vs CSV)
|
||||
├─ vs JSON (−59.0%) 22,245 tokens
|
||||
├─ vs JSON compact (−35.9%) 14,211 tokens
|
||||
├─ vs YAML (−49.0%) 17,858 tokens
|
||||
└─ vs XML (−65.8%) 26,616 tokens
|
||||
|
||||
⭐ Top 100 GitHub repositories ┊ Tabular: 100%
|
||||
│
|
||||
CSV ███████████████████░ 8,512 tokens
|
||||
TOON ████████████████████ 8,744 tokens (+2.7% vs CSV)
|
||||
├─ vs JSON (−42.3%) 15,144 tokens
|
||||
├─ vs JSON compact (−23.7%) 11,454 tokens
|
||||
├─ vs YAML (−33.4%) 13,128 tokens
|
||||
└─ vs XML (−48.9%) 17,095 tokens
|
||||
|
||||
──────────────────────────────────── Total ────────────────────────────────────
|
||||
CSV ███████████████████░ 63,997 tokens
|
||||
TOON ████████████████████ 67,778 tokens (+5.9% vs CSV)
|
||||
├─ vs JSON (−58.8%) 164,452 tokens
|
||||
├─ vs JSON compact (−35.3%) 104,724 tokens
|
||||
├─ vs YAML (−48.3%) 130,997 tokens
|
||||
└─ vs XML (−64.4%) 190,290 tokens
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary><strong>Show detailed examples</strong></summary>
|
||||
|
||||
#### 📈 Time-series analytics data
|
||||
|
||||
**Savings:** 13,130 tokens (59.0% reduction vs JSON)
|
||||
|
||||
**JSON** (22,245 tokens):
|
||||
|
||||
```json
|
||||
{
|
||||
"metrics": [
|
||||
{
|
||||
"date": "2025-01-01",
|
||||
"views": 6138,
|
||||
"clicks": 174,
|
||||
"conversions": 12,
|
||||
"revenue": 2712.49,
|
||||
"bounceRate": 0.35
|
||||
},
|
||||
{
|
||||
"date": "2025-01-02",
|
||||
"views": 4616,
|
||||
"clicks": 274,
|
||||
"conversions": 34,
|
||||
"revenue": 9156.29,
|
||||
"bounceRate": 0.56
|
||||
},
|
||||
{
|
||||
"date": "2025-01-03",
|
||||
"views": 4460,
|
||||
"clicks": 143,
|
||||
"conversions": 8,
|
||||
"revenue": 1317.98,
|
||||
"bounceRate": 0.59
|
||||
},
|
||||
{
|
||||
"date": "2025-01-04",
|
||||
"views": 4740,
|
||||
"clicks": 125,
|
||||
"conversions": 13,
|
||||
"revenue": 2934.77,
|
||||
"bounceRate": 0.37
|
||||
},
|
||||
{
|
||||
"date": "2025-01-05",
|
||||
"views": 6428,
|
||||
"clicks": 369,
|
||||
"conversions": 19,
|
||||
"revenue": 1317.24,
|
||||
"bounceRate": 0.3
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**TOON** (9,115 tokens):
|
||||
|
||||
```
|
||||
metrics[5]{date,views,clicks,conversions,revenue,bounceRate}:
|
||||
2025-01-01,6138,174,12,2712.49,0.35
|
||||
2025-01-02,4616,274,34,9156.29,0.56
|
||||
2025-01-03,4460,143,8,1317.98,0.59
|
||||
2025-01-04,4740,125,13,2934.77,0.37
|
||||
2025-01-05,6428,369,19,1317.24,0.3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### ⭐ Top 100 GitHub repositories
|
||||
|
||||
**Savings:** 6,400 tokens (42.3% reduction vs JSON)
|
||||
|
||||
**JSON** (15,144 tokens):
|
||||
|
||||
```json
|
||||
{
|
||||
"repositories": [
|
||||
{
|
||||
"id": 28457823,
|
||||
"name": "freeCodeCamp",
|
||||
"repo": "freeCodeCamp/freeCodeCamp",
|
||||
"description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…",
|
||||
"createdAt": "2014-12-24T17:49:19Z",
|
||||
"updatedAt": "2025-10-28T11:58:08Z",
|
||||
"pushedAt": "2025-10-28T10:17:16Z",
|
||||
"stars": 430886,
|
||||
"watchers": 8583,
|
||||
"forks": 42146,
|
||||
"defaultBranch": "main"
|
||||
},
|
||||
{
|
||||
"id": 132750724,
|
||||
"name": "build-your-own-x",
|
||||
"repo": "codecrafters-io/build-your-own-x",
|
||||
"description": "Master programming by recreating your favorite technologies from scratch.",
|
||||
"createdAt": "2018-05-09T12:03:18Z",
|
||||
"updatedAt": "2025-10-28T12:37:11Z",
|
||||
"pushedAt": "2025-10-10T18:45:01Z",
|
||||
"stars": 430877,
|
||||
"watchers": 6332,
|
||||
"forks": 40453,
|
||||
"defaultBranch": "master"
|
||||
},
|
||||
{
|
||||
"id": 21737465,
|
||||
"name": "awesome",
|
||||
"repo": "sindresorhus/awesome",
|
||||
"description": "😎 Awesome lists about all kinds of interesting topics",
|
||||
"createdAt": "2014-07-11T13:42:37Z",
|
||||
"updatedAt": "2025-10-28T12:40:21Z",
|
||||
"pushedAt": "2025-10-27T17:57:31Z",
|
||||
"stars": 410052,
|
||||
"watchers": 8017,
|
||||
"forks": 32029,
|
||||
"defaultBranch": "main"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**TOON** (8,744 tokens):
|
||||
|
||||
```
|
||||
repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
|
||||
28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…","2014-12-24T17:49:19Z","2025-10-28T11:58:08Z","2025-10-28T10:17:16Z",430886,8583,42146,main
|
||||
132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-28T12:37:11Z","2025-10-10T18:45:01Z",430877,6332,40453,master
|
||||
21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-28T12:40:21Z","2025-10-27T17:57:31Z",410052,8017,32029,main
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<!-- /automd -->
|
||||
|
||||
## Related Resources
|
||||
|
||||
- [Formal Byte-Level Model](/reference/efficiency-formalization) – Mathematical analysis of byte efficiency compared to JSON
|
||||
- [Specification](/reference/spec) – Formal TOON specification
|
||||
@@ -0,0 +1,365 @@
|
||||
---
|
||||
description: TOON syntax with concrete examples – objects, arrays, headers, key folding, and quoting rules.
|
||||
---
|
||||
|
||||
# Format Overview
|
||||
|
||||
TOON syntax reference with concrete examples. See [Getting Started](/guide/getting-started) for an introduction.
|
||||
|
||||
## Data Model
|
||||
|
||||
TOON models data the same way as JSON:
|
||||
|
||||
- **Primitives**: strings, numbers, booleans, and `null`
|
||||
- **Objects**: mappings from string keys to values
|
||||
- **Arrays**: ordered sequences of values
|
||||
|
||||
### Root Forms
|
||||
|
||||
A TOON document can represent different root forms:
|
||||
|
||||
- **Root object** (most common): Fields appear at depth 0 with no parent key
|
||||
- **Root array**: Begins with `[N]:` or `[N]{fields}:` at depth 0
|
||||
- **Root primitive**: A single primitive value (string, number, boolean, or null)
|
||||
|
||||
Most examples in these docs use root objects, but the format supports all three forms equally ([spec §5](https://github.com/toon-format/spec/blob/main/SPEC.md#5-concrete-syntax-and-root-form)).
|
||||
|
||||
## Objects
|
||||
|
||||
### Simple Objects
|
||||
|
||||
Objects with primitive values use `key: value` syntax, with one field per line:
|
||||
|
||||
```yaml
|
||||
id: 123
|
||||
name: Ada
|
||||
active: true
|
||||
```
|
||||
|
||||
Indentation replaces braces. One space follows the colon.
|
||||
|
||||
### Nested Objects
|
||||
|
||||
Nested objects add one indentation level (default: 2 spaces):
|
||||
|
||||
```yaml
|
||||
user:
|
||||
id: 123
|
||||
name: Ada
|
||||
```
|
||||
|
||||
When a key ends with `:` and has no value on the same line, it opens a nested object. All lines at the next indentation level belong to that object.
|
||||
|
||||
### Empty Objects
|
||||
|
||||
An empty object at the root yields an empty document (no lines). A nested empty object is `key:` alone, with no children.
|
||||
|
||||
## Arrays
|
||||
|
||||
TOON detects array structure and chooses the most efficient representation. Arrays always declare their length in brackets: `[N]`.
|
||||
|
||||
### Primitive Arrays (Inline)
|
||||
|
||||
Arrays of primitives (strings, numbers, booleans, null) are rendered inline:
|
||||
|
||||
```yaml
|
||||
tags[3]: admin,ops,dev
|
||||
```
|
||||
|
||||
The delimiter (comma by default) separates values. Strings containing the active delimiter must be quoted.
|
||||
|
||||
### Arrays of Objects (Tabular)
|
||||
|
||||
When all objects in an array share the same set of primitive-valued keys, TOON uses tabular format:
|
||||
|
||||
::: code-group
|
||||
|
||||
```yaml [Basic Tabular]
|
||||
items[2]{sku,qty,price}:
|
||||
A1,2,9.99
|
||||
B2,1,14.5
|
||||
```
|
||||
|
||||
```yaml [With Spaces in Values]
|
||||
users[2]{id,name,role}:
|
||||
1,Alice Admin,admin
|
||||
2,"Bob Smith",user
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
The header `items[2]{sku,qty,price}:` declares:
|
||||
- **Array length**: `[2]` means 2 rows
|
||||
- **Field names**: `{sku,qty,price}` defines the columns
|
||||
- **Active delimiter**: comma (default)
|
||||
|
||||
Each row contains values in the same order as the field list. Values are encoded as primitives (strings, numbers, booleans, null) and separated by the delimiter.
|
||||
|
||||
> [!NOTE]
|
||||
> Tabular format requires identical field sets across all objects (same keys, order per object may vary), primitive values only (no nested arrays/objects), and at least one key per object – arrays that contain an empty `{}` element fall back to the expanded list form below.
|
||||
|
||||
### Mixed and Non-Uniform Arrays
|
||||
|
||||
Arrays that don't meet the tabular requirements use list format with hyphen markers:
|
||||
|
||||
```yaml
|
||||
items[3]:
|
||||
- 1
|
||||
- a: 1
|
||||
- text
|
||||
```
|
||||
|
||||
Each element starts with `- ` at one indentation level deeper than the parent array header.
|
||||
|
||||
### Objects as List Items
|
||||
|
||||
When an array element is an object, it appears as a list item:
|
||||
|
||||
```yaml
|
||||
items[2]:
|
||||
- id: 1
|
||||
name: First
|
||||
- id: 2
|
||||
name: Second
|
||||
extra: true
|
||||
```
|
||||
|
||||
When a tabular array is the first field of a list-item object, the tabular header appears on the hyphen line, with rows indented two levels deeper and other fields indented one level deeper:
|
||||
|
||||
```yaml
|
||||
items[1]:
|
||||
- users[2]{id,name}:
|
||||
1,Ada
|
||||
2,Bob
|
||||
status: active
|
||||
```
|
||||
|
||||
When the object has only a single tabular field, the same pattern applies:
|
||||
|
||||
```yaml
|
||||
items[1]:
|
||||
- users[2]{id,name}:
|
||||
1,Ada
|
||||
2,Bob
|
||||
```
|
||||
|
||||
This is the canonical encoding for list-item objects whose first field is a tabular array.
|
||||
|
||||
### Arrays of Arrays
|
||||
|
||||
When you have arrays containing primitive inner arrays:
|
||||
|
||||
```yaml
|
||||
pairs[2]:
|
||||
- [2]: 1,2
|
||||
- [2]: 3,4
|
||||
```
|
||||
|
||||
Each inner array gets its own header on the list-item line.
|
||||
|
||||
When the inner arrays are themselves arrays of objects or non-uniform arrays, the same `- [N]:` header appears on the hyphen line and the nested items follow one indent deeper:
|
||||
|
||||
```yaml
|
||||
items[3]:
|
||||
- summary
|
||||
- id: 1
|
||||
name: Ada
|
||||
- [2]:
|
||||
- id: 2
|
||||
- status: draft
|
||||
```
|
||||
|
||||
### Empty Arrays
|
||||
|
||||
Empty arrays render as `key: []` for fields and `[]` at the root:
|
||||
|
||||
```yaml
|
||||
items: []
|
||||
```
|
||||
|
||||
The legacy `items[0]:` form is still decoded for backward compatibility.
|
||||
|
||||
## Array Headers
|
||||
|
||||
### Header Syntax
|
||||
|
||||
Array headers follow this pattern:
|
||||
|
||||
```
|
||||
key[N<delimiter?>]<{fields}>:
|
||||
```
|
||||
|
||||
Where:
|
||||
- **N** is the non-negative integer length
|
||||
- **delimiter** (optional) explicitly declares the active delimiter:
|
||||
- Absent → comma (`,`)
|
||||
- `\t` (tab character) → tab delimiter
|
||||
- `|` → pipe delimiter
|
||||
- **fields** (optional) for tabular arrays: `{field1,field2,field3}`
|
||||
|
||||
> [!NOTE]
|
||||
> The array length `[N]` helps LLMs validate structure. If you ask a model to generate TOON output, explicit lengths let you detect truncation or malformed data.
|
||||
|
||||
### Delimiter Options
|
||||
|
||||
TOON supports three delimiters: comma (default), tab, and pipe. The delimiter is scoped to the array header that declares it.
|
||||
|
||||
::: code-group
|
||||
|
||||
```yaml [Comma (default)]
|
||||
items[2]{sku,name,qty,price}:
|
||||
A1,Widget,2,9.99
|
||||
B2,Gadget,1,14.5
|
||||
```
|
||||
|
||||
```yaml [Tab]
|
||||
items[2 ]{sku name qty price}:
|
||||
A1 Widget 2 9.99
|
||||
B2 Gadget 1 14.5
|
||||
```
|
||||
|
||||
```yaml [Pipe]
|
||||
items[2|]{sku|name|qty|price}:
|
||||
A1|Widget|2|9.99
|
||||
B2|Gadget|1|14.5
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
Tab and pipe delimiters are explicitly encoded in the header brackets and field braces. Inside an array scope, only the active delimiter triggers quoting – the others are literal data. Object field values (`key: value`) follow the document delimiter (§11.1) regardless of any surrounding array's active delimiter.
|
||||
|
||||
> [!TIP]
|
||||
> Tab delimiters often tokenize more efficiently than commas, especially for data with few quoted strings. Use `encode(data, { delimiter: '\t' })` for additional token savings.
|
||||
|
||||
## Key Folding (Optional)
|
||||
|
||||
Key folding is an optional encoder feature (since spec v1.5) that collapses chains of single-key objects into dotted paths, reducing tokens for deeply nested data.
|
||||
|
||||
### Basic Folding
|
||||
|
||||
Standard nesting:
|
||||
|
||||
```yaml
|
||||
data:
|
||||
metadata:
|
||||
items[2]: a,b
|
||||
```
|
||||
|
||||
With key folding (`keyFolding: 'safe'`):
|
||||
|
||||
```yaml
|
||||
data.metadata.items[2]: a,b
|
||||
```
|
||||
|
||||
The three nested objects collapse into a single dotted key `data.metadata.items`.
|
||||
|
||||
### When Folding Applies
|
||||
|
||||
A chain of objects is foldable when:
|
||||
- Each object in the chain has exactly one key (leading to the next object or a leaf value)
|
||||
- The leaf value is a primitive, array, or empty object
|
||||
- All segments are valid identifier segments (letters, digits, underscores only; no dots within segments)
|
||||
- The resulting folded key doesn't collide with existing keys
|
||||
|
||||
::: details Advanced Folding Rules
|
||||
**Segment Requirements (safe mode):**
|
||||
- All folded segments must match `^[A-Za-z_][A-Za-z0-9_]*$` (no dots, hyphens, or other special characters)
|
||||
- No segment may require quoting per §7.3 of the spec
|
||||
- The resulting folded key must not equal any existing sibling literal key at the same depth (collision avoidance)
|
||||
|
||||
**Depth Limit:**
|
||||
- The `flattenDepth` option (default: `Infinity`) controls how many segments to fold
|
||||
- `flattenDepth: 2` folds only two-segment chains: `{a: {b: val}}` → `a.b: val`
|
||||
- Values less than 2 have no practical effect
|
||||
|
||||
**Round-Trip with Path Expansion:**
|
||||
To reconstruct the original structure when decoding, use `expandPaths: 'safe'`. This splits dotted keys back into nested objects using the same safety rules ([spec §13.4](https://github.com/toon-format/spec/blob/main/SPEC.md#134-key-folding-and-path-expansion)).
|
||||
:::
|
||||
|
||||
### Round-Trip with Path Expansion
|
||||
|
||||
When decoding TOON that used key folding, enable path expansion to restore the nested structure:
|
||||
|
||||
```ts
|
||||
import { decode, encode } from '@toon-format/toon'
|
||||
|
||||
const original = { data: { metadata: { items: ['a', 'b'] } } }
|
||||
|
||||
// Encode with folding
|
||||
const toon = encode(original, { keyFolding: 'safe' })
|
||||
// → "data.metadata.items[2]: a,b"
|
||||
|
||||
// Decode with expansion
|
||||
const restored = decode(toon, { expandPaths: 'safe' })
|
||||
// → { data: { metadata: { items: ['a', 'b'] } } }
|
||||
```
|
||||
|
||||
Path expansion is off by default, so dotted keys are treated as literal keys unless explicitly enabled.
|
||||
|
||||
## Quoting and Types
|
||||
|
||||
### When Strings Need Quotes
|
||||
|
||||
TOON quotes strings **only when necessary** to maximize token efficiency. A string must be quoted if:
|
||||
|
||||
- It's empty (`""`)
|
||||
- It has leading or trailing whitespace
|
||||
- It equals `true`, `false`, or `null` (case-sensitive)
|
||||
- It looks like a number (e.g., `"42"`, `"-3.14"`, `"1e-6"`, `"05"`)
|
||||
- It contains special characters: colon (`:`), quote (`"`), backslash (`\`), brackets, braces, or any control character in U+0000–U+001F
|
||||
- It contains the relevant delimiter (the active delimiter inside an array scope, or the document delimiter elsewhere)
|
||||
- It equals `"-"` or starts with `"-"` followed by any character
|
||||
|
||||
Otherwise, strings can be unquoted. Unicode, emoji, and strings with internal (non-leading/trailing) spaces are safe unquoted:
|
||||
|
||||
```yaml
|
||||
message: Hello 世界 👋
|
||||
note: This has inner spaces
|
||||
```
|
||||
|
||||
### Escape Sequences
|
||||
|
||||
In quoted strings and keys, six escape sequences are valid:
|
||||
|
||||
| Character | Escape |
|
||||
|-----------|--------|
|
||||
| Backslash (`\`) | `\\` |
|
||||
| Double quote (`"`) | `\"` |
|
||||
| Newline (U+000A) | `\n` |
|
||||
| Carriage return (U+000D) | `\r` |
|
||||
| Tab (U+0009) | `\t` |
|
||||
| Any other U+0000–U+001F control character | `\uXXXX` |
|
||||
|
||||
Other escapes (e.g., `\x`, `\0`, `\b`) are always rejected, as are lone-surrogate `\uXXXX` values (U+D800–U+DFFF).
|
||||
|
||||
### Type Conversions
|
||||
|
||||
Numbers are emitted in canonical decimal form for values in the §2 carve-out range; exponent notation is permitted outside. Non-JSON types (`NaN`, `Infinity`, `BigInt`, `Date`, `Set`, `Map`, `undefined`, etc.) are normalized before encoding – see [API Reference – Type Normalization](/reference/api#type-normalization) for the full mapping.
|
||||
|
||||
Decoders accept both decimal and exponent forms on input (e.g., `42`, `-3.14`, `1e-6`), and treat tokens with forbidden leading zeros (e.g., `"05"`) as strings, not numbers.
|
||||
|
||||
### Custom Serialization with toJSON
|
||||
|
||||
Objects with a `toJSON()` method are serialized by calling the method and normalizing its result before encoding, similar to `JSON.stringify`:
|
||||
|
||||
```ts
|
||||
const obj = {
|
||||
data: 'example',
|
||||
toJSON() {
|
||||
return { info: this.data }
|
||||
}
|
||||
}
|
||||
|
||||
encode(obj)
|
||||
// info: example
|
||||
```
|
||||
|
||||
The `toJSON()` method:
|
||||
|
||||
- Takes precedence over built-in normalization (Date, Array, Set, Map)
|
||||
- Results are recursively normalized
|
||||
- Is called for objects with `toJSON` in their prototype chain
|
||||
|
||||
---
|
||||
|
||||
For complete rules on quoting, escaping, type conversions, and strict-mode decoding, see [spec §2–4 (data model), §7 (strings and keys), and §14 (strict mode)](https://github.com/toon-format/spec/blob/main/SPEC.md).
|
||||
@@ -0,0 +1,248 @@
|
||||
---
|
||||
description: What TOON is, when to use it, and a first encode/decode example with the TypeScript library.
|
||||
---
|
||||
|
||||
# Getting Started
|
||||
|
||||
## What Is TOON?
|
||||
|
||||
**Token-Oriented Object Notation** is a compact, human-readable encoding of the JSON data model that minimizes tokens and makes structure easy for models to follow. It is intended for *LLM input* as a drop-in, lossless representation of your existing JSON.
|
||||
|
||||
TOON combines YAML's indentation-based structure for nested objects with a CSV-style tabular layout for uniform arrays. TOON's sweet spot is uniform arrays of objects (multiple fields per row, same structure across items), achieving CSV-like compactness while adding explicit structure that helps LLMs parse and validate data reliably.
|
||||
|
||||
Think of it as a translation layer: use JSON programmatically, and encode it as TOON for LLM input.
|
||||
|
||||
### Why TOON?
|
||||
|
||||
Standard JSON is verbose and token-expensive. For uniform arrays of objects, JSON repeats every field name for every record:
|
||||
|
||||
```json
|
||||
{
|
||||
"users": [
|
||||
{ "id": 1, "name": "Alice", "role": "admin" },
|
||||
{ "id": 2, "name": "Bob", "role": "user" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
YAML already reduces some redundancy with indentation instead of braces:
|
||||
|
||||
```yaml
|
||||
users:
|
||||
- id: 1
|
||||
name: Alice
|
||||
role: admin
|
||||
- id: 2
|
||||
name: Bob
|
||||
role: user
|
||||
```
|
||||
|
||||
TOON goes further by declaring fields once and streaming data as rows:
|
||||
|
||||
```yaml
|
||||
users[2]{id,name,role}:
|
||||
1,Alice,admin
|
||||
2,Bob,user
|
||||
```
|
||||
|
||||
The `[2]` declares the array length, letting LLMs answer dataset-size questions and detect truncation. The `{id,name,role}` declares the field names. Each row is a compact, comma-separated list of values. The pattern is the same throughout TOON: declare structure once, stream data compactly. The result lands close to CSV density with explicit structure preserved.
|
||||
|
||||
For a more realistic example, here's how TOON handles a dataset with both nested objects and tabular arrays:
|
||||
|
||||
::: code-group
|
||||
|
||||
```json [JSON (235 tokens)]
|
||||
{
|
||||
"context": {
|
||||
"task": "Our favorite hikes together",
|
||||
"location": "Boulder",
|
||||
"season": "spring_2025"
|
||||
},
|
||||
"friends": ["ana", "luis", "sam"],
|
||||
"hikes": [
|
||||
{
|
||||
"id": 1,
|
||||
"name": "Blue Lake Trail",
|
||||
"distanceKm": 7.5,
|
||||
"elevationGain": 320,
|
||||
"companion": "ana",
|
||||
"wasSunny": true
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"name": "Ridge Overlook",
|
||||
"distanceKm": 9.2,
|
||||
"elevationGain": 540,
|
||||
"companion": "luis",
|
||||
"wasSunny": false
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"name": "Wildflower Loop",
|
||||
"distanceKm": 5.1,
|
||||
"elevationGain": 180,
|
||||
"companion": "sam",
|
||||
"wasSunny": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
```yaml [TOON (106 tokens)]
|
||||
context:
|
||||
task: Our favorite hikes together
|
||||
location: Boulder
|
||||
season: spring_2025
|
||||
friends[3]: ana,luis,sam
|
||||
hikes[3]{id,name,distanceKm,elevationGain,companion,wasSunny}:
|
||||
1,Blue Lake Trail,7.5,320,ana,true
|
||||
2,Ridge Overlook,9.2,540,luis,false
|
||||
3,Wildflower Loop,5.1,180,sam,true
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
Notice how TOON combines YAML's indentation for the `context` object with inline format for the primitive `friends` array and tabular format for the structured `hikes` array. Each format is chosen automatically based on the data structure.
|
||||
|
||||
### Design Goals
|
||||
|
||||
TOON is optimized for specific use cases. It aims to:
|
||||
|
||||
- Make uniform arrays of objects as compact as possible by declaring structure once and streaming data.
|
||||
- Stay fully lossless and deterministic – round-trips preserve all data and structure.
|
||||
- Keep parsing simple and robust for both LLMs and humans through explicit structure markers.
|
||||
- Provide validation guardrails (array lengths, field counts) that help detect truncation and malformed output.
|
||||
|
||||
## When to Use TOON
|
||||
|
||||
TOON excels with uniform arrays of objects – data with the same structure across items. For LLM prompts, the format produces deterministic, minimally quoted text with built-in validation. Explicit array lengths (`[N]`) and field headers (`{fields}`) help detect truncation and malformed data, while the tabular structure declares fields once rather than repeating them in every row.
|
||||
|
||||
::: tip
|
||||
The TOON format is stable, but also an idea in progress. Nothing's set in stone – help shape where it goes by contributing to the [spec](https://github.com/toon-format/spec) or sharing feedback.
|
||||
:::
|
||||
|
||||
## When Not to Use TOON
|
||||
|
||||
TOON is not always the best choice. Consider alternatives when:
|
||||
|
||||
- **Deeply nested or non-uniform structures** (tabular eligibility ≈ 0%): JSON-compact often uses fewer tokens. Example: complex configuration objects with many nested levels.
|
||||
- **Semi-uniform arrays** (~40–60% tabular eligibility): Token savings diminish. Prefer JSON if your pipelines already rely on it.
|
||||
- **Pure tabular data**: CSV is smaller than TOON for flat tables. TOON adds minimal overhead (~5–10%) to provide structure (array length declarations, field headers, delimiter scoping) that improves LLM reliability.
|
||||
- **Latency-critical applications**: Benchmark on your exact setup. Some deployments (especially local/quantized models) may process compact JSON faster despite TOON's lower token count.
|
||||
|
||||
::: info
|
||||
For data-driven comparisons across different structures, see [Benchmarks](/guide/benchmarks). When optimizing for latency, measure TTFT, tokens/sec, and total time for both TOON and JSON-compact, and use whichever is faster in your specific environment.
|
||||
:::
|
||||
|
||||
## Installation
|
||||
|
||||
### TypeScript Library
|
||||
|
||||
Install the library via your preferred package manager:
|
||||
|
||||
::: code-group
|
||||
|
||||
```bash [npm]
|
||||
npm install @toon-format/toon
|
||||
```
|
||||
|
||||
```bash [pnpm]
|
||||
pnpm add @toon-format/toon
|
||||
```
|
||||
|
||||
```bash [yarn]
|
||||
yarn add @toon-format/toon
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
### CLI
|
||||
|
||||
The CLI can be used without installation via `npx`, or installed globally:
|
||||
|
||||
::: code-group
|
||||
|
||||
```bash [npx (no install)]
|
||||
npx @toon-format/cli input.json -o output.toon
|
||||
```
|
||||
|
||||
```bash [npm]
|
||||
npm install -g @toon-format/cli
|
||||
```
|
||||
|
||||
```bash [pnpm]
|
||||
pnpm add -g @toon-format/cli
|
||||
```
|
||||
|
||||
```bash [yarn]
|
||||
yarn global add @toon-format/cli
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
For full CLI documentation, see the [CLI reference](/cli/).
|
||||
|
||||
## Media Type & File Extension
|
||||
|
||||
TOON files conventionally use the `.toon` extension. For HTTP transmission, the provisional media type is `text/toon`, always with UTF-8 encoding. While you may specify `charset=utf-8` explicitly, it's optional – UTF-8 is the default assumption. This follows the registration process outlined in [spec §17](https://github.com/toon-format/spec/blob/main/SPEC.md#17-iana-considerations).
|
||||
|
||||
## Your First Example
|
||||
|
||||
The examples below use the TypeScript library for demonstration, but the same operations work in any language with a TOON implementation.
|
||||
|
||||
Let's encode a simple dataset with the TypeScript library:
|
||||
|
||||
```ts
|
||||
import { encode } from '@toon-format/toon'
|
||||
|
||||
const data = {
|
||||
users: [
|
||||
{ id: 1, name: 'Alice', role: 'admin' },
|
||||
{ id: 2, name: 'Bob', role: 'user' }
|
||||
]
|
||||
}
|
||||
|
||||
console.log(encode(data))
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```yaml
|
||||
users[2]{id,name,role}:
|
||||
1,Alice,admin
|
||||
2,Bob,user
|
||||
```
|
||||
|
||||
### Decoding Back to JSON
|
||||
|
||||
Decoding is just as simple:
|
||||
|
||||
```ts
|
||||
import { decode } from '@toon-format/toon'
|
||||
|
||||
const toon = `
|
||||
users[2]{id,name,role}:
|
||||
1,Alice,admin
|
||||
2,Bob,user
|
||||
`
|
||||
|
||||
const data = decode(toon)
|
||||
console.log(JSON.stringify(data, null, 2))
|
||||
```
|
||||
|
||||
**Output:**
|
||||
|
||||
```json
|
||||
{
|
||||
"users": [
|
||||
{ "id": 1, "name": "Alice", "role": "admin" },
|
||||
{ "id": 2, "name": "Bob", "role": "user" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Round-tripping is lossless: `decode(encode(x))` always equals `x` (after normalization of non-JSON types like `Date`, `NaN`, etc.).
|
||||
|
||||
## Where to Go Next
|
||||
|
||||
Now that you've seen your first TOON document, read the [Format Overview](/guide/format-overview) for complete syntax details (objects, arrays, quoting rules, key folding), then explore [Using TOON with LLMs](/guide/llm-prompts) to see how to use it effectively in prompts. For implementation details, check the [API Reference](/reference/api) (TypeScript) or the [Specification](/reference/spec) (language-agnostic normative rules).
|
||||
@@ -0,0 +1,191 @@
|
||||
---
|
||||
description: Prompting strategies for sending TOON to LLMs and validating TOON they generate, with examples.
|
||||
---
|
||||
|
||||
# Using TOON with LLMs
|
||||
|
||||
TOON is designed for passing structured data to Large Language Models with reduced token costs and improved reliability. This guide shows how to use TOON effectively in prompts, both for input (sending data to models) and output (getting models to generate TOON).
|
||||
|
||||
This guide is about the TOON format itself. Code examples use the TypeScript library for demonstration, but the same patterns and techniques apply regardless of which programming language you're using.
|
||||
|
||||
## Why TOON for LLMs
|
||||
|
||||
LLM tokens cost money, and JSON is verbose – repeating every field name for every record in an array. TOON minimizes tokens especially for uniform arrays by declaring fields once and streaming data as rows, typically saving 30–60% compared to formatted JSON.
|
||||
|
||||
TOON adds structure guardrails: explicit `[N]` lengths and `{fields}` headers make it easier for models to track rows and for you to validate output. Strict mode helps detect truncation and malformed TOON when decoding model responses.
|
||||
|
||||
## Sending TOON as Input
|
||||
|
||||
TOON works best when you show the format instead of describing it. The structure is self-documenting – models parse it naturally once they see the pattern.
|
||||
|
||||
Wrap your encoded data in a fenced code block (label it ` ```toon` for clarity):
|
||||
|
||||
````md
|
||||
Data is in TOON format (2-space indent, arrays show length and fields).
|
||||
|
||||
```toon
|
||||
users[3]{id,name,role,lastLogin}:
|
||||
1,Alice,admin,"2025-01-15T10:30:00Z"
|
||||
2,Bob,user,"2025-01-14T15:22:00Z"
|
||||
3,Charlie,user,"2025-01-13T09:45:00Z"
|
||||
```
|
||||
|
||||
Task: Summarize the user roles and their last activity.
|
||||
````
|
||||
|
||||
The indentation and headers are usually enough – models treat TOON like familiar YAML or CSV. The explicit array lengths (`[N]`) and field headers (`{fields}`) help the model track structure, especially for large tables.
|
||||
|
||||
> [!NOTE]
|
||||
> Most models don't have built-in TOON syntax highlighting, so ` ```toon` or ` ```yaml` both work fine. The structure is what matters.
|
||||
|
||||
## Generating TOON from LLMs
|
||||
|
||||
For output, be more explicit. When you want the model to **generate** TOON:
|
||||
|
||||
- **Show the expected header** (e.g., `users[N]{id,name,role}:`). The model fills rows instead of repeating keys, reducing generation errors.
|
||||
- **State the rules**: 2-space indent, no trailing spaces, `[N]` matches row count.
|
||||
|
||||
Here's a prompt that works for both reading and generating:
|
||||
|
||||
````md
|
||||
Data is in TOON format (2-space indent, arrays show length and fields).
|
||||
|
||||
```toon
|
||||
users[3]{id,name,role,lastLogin}:
|
||||
1,Alice,admin,"2025-01-15T10:30:00Z"
|
||||
2,Bob,user,"2025-01-14T15:22:00Z"
|
||||
3,Charlie,user,"2025-01-13T09:45:00Z"
|
||||
```
|
||||
|
||||
Task: Return only users with role "user" as TOON. Use the same header format. Set [N] to match the row count. Output only the code block.
|
||||
````
|
||||
|
||||
**Expected output:**
|
||||
|
||||
```toon
|
||||
users[2]{id,name,role,lastLogin}:
|
||||
2,Bob,user,"2025-01-14T15:22:00Z"
|
||||
3,Charlie,user,"2025-01-13T09:45:00Z"
|
||||
```
|
||||
|
||||
The model adjusts `[N]` to `2` and generates two rows.
|
||||
|
||||
### Validation with Strict Mode
|
||||
|
||||
When decoding model-generated TOON, use strict mode (default) to catch errors:
|
||||
|
||||
```ts
|
||||
import { decode } from '@toon-format/toon'
|
||||
|
||||
try {
|
||||
const data = decode(modelOutput, { strict: true })
|
||||
// Success – data is valid
|
||||
}
|
||||
catch (error) {
|
||||
// Model output was malformed (count mismatch, invalid escapes, etc.)
|
||||
console.error('Validation failed:', error.message)
|
||||
}
|
||||
```
|
||||
|
||||
Strict mode checks counts, indentation, and escaping so you can detect truncation or malformed TOON. For complete details, see the [API Reference](/reference/api#decode-input-options).
|
||||
|
||||
## Delimiter Choices for Token Efficiency
|
||||
|
||||
Use `delimiter: '\t'` for tab-separated tables if you want even fewer tokens. Tabs are single characters, often tokenize more efficiently than commas, and rarely appear in natural text (reducing quote-escaping).
|
||||
|
||||
```ts
|
||||
const toon = encode(data, { delimiter: '\t' })
|
||||
```
|
||||
|
||||
Tell the model "fields are tab-separated" when using tabs. For more on delimiters, see the [Format Overview](/guide/format-overview#delimiter-options).
|
||||
|
||||
## Streaming Large Outputs
|
||||
|
||||
When working with large datasets (thousands of records or deeply nested structures), use `encodeLines()` to stream TOON output line-by-line instead of building the full string in memory.
|
||||
|
||||
```ts
|
||||
import { encodeLines } from '@toon-format/toon'
|
||||
|
||||
const largeData = await fetchThousandsOfRecords()
|
||||
|
||||
// Stream large dataset without loading full string in memory
|
||||
for (const line of encodeLines(largeData, { delimiter: '\t' })) {
|
||||
process.stdout.write(`${line}\n`)
|
||||
}
|
||||
```
|
||||
|
||||
The CLI also supports streaming for memory-efficient JSON-to-TOON conversion:
|
||||
|
||||
```bash
|
||||
toon large-dataset.json -o output.toon
|
||||
```
|
||||
|
||||
This streaming approach prevents out-of-memory errors when preparing large context windows for LLMs. For complete details on `encodeLines()`, see the [API Reference](/reference/api#encodelines-input-options).
|
||||
|
||||
**Consuming streaming LLM outputs:** If your LLM client exposes streaming text and you buffer by lines, you can decode TOON incrementally:
|
||||
|
||||
```ts
|
||||
import { decodeFromLines } from '@toon-format/toon'
|
||||
|
||||
// Buffer streaming response into lines
|
||||
const lines: string[] = []
|
||||
let buffer = ''
|
||||
|
||||
for await (const chunk of modelStream) {
|
||||
buffer += chunk
|
||||
let index: number
|
||||
|
||||
while ((index = buffer.indexOf('\n')) !== -1) {
|
||||
lines.push(buffer.slice(0, index))
|
||||
buffer = buffer.slice(index + 1)
|
||||
}
|
||||
}
|
||||
|
||||
// Decode buffered lines
|
||||
const data = decodeFromLines(lines)
|
||||
```
|
||||
|
||||
For streaming decode APIs, see [`decodeFromLines()`](/reference/api#decodefromlines-lines-options) and [`decodeStream()`](/reference/api#decodestream-source-options).
|
||||
|
||||
## Tips and Pitfalls
|
||||
|
||||
**Show, don't describe.** Don't explain TOON syntax in detail – just show an example. Models learn the pattern from context. A simple code block with 2–5 rows is more effective than paragraphs of explanation.
|
||||
|
||||
**Keep examples small.** Use 2–5 rows in your examples, not hundreds. The model generalizes from the pattern. Large examples waste tokens without improving accuracy.
|
||||
|
||||
**Always validate output.** Decode generated TOON with `strict: true` (default) to catch errors early. Don't assume model output is valid TOON without checking.
|
||||
|
||||
## Real-World Example
|
||||
|
||||
Here's a complete workflow: send data to a model and validate its TOON response.
|
||||
|
||||
**Prompt with TOON input:**
|
||||
|
||||
````md
|
||||
System logs in TOON format (tab-separated):
|
||||
|
||||
```toon
|
||||
events[4 ]{id level message timestamp}:
|
||||
1 error Connection timeout "2025-01-15T10:00:00Z"
|
||||
2 warn Slow query "2025-01-15T10:05:00Z"
|
||||
3 info User login "2025-01-15T10:10:00Z"
|
||||
4 error Database error "2025-01-15T10:15:00Z"
|
||||
```
|
||||
|
||||
Task: Return only error-level events as TOON. Use the same format.
|
||||
````
|
||||
|
||||
**Validate the response:**
|
||||
|
||||
```ts
|
||||
import { decode } from '@toon-format/toon'
|
||||
|
||||
const modelResponse = `
|
||||
events[2 ]{id level message timestamp}:
|
||||
1 error Connection timeout "2025-01-15T10:00:00Z"
|
||||
4 error Database error "2025-01-15T10:15:00Z"
|
||||
`
|
||||
|
||||
const filtered = decode(modelResponse, { strict: true })
|
||||
// ✓ Validated – model correctly filtered and adjusted [N] to 2
|
||||
```
|
||||
Reference in New Issue
Block a user