wehub/toon-format--toon
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
CI / ci (push) Has been cancelled
Details
Deploy Docs / Deploy Docs (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:39:56 +08:00
commit bde7ea0d58
129 changed files with 28635 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference/api.md
+766
View File
@@ -0,0 +1,766 @@
---
description: TypeScript and JavaScript encode and decode functions, options, error types, and streaming decoders for @toon-format/toon.
---
# API Reference
TypeScript/JavaScript API documentation for the `@toon-format/toon` package. For format rules, see the [Format Overview](/guide/format-overview) or the [Specification](/reference/spec). For other languages, see [Implementations](/ecosystem/implementations).
## Installation
::: code-group
```bash [npm]
npm install @toon-format/toon
```
```bash [pnpm]
pnpm add @toon-format/toon
```
```bash [yarn]
yarn add @toon-format/toon
```
:::
## Encoding Functions
### `encode(input, options?)`
Converts any JSON-serializable value to TOON format.
```ts
import { encode } from '@toon-format/toon'
const toon = encode(data, {
indent: 2,
delimiter: ',',
keyFolding: 'off',
flattenDepth: Infinity
})
```
#### Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `input` | `unknown` | Any JSON-serializable value (object, array, primitive, or nested structure) |
| `options` | `EncodeOptions?` | Optional encoding options (see [Configuration Reference](#configuration-reference)) |
#### Return Value
Returns a TOON-formatted string with no trailing newline or spaces.
#### Type Normalization
Non-JSON-serializable values are normalized before encoding:
| Input | Output |
|-------|--------|
| `Object` with `toJSON()` method | Result of calling `toJSON()`, recursively normalized |
| Finite number in `[1e-6, 1e21)` (or zero) | Canonical decimal (e.g., `1e6` → `1000000`, `-0` → `0`) |
| Finite number outside that range | Exponent form permitted (e.g., `1e-7`, `1e+21`) |
| `NaN`, `Infinity`, `-Infinity` | `null` |
| `BigInt` (within safe range) | Number |
| `BigInt` (out of range) | Quoted decimal string (e.g., `"9007199254740993"`) |
| `Date` | ISO string in quotes (e.g., `"2025-01-01T00:00:00.000Z"`) |
| `Set` | Array of normalized values |
| `Map` | Object with `String(key)` keys |
| `undefined`, `function`, `symbol` | `null` |
::: info
TOON itself doesn't specify how `Date` should be encoded – the spec leaves this to implementations. This library emits an ISO 8601 string in quotes; other implementations may choose differently.
:::
#### Example
```ts
import { encode } from '@toon-format/toon'
const items = [
{ sku: 'A1', qty: 2, price: 9.99 },
{ sku: 'B2', qty: 1, price: 14.5 }
]
console.log(encode({ items }))
```
**Output:**
```yaml
items[2]{sku,qty,price}:
A1,2,9.99
B2,1,14.5
```
### `encodeLines(input, options?)`
**Preferred method for streaming TOON output.** Converts any JSON-serializable value to TOON format as a sequence of lines, without building the full string in memory. Suitable for streaming large outputs to files, HTTP responses, or process stdout.
```ts
import { encodeLines } from '@toon-format/toon'
// Stream to stdout (Node.js)
for (const line of encodeLines(data)) {
process.stdout.write(`${line}\n`)
}
// Write to file line-by-line
const lines = encodeLines(data, { indent: 2, delimiter: '\t' })
for (const line of lines) {
await writeToStream(`${line}\n`)
}
// Collect to array
const lineArray = Array.from(encodeLines(data))
```
#### Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `input` | `unknown` | Any JSON-serializable value (object, array, primitive, or nested structure) |
| `options` | `EncodeOptions?` | Optional encoding options (see [Configuration Reference](#configuration-reference)) |
#### Return Value
Returns an `Iterable<string>` that yields TOON lines one at a time. **Each yielded string is a single line without a trailing newline character** – you must add `\n` when writing to streams or stdout.
::: info Relationship to `encode()`
`encode(value, options)` is equivalent to:
```ts
Array.from(encodeLines(value, options)).join('\n')
```
:::
#### Example
```ts
import { createWriteStream } from 'node:fs'
import { encodeLines } from '@toon-format/toon'
const data = {
items: Array.from({ length: 100000 }, (_, i) => ({
id: i,
name: `Item ${i}`,
value: Math.random()
}))
}
// Stream large dataset to file
const stream = createWriteStream('output.toon')
for (const line of encodeLines(data, { delimiter: '\t' })) {
stream.write(`${line}\n`)
}
stream.end()
```
### Replacer Function
The `replacer` option allows you to transform or filter values during encoding. It works similarly to `JSON.stringify`'s replacer parameter, but with path tracking for more precise control.
#### Type Signature
```typescript
type EncodeReplacer = (
key: string,
value: JsonValue,
path: readonly (string | number)[]
) => unknown
```
#### Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `key` | `string` | Property name, array index (as string), or empty string for root |
| `value` | `JsonValue` | The normalized value at this location |
| `path` | `readonly (string \| number)[]` | Path from root to current value |
#### Return Value
- Return the value unchanged to keep it
- Return a different value to replace it (will be normalized)
- Return `undefined` to omit properties/array elements
- For root value, `undefined` means "no change" (root cannot be omitted)
#### Examples
**Filtering sensitive data:**
```typescript
import { encode } from '@toon-format/toon'
const data = {
user: { name: 'Alice', password: 'secret123', email: 'alice@example.com' }
}
function replacer(key, value) {
if (key === 'password')
return undefined
return value
}
console.log(encode(data, { replacer }))
```
**Output:**
```yaml
user:
name: Alice
email: alice@example.com
```
**Transforming values:**
```typescript
const data = { user: 'alice', role: 'admin' }
function replacer(key, value) {
if (typeof value === 'string')
return value.toUpperCase()
return value
}
console.log(encode(data, { replacer }))
```
**Output:**
```yaml
user: ALICE
role: ADMIN
```
**Path-based transformations:**
```typescript
const data = {
metadata: { created: '2025-01-01' },
user: { created: '2025-01-02' }
}
function replacer(key, value, path) {
// Add timezone info only to top-level metadata
if (path.length === 1 && path[0] === 'metadata' && key === 'created') {
return `${value}T00:00:00Z`
}
return value
}
console.log(encode(data, { replacer }))
```
**Output:**
```yaml
metadata:
created: "2025-01-01T00:00:00Z"
user:
created: 2025-01-02
```
::: info Replacer Execution Order
The replacer is called in a depth-first manner:
1. Root value first (key = `''`, path = `[]`)
2. Then each property/element (with proper key and path)
3. Values are re-normalized after replacement
4. Children are processed after parent transformation
:::
::: warning Array Indices as Strings
Following `JSON.stringify` behavior, array indices are passed as strings (`'0'`, `'1'`, `'2'`, etc.) to the replacer, not as numbers.
:::
## Decoding Functions
### `decode(input, options?)`
Converts a TOON-formatted string back to JavaScript values.
```ts
import { decode } from '@toon-format/toon'
const data = decode(toon, {
indent: 2,
strict: true,
expandPaths: 'off'
})
```
#### Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `input` | `string` | A TOON-formatted string to parse |
| `options` | `DecodeOptions?` | Optional decoding options (see [Configuration Reference](#configuration-reference)) |
#### Return Value
Returns a JavaScript value (object, array, or primitive) representing the parsed TOON data.
#### Example
```ts
import { decode } from '@toon-format/toon'
const toon = `
items[2]{sku,qty,price}:
A1,2,9.99
B2,1,14.5
`
const data = decode(toon)
console.log(data)
```
**Output:**
```json
{
"items": [
{ "sku": "A1", "qty": 2, "price": 9.99 },
{ "sku": "B2", "qty": 1, "price": 14.5 }
]
}
```
### `decodeFromLines(lines, options?)`
Decodes TOON format from pre-split lines into a JavaScript value. This is a streaming-friendly wrapper around the event-based decoder that builds the full value in memory.
Useful when you already have lines as an array or iterable (e.g., from file streams, readline interfaces, or network responses) and want the standard decode behavior with path expansion support.
#### Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `lines` | `Iterable<string>` | Iterable of TOON lines (without trailing newlines) |
| `options` | `DecodeOptions?` | Optional decoding configuration (see [Configuration Reference](#configuration-reference)) |
#### Return Value
Returns a `JsonValue` (the parsed JavaScript value: object, array, or primitive).
#### Example
**Basic usage with arrays:**
```ts
import { decodeFromLines } from '@toon-format/toon'
const lines = ['name: Alice', 'age: 30']
const value = decodeFromLines(lines)
// { name: 'Alice', age: 30 }
```
**Streaming from Node.js readline:**
```ts
import { createReadStream } from 'node:fs'
import { createInterface } from 'node:readline'
import { decodeFromLines } from '@toon-format/toon'
const rl = createInterface({
input: createReadStream('data.toon'),
crlfDelay: Infinity,
})
const value = decodeFromLines(rl)
console.log(value)
```
**With path expansion:**
```ts
const lines = ['user.name: Alice', 'user.age: 30']
const value = decodeFromLines(lines, { expandPaths: 'safe' })
// { user: { name: 'Alice', age: 30 } }
```
### Choosing the Right Decoder
| Function | Input | Output | Async | Path Expansion | Use When |
|----------|-------|--------|-------|----------------|----------|
| `decode()` | String | Value | No | Yes | You have a complete TOON string |
| `decodeFromLines()` | Lines | Value | No | Yes | You have lines and want the full value |
| `decodeStreamSync()` | Lines | Events | No | No | You need event-by-event processing (sync) |
| `decodeStream()` | Lines | Events | Yes | No | You need event-by-event processing (async) |
::: info Key Differences
- **Value vs. Events**: Functions ending in `Stream` yield events without building the full value in memory.
- **Path expansion**: Only `decode()` and `decodeFromLines()` support `expandPaths: 'safe'`.
- **Async support**: Only `decodeStream()` accepts async iterables (useful for file/network streams).
:::
## Streaming Decoders
### `decodeStreamSync(lines, options?)`
Synchronously decodes TOON lines into a stream of JSON events. This function yields structured events that represent the JSON data model without building the full value tree.
Useful for streaming processing, custom transformations, or memory-efficient parsing of large datasets where you don't need the full value in memory.
::: tip Event Streaming
This is a low-level API that returns individual parse events. For most use cases, [`decodeFromLines()`](#decodefromlines-lines-options) or [`decode()`](#decode-input-options) are more convenient.
Path expansion (`expandPaths: 'safe'`) is **not supported** in streaming mode since it requires the full value tree.
:::
#### Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `lines` | `Iterable<string>` | Iterable of TOON lines (without trailing newlines) |
| `options` | `DecodeStreamOptions?` | Optional streaming decoding configuration (see [Configuration Reference](#configuration-reference)) |
#### Return Value
Returns an `Iterable<JsonStreamEvent>` that yields structured events (see [TypeScript Types](#typescript-types) for event structure).
#### Example
**Basic event streaming:**
```ts
import { decodeStreamSync } from '@toon-format/toon'
const lines = ['name: Alice', 'age: 30']
for (const event of decodeStreamSync(lines)) {
console.log(event)
}
// Output:
// { type: 'startObject' }
// { type: 'key', key: 'name' }
// { type: 'primitive', value: 'Alice' }
// { type: 'key', key: 'age' }
// { type: 'primitive', value: 30 }
// { type: 'endObject' }
```
**Custom processing:**
```ts
import { decodeStreamSync } from '@toon-format/toon'
const lines = ['users[2]{id,name}:', ' 1,Alice', ' 2,Bob']
let userCount = 0
for (const event of decodeStreamSync(lines)) {
if (event.type === 'endObject' && userCount < 2) {
userCount++
console.log(`Processed user ${userCount}`)
}
}
```
### `decodeStream(source, options?)`
Asynchronously decodes TOON lines into a stream of JSON events. This is the async version of [`decodeStreamSync()`](#decodestreamsync-lines-options), supporting both synchronous and asynchronous iterables.
Useful for processing file streams, network responses, or other async sources where you want to handle data incrementally as it arrives.
#### Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `source` | `AsyncIterable<string>` \| `Iterable<string>` | Async or sync iterable of TOON lines (without trailing newlines) |
| `options` | `DecodeStreamOptions?` | Optional streaming decoding configuration (see [Configuration Reference](#configuration-reference)) |
#### Return Value
Returns an `AsyncIterable<JsonStreamEvent>` that yields structured events asynchronously (see [TypeScript Types](#typescript-types) for event structure).
#### Example
**Streaming from file:**
```ts
import { createReadStream } from 'node:fs'
import { createInterface } from 'node:readline'
import { decodeStream } from '@toon-format/toon'
const fileStream = createReadStream('data.toon', 'utf-8')
const rl = createInterface({ input: fileStream, crlfDelay: Infinity })
for await (const event of decodeStream(rl)) {
console.log(event)
// Process events as they arrive
}
```
## Error Handling
Decoding throws a `ToonDecodeError` when input cannot be parsed. The class extends `SyntaxError`, so existing `error instanceof SyntaxError` checks keep working without code changes.
### `ToonDecodeError`
```ts
import { ToonDecodeError } from '@toon-format/toon'
```
#### Fields
| Field | Type | Description |
|-------|------|-------------|
| `name` | `'ToonDecodeError'` | Discriminator – `error.name === 'ToonDecodeError'` |
| `message` | `string` | Human-readable message; prefixed with `Line N: ` when a line is known |
| `line` | `number?` | 1-based line number where the error was detected |
| `source` | `string?` | Raw source line (including its leading whitespace) |
| `cause` | `unknown?` | The original error when the decoder enriched a lower-level parser failure |
The `line` and `source` fields are populated for every error that has line context – essentially every parse error during normal decoding. The `cause` chain points back to the underlying `SyntaxError` or `TypeError` thrown by the token-level parser, so debuggers and verbose loggers can show the original frame.
#### Example
```ts
import { decode, ToonDecodeError } from '@toon-format/toon'
try {
decode('a:\n\tb: 1')
}
catch (error) {
if (error instanceof ToonDecodeError) {
console.error(`Line ${error.line}:`, error.source)
console.error(error.message)
// Line 2: b: 1
// Line 2: Tabs are not allowed in indentation in strict mode
}
else {
throw error
}
}
```
::: info Backwards Compatibility
`ToonDecodeError` extends `SyntaxError`. Code written against earlier versions that catches `SyntaxError` continues to match these errors. The class adds structured fields without removing anything.
:::
## Configuration Reference
### `EncodeOptions`
Configuration for [`encode()`](#encode-input-options) and [`encodeLines()`](#encodelines-input-options):
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `indent` | `number` | `2` | Number of spaces per indentation level |
| `delimiter` | `','` \| `'\t'` \| `'\|'` | `','` | Delimiter for array values and tabular rows |
| `keyFolding` | `'off'` \| `'safe'` | `'off'` | Enable key folding to collapse single-key wrapper chains into dotted paths |
| `flattenDepth` | `number` | `Infinity` | Maximum number of segments to fold when `keyFolding` is enabled (values 0-1 have no practical effect) |
| `replacer` | `EncodeReplacer` | `undefined` | Optional hook to transform or omit values before encoding (see [Replacer Function](#replacer-function)) |
**Delimiter options:**
::: code-group
```ts [Comma (default)]
encode(data, { delimiter: ',' })
```
```ts [Tab]
encode(data, { delimiter: '\t' })
```
```ts [Pipe]
encode(data, { delimiter: '|' })
```
:::
See [Delimiter Strategies](#delimiter-strategies) for guidance on choosing delimiters.
### `DecodeOptions`
Configuration for [`decode()`](#decode-input-options) and [`decodeFromLines()`](#decodefromlines-lines-options):
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `indent` | `number` | `2` | Expected number of spaces per indentation level |
| `strict` | `boolean` | `true` | Enable strict validation (array counts, indentation, delimiter consistency) |
| `expandPaths` | `'off'` \| `'safe'` | `'off'` | Enable path expansion to reconstruct dotted keys into nested objects (pairs with `keyFolding: 'safe'`) |
By default (`strict: true`), the decoder validates input strictly:
- **Invalid escape sequences**: Throws on `\x`, unterminated strings, lone-surrogate `\uXXXX`
- **Syntax errors**: Throws on missing colons, malformed headers
- **Array length mismatches**: Throws when declared length doesn't match actual count
- **Header delimiter mismatch**: Throws when the bracket-declared delimiter differs from the field-list delimiter (§14.2)
- **Indentation errors**: Throws when leading spaces aren't exact multiples of `indent`
- **Header structure**: Throws on leading-zero or non-integer array lengths and on intervening content between bracket/fields/colon
- **Duplicate sibling keys**: Throws when an object has two children with the same key (§14.4)
- **Path-expansion conflicts**: When `expandPaths: 'safe'` is set, throws on overlapping dotted paths that would collide
All decode errors are thrown as [`ToonDecodeError`](#error-handling) instances with structured `line` and `source` fields.
Set `strict: false` to skip these checks. Duplicate sibling keys and path-expansion conflicts then resolve with last-write-wins in document order.
See [Key Folding & Path Expansion](#key-folding-path-expansion) for more details on path expansion behavior and conflict resolution.
### `DecodeStreamOptions`
Configuration for [`decodeStreamSync()`](#decodestreamsync-lines-options) and [`decodeStream()`](#decodestream-source-options):
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `indent` | `number` | `2` | Expected number of spaces per indentation level |
| `strict` | `boolean` | `true` | Enable strict validation (array counts, indentation, delimiter consistency) |
::: warning Path Expansion Not Supported
Path expansion requires building the full value tree, which is incompatible with event streaming. Use [`decodeFromLines()`](#decodefromlines-lines-options) if you need path expansion.
:::
## TypeScript Types
### `JsonStreamEvent`
Events emitted by [`decodeStreamSync()`](#decodestreamsync-lines-options) and [`decodeStream()`](#decodestream-source-options):
```ts
type JsonStreamEvent
= | { type: 'startObject' }
| { type: 'endObject' }
| { type: 'startArray', length: number }
| { type: 'endArray' }
| { type: 'key', key: string, wasQuoted?: boolean }
| { type: 'primitive', value: JsonPrimitive }
```
### Delimiters
```ts
import { DEFAULT_DELIMITER, DELIMITERS } from '@toon-format/toon'
DEFAULT_DELIMITER // ','
DELIMITERS // { comma: ',', tab: '\t', pipe: '|' }
```
| Export | Description |
|--------|-------------|
| `DEFAULT_DELIMITER` | The default delimiter character (`,`) used when none is specified |
| `DELIMITERS` | Frozen record mapping delimiter names to their characters |
| `Delimiter` | Type union of valid delimiter characters: `',' \| '\t' \| '\|'` |
| `DelimiterKey` | Type union of delimiter names: `'comma' \| 'tab' \| 'pipe'` |
### Option Types
| Export | Description |
|--------|-------------|
| `EncodeOptions` | Options accepted by [`encode()`](#encode-input-options) and [`encodeLines()`](#encodelines-input-options) |
| `DecodeOptions` | Options accepted by [`decode()`](#decode-input-options) and [`decodeFromLines()`](#decodefromlines-lines-options) |
| `DecodeStreamOptions` | Options accepted by [`decodeStreamSync()`](#decodestreamsync-lines-options) and [`decodeStream()`](#decodestream-source-options) |
| `EncodeReplacer` | Signature of the [replacer function](#replacer-function) |
| `ResolvedEncodeOptions` | `EncodeOptions` after defaults are applied (advanced) |
| `ResolvedDecodeOptions` | `DecodeOptions` after defaults are applied (advanced) |
## Guides & Examples
### Round-Trip Compatibility
TOON provides lossless round-trips after normalization:
```ts
import { decode, encode } from '@toon-format/toon'
const original = {
users: [
{ id: 1, name: 'Alice', role: 'admin' },
{ id: 2, name: 'Bob', role: 'user' }
]
}
const toon = encode(original)
const restored = decode(toon)
console.log(JSON.stringify(original) === JSON.stringify(restored))
// true
```
**With Key Folding:**
```ts
import { decode, encode } from '@toon-format/toon'
const original = { data: { metadata: { items: ['a', 'b'] } } }
// Encode with folding
const toon = encode(original, { keyFolding: 'safe' })
// → "data.metadata.items[2]: a,b"
// Decode with expansion
const restored = decode(toon, { expandPaths: 'safe' })
// → { data: { metadata: { items: ['a', 'b'] } } }
console.log(JSON.stringify(original) === JSON.stringify(restored))
// true
```
### Key Folding & Path Expansion
**Key Folding** (`keyFolding: 'safe'`) collapses single-key wrapper chains during encoding:
```ts
import { encode } from '@toon-format/toon'
const data = { data: { metadata: { items: ['a', 'b'] } } }
// Without folding
encode(data)
// data:
// metadata:
// items[2]: a,b
// With folding
encode(data, { keyFolding: 'safe' })
// data.metadata.items[2]: a,b
```
**Path Expansion** (`expandPaths: 'safe'`) reverses this during decoding:
```ts
import { decode } from '@toon-format/toon'
const toon = 'data.metadata.items[2]: a,b'
const data = decode(toon, { expandPaths: 'safe' })
console.log(data)
// { data: { metadata: { items: ['a', 'b'] } } }
```
**Expansion Conflict Resolution:**
When multiple expanded keys construct overlapping paths, the decoder merges them recursively:
- **Object + Object**: Deep merge recursively
- **Object + Non-object** (array or primitive): Conflict
- With `strict: true` (default): Error
- With `strict: false`: Last-write-wins (LWW)
Duplicate sibling keys (independent of `expandPaths`) follow the same policy: strict mode throws, lenient mode keeps the last value seen.
### Delimiter Strategies
Tab delimiters (`\t`) often tokenize more efficiently than commas. Tabs are single characters that rarely appear in natural text, which reduces the need for quote-escaping and leads to smaller token counts in large datasets.
Example:
```yaml
items[2 ]{sku name qty price}:
A1 Widget 2 9.99
B2 Gadget 1 14.5
```
For maximum token savings on large tabular data, combine tab delimiters with key folding:
```ts
encode(data, { delimiter: '\t', keyFolding: 'safe' })
```
**Choosing a Delimiter:**
- **Comma (`,`)**: Default, widely understood, good for simple tabular data.
- **Tab (`\t`)**: Best for LLM token efficiency, excellent for large datasets.
- **Pipe (`|`)**: Alternative when commas appear frequently in data.
docs/reference/efficiency-formalization.md
+509
View File
@@ -0,0 +1,509 @@
---
description: Mathematical model of TOON's byte-level overhead vs JSON across structure families, with formulas and worked examples.
---
# TOON vs JSON: Byte-Level Efficiency Model
A mathematical analysis of TOON's byte efficiency compared to JSON across different data structures.
::: info Scope of This Document
This page presents a theoretical, character-based comparison between TOON and JSON. For practical benchmarks and token counts, see [Benchmarks](/guide/benchmarks). It is an **advanced, non-normative** reference: it explains TOON's design from a mathematical angle but does not change the TOON specification.
:::
## Overview
Standard JSON introduces structural verbosity that inflates token usage and inference cost. This page formalises a byte-level comparison between TOON and JSON to evaluate whether TOON achieves quantifiable efficiency gains by removing structural redundancy.
Under the assumptions described below (compact JSON, canonical TOON, ASCII keys and punctuation, shallow to moderate nesting, and mostly unquoted TOON strings), TOON's **structural overhead is lower than compact JSON** for the structure families analyzed here, except arrays of arrays.
### Key Findings
- **Tabular arrays** represent TOON's optimal use case, with efficiency gains scaling linearly with both row count and field count.
- **Simple objects and primitive arrays** show consistent byte reduction, with savings proportional to the number of fields or elements.
- **Nested objects** benefit from reduced overhead, though efficiency decreases with depth due to indentation costs; at sufficient depth, compact JSON can become smaller.
- **Arrays of arrays** are the only structure where TOON is less efficient than JSON in this analysis, due to TOON's explicit list markers and inner array headers.
## Methodology
We define recursive byte-length functions $L_{\text{json}}$ and $L_{\text{toon}}$ for both formats, then derive the efficiency delta:
$$
\Delta = L_{\text{json}}(\Omega) - L_{\text{toon}}(\Omega)
$$
Where $\Omega$ represents the data structure under comparison. If $\Delta > 0$, TOON uses fewer bytes than JSON for that structure.
::: info Scope & Assumptions
- **Compact JSON**: JSON is assumed to be compact (no spaces or newlines outside strings). Byte counts are computed on this compact form.
- **Canonical TOON**: TOON is assumed to follow canonical formatting (indent = 2 spaces, exactly one space after `:`, no spaces after commas in arrays/field lists, no trailing spaces).
- **Keys and strings**: All keys are "simple" ASCII identifier-style keys that:
- must be quoted in JSON, and
- can be left unquoted in TOON (no characters that would force quoting).
Many examples assume values are numbers, booleans, null, or TOON-safe strings that can be unquoted in TOON but must be quoted in JSON.
- **Numbers**: For this analysis only, both formats are assumed to use the same canonical decimal representation. JSON could use exponent forms; we ignore that here to isolate structural differences.
- **ASCII/UTF-8**: Keys and structural tokens are assumed ASCII, so byte length equals character count ($|x|_{\text{utf8}} = |x|_{\text{char}}$). Non-ASCII content affects both formats similarly and does not change the structural conclusions.
- **Nesting depth**: Closed-form expressions are given for flat structures and a single level of nesting. Each additional nesting level in TOON adds 2 bytes of indentation per nested line. At sufficient depth, the braces of compact JSON can win over TOON's indentation (as seen in [When Not to Use TOON](/guide/getting-started#when-not-to-use-toon)).
- **Byte vs token count**: Modern LLM tokenizers operate over UTF-8 bytes, so byte length is a good upper bound and first-order proxy for token count, even though the mapping is not exactly linear.
:::
Think of this as a simplified structural model: we strip away real-world noise and ask, "if you only count structural characters, how do JSON and TOON compare?"
## Formal Notation
### Data Model
Let $\omega$ be a primitive value such that $\omega \in \{\text{string, number, boolean, null}\}$.
Let $\mathcal{O}$ be an object composed of $n$ key-value pairs:
$$
\mathcal{O} = \{(k_1, v_1), (k_2, v_2), \dots, (k_n, v_n)\}
$$
Let $\mathcal{A}$ be an array composed of $n$ elements:
$$
\mathcal{A} = \{v_1, v_2, \dots, v_n\}
$$
Where:
- $k_i$ is a key (string)
- $v_i$ can be a primitive value $\omega$, an object $\mathcal{O}$, or an array $\mathcal{A}$
Therefore: $v_i \in \{\omega, \mathcal{O}, \mathcal{A}\}$
### String Length
Let $\mathcal{S}$ be the set of valid Unicode strings. For any string $x \in \mathcal{S}$, we denote $|x|_{\text{utf8}}$ as the byte-length of $x$ under UTF-8 encoding.
### Integer Length
Let $n \in \mathbb{Z}_{\ge 0}$ be a non-negative integer. The number of bytes required to represent $n$ in decimal format is:
$$
L_{\text{num}}(n) = \begin{cases}
1 & \text{if } n = 0 \\
\lfloor \log_{10}(|n|) \rfloor + 1 & \text{if } n > 0
\end{cases}
$$
## JSON Size Functions
For a flat object of $n$ keys:
$$
L_{\text{json}}(\mathcal{O}) = \underbrace{2}_{\{\}} + \sum_{i=1}^{n} (L_{\text{str}}(k_i) + \underbrace{1}_{:} + L_{\text{json}}(v_i)) + \underbrace{(n-1)}_{\text{commas}}
$$
Where $L_{\text{str}}(k)$ is the length of the key including its mandatory quotes:
$$
L_{\text{str}}(k) = |k|_{\text{utf8}} + \underbrace{2}_{\text{quotes}}
$$
### Primitive Values in JSON
When $v_i$ is a primitive data type $\omega$:
| Type | Formula |
|------|---------|
| String | $L_{\text{str}}(v_i) = \lvert v_i\rvert_{\text{utf8}} + 2$ |
| Number | $L_{\text{num}}(v_i) = \lvert v_i\rvert_{\text{utf8}}$ |
| Boolean | $L_{\text{bool}}(v_i) = \lvert v_i\rvert_{\text{utf8}}$ |
| Null | $L_{\text{null}}(v_i) = \lvert v_i\rvert_{\text{utf8}}$ |
### Arrays in JSON
When $v_i$ is an array $\mathcal{A}$:
$$
L_{\text{json}}(\mathcal{A}) = \underbrace{2}_{\text{[]}} + \sum_{i=1}^{n} L_{\text{json}}(v_i) + \underbrace{(n-1)}_{\text{commas}}
$$
## TOON Size Functions
For a flat object of $n$ keys:
$$
L_{\text{toon}}(\mathcal{O}) = \sum_{i=1}^{n} (L_{\text{str}}(k_i) + \underbrace{1}_{:} + \underbrace{1}_{\text{space}} + L_{\text{toon}}(v_i)) + \underbrace{(n-1)}_{\text{newlines}}
$$
Where $L_{\text{str}}(k)$ is the length of the key (no quotes required for simple keys):
$$
L_{\text{str}}(k) = |k|_{\text{utf8}}
$$
### Primitive Values in TOON
When $v_i$ is a primitive data type $\omega$:
| Type | Formula |
|------|---------|
| String (normal) | $L_{\text{str}}(v_i) = \lvert v_i\rvert_{\text{utf8}}$ |
| String (looks like number/boolean) | $L_{\text{str}}(v_i) = \lvert v_i\rvert_{\text{utf8}} + 2$ |
| Number | $L_{\text{num}}(v_i) = \lvert v_i\rvert_{\text{utf8}}$ |
| Boolean | $L_{\text{bool}}(v_i) = \lvert v_i\rvert_{\text{utf8}}$ |
| Null | $L_{\text{null}}(v_i) = \lvert v_i\rvert_{\text{utf8}}$ |
### Simple Arrays in TOON
Here $L_{\text{toon}}(\mathcal{A})$ refers to the length of the whole field line `key[N]: ...`, not just the array value.
When $v_i$ is a simple array $\mathcal{A}$:
$$
L_{\text{toon}}(\mathcal{A}) = L_{\text{str}}(k_i) + \underbrace{1}_{\text{[}} + L_{\text{num}}(n) + \underbrace{1}_{\text{]}} + \underbrace{1}_{:} + \underbrace{1}_{\text{space}} + \sum_{i=1}^{n} L_{\text{toon}}(v_i) + \underbrace{(n-1)}_{\text{commas}}
$$
### Tabular Arrays in TOON
When $v_i$ is an array of objects with $m$ fields:
$$
\begin{split}
L_{\text{toon}}(\mathcal{A}') = L_{\text{str}}(k_i) + \underbrace{1}_{\text{[}} + L_{\text{num}}(n) + \underbrace{1}_{\text{]}} + \underbrace{1}_{\{} + \\
\sum_{i=1}^{m} L_{\text{str}}(k_i) + \underbrace{(m-1)}_{\text{commas}} + \underbrace{1}_{\}} + \underbrace{1}_{:} + \\
\underbrace{2n}_{\text{indents}} + \sum_{i=1}^{n}\sum_{j=1}^{m} L_{\text{toon}}(v_{ij}) + \underbrace{(m-1)n}_{\text{commas}} + \underbrace{n}_{\text{newlines}}
\end{split}
$$
*Note: The term $2n$ assumes an indentation size of 2 spaces.*
## Efficiency Analysis by Structure
Each subsection below focuses on a particular structure family, states the resulting formula, and shows a small example. Intuitively, TOON tends to win when it can:
- avoid repeating keys (tabular arrays),
- avoid quoting keys and many values,
- and replace braces with indentation,
and tends to lose when it pays a fixed overhead per element (arrays of arrays) or deep indentation (heavily nested configs).
### Simple Objects
Flat objects with primitive string values are the easiest win: JSON pays for braces and quoted keys and strings, while TOON drops braces at the root, omits quotes on simple keys, and uses one line per field.
For objects with only string primitives:
$$
\Delta_{\text{obj}} = 2 + n + \sum_{i=1}^{n}(L_{\text{json}}(v_i)) - \sum_{i=1}^{n}(L_{\text{toon}}(v_i))
$$
If all values are strings that can be unquoted in TOON, this simplifies to:
$$
f(n) = 2 + 3n
$$
**Example:** For 1,000,000 objects, TOON saves **3,000,002 bytes ≈ 2.86 MB**.
#### Empirical Validation
::: code-group
```json [JSON (21 bytes)]
{ "id": 1, "name": "Ada" }
```
```yaml [TOON (15 bytes)]
id: 1
name: Ada
```
:::
$$
\Delta_{\text{obj}} = 2 + \underbrace{2}_{n} + \underbrace{6}_{\sum L_{\text{json}}(v_i)} - \underbrace{4}_{\sum L_{\text{toon}}(v_i)} = 6
$$
### Nested Objects
Adding a wrapper object (one extra level of nesting) introduces extra braces for JSON and extra indentation and newlines for TOON. For a single level of nesting with primitive values, TOON still comes out ahead, but the net advantage is smaller.
For a single level of nesting with primitives:
$$
f(n) = 5 + n
$$
**Example:** For 1,000,000 nested objects (depth 1), TOON saves **1,000,005 bytes ≈ 0.95 MB**.
::: warning Caveat
This formula is for a single nesting level. Each additional nesting level adds 2 spaces of indentation per nested line; at sufficient depth, compact JSON can become smaller, especially when tabular opportunities disappear (see [When Not to Use TOON](/guide/getting-started#when-not-to-use-toon) and the "Deeply nested configuration" dataset in [Benchmarks](/guide/benchmarks)).
:::
#### Empirical Validation
::: code-group
```json [JSON (30 bytes)]
{ "user": { "id": 1, "name": "Ada" } }
```
```yaml [TOON (25 bytes)]
user:
id: 1
name: Ada
```
:::
$$
\Delta_{\text{nested}} = 5
$$
### Primitive Arrays
For arrays of string primitives, JSON writes `["foo","bar","baz"]`, quoting every string and using `[]` for the array. TOON writes `key[N]: foo,bar,baz`, paying once for the length marker but omitting most quotes.
For arrays of $n$ string primitives:
$$
\Delta_{\text{arr}} = 3 - L_{\text{num}}(n) + \sum_{i=1}^{n}(L_{\text{json}}(v_i)) - \sum_{i=1}^{n}(L_{\text{toon}}(v_i))
$$
With string values that can be unquoted in TOON, this simplifies to:
$$
f(n) = 2 + 2n - \lfloor \log_{10}(|n|) \rfloor
$$
**Example:** For 1,000,000 elements, TOON saves **1,999,996 bytes ≈ 1.91 MB**.
#### Empirical Validation
::: code-group
```json [JSON (28 bytes)]
{ "tags": ["foo", "bar", "baz"] }
```
```yaml [TOON (20 bytes)]
tags[3]: foo,bar,baz
```
:::
$$
\Delta_{\text{arr}} = 3 - \underbrace{1}_{L_{\text{num}}(3)} + \underbrace{15}_{\sum L_{\text{json}}} - \underbrace{9}_{\sum L_{\text{toon}}} = 8
$$
### Root Arrays
At the root, JSON writes `["x","y","z"]`; TOON writes `[3]: x,y,z`. There is no object key cost, so the advantage mainly comes from not quoting TOON-safe strings and from replacing `[]` with `[N]:`.
For root-level arrays of $n$ string primitives:
$$
f(n) = -3 + 2n - \lfloor \log_{10}(|n|) \rfloor
$$
**Example:** For 1,000,000 elements, TOON saves **1,999,991 bytes ≈ 1.91 MB**.
#### Empirical Validation
::: code-group
```json [JSON (13 bytes)]
["x", "y", "z"]
```
```yaml [TOON (10 bytes)]
[3]: x,y,z
```
:::
$$
\Delta_{\text{root}} = \underbrace{9}_{\sum L_{\text{json}}} - 2 - \underbrace{1}_{L_{\text{num}}(3)} - \underbrace{3}_{\sum L_{\text{toon}}} = 3
$$
### Tabular Arrays
Uniform arrays of objects are TOON's sweet spot. JSON repeats every key for every row, while TOON declares the length and column names once (`key[N]{id,qty,...}:`) and streams rows as bare values.
For arrays of objects with $n$ rows and $m$ fields, assuming numeric values and $|k| = 3$:
$$
f(n) = 1 + nm(3 + |k|) - m(1 + |k|) - \lfloor \log_{10}(|n|) \rfloor
$$
**Example:** For 1,000,000 rows with 2 fields and 3-character field names, TOON saves **11,999,987 bytes ≈ 11.44 MB**.
This is where TOON's design (declare fields once, stream rows) pays off most strongly: savings grow linearly with both row count and field count.
#### Empirical Validation
::: code-group
```json [JSON (45 bytes)]
{ "items": [{ "id": 1, "qty": 5 }, { "id": 2, "qty": 3 }] }
```
```yaml [TOON (29 bytes)]
items[2]{id,qty}:
1,5
2,3
```
:::
$$
\Delta_{\text{tab}} = 2 + \underbrace{4}_{nm} - \underbrace{2}_{m} + \underbrace{22}_{\Sigma L_{\text{json}}} - \underbrace{1}_{L_{\text{num}}(n)} - \underbrace{5}_{\Sigma L_{\text{toon}}(k)} - \underbrace{4}_{\Sigma L_{\text{toon}}(v)} = 16
$$
### Arrays of Arrays
Arrays of arrays of primitives are where TOON structurally loses: each inner array becomes a list item with its own header, so TOON pays a fixed overhead per inner array (`"- "` plus `"[m]: "`), while JSON just uses commas.
::: info Practical Note
For arrays of arrays of primitives, this model predicts that JSON is more byte-efficient than TOON, because TOON pays ~6 extra bytes per inner array (2 for `"- "`, 4 for `"[m]: "`), plus the length marker.
:::
For arrays of arrays with $n$ outer elements and $m$ inner elements:
$$
\begin{split}
\Delta_{\text{arrarr}} = 2 - 6n - \sum_{i=1}^{n}\sum_{j=1}^{m} L_{\text{num}}(m) + \\
\sum_{i=1}^{n}\sum_{j=1}^{m} L_{\text{json}}(v_{ij}) - \sum_{i=1}^{n}\sum_{j=1}^{m} L_{\text{toon}}(v_{ij})
\end{split}
$$
With string primitives and $m = 2$:
$$
f(n) = 2 - 6n - \sum_{i=1}^{n}\sum_{j=1}^{m} (\lfloor \log_{10}(|m|) \rfloor + 1) + 2nm
$$
**Example:** For 1,000,000 arrays with $m = 2$, TOON **wastes 2,999,998 bytes ≈ 2.86 MB** relative to JSON under this model.
#### Empirical Validation
::: code-group
```json [JSON (23 bytes)]
{ "pairs": [[1, 2], [3, 4]] }
```
```yaml [TOON (35 bytes)]
pairs[2]:
- [2]: 1,2
- [2]: 3,4
```
:::
$$
\Delta_{\text{arrarr}} = 2 - \underbrace{12}_{6n} - \underbrace{2}_{\sum L_{\text{num}}(m)} + \underbrace{4}_{\sum L_{\text{json}}} - \underbrace{4}_{\sum L_{\text{toon}}} = -12
$$
### Strings That Look Like Literals
Strings that look like numbers or booleans (e.g. `"123"`, `"true"`) must be quoted in both JSON and TOON, slightly reducing TOON's advantage because it no longer saves quotes on those values.
For objects containing such strings:
$$
\Delta_{\text{strlit}} = 2 + n
$$
**Example:** For 1,000,000 objects, TOON saves **2,000,002 bytes ≈ 1.91 MB**.
#### Empirical Validation
::: code-group
```json [JSON (34 bytes)]
{ "version": "123", "enabled": "true" }
```
```yaml [TOON (30 bytes)]
version: "123"
enabled: "true"
```
:::
$$
\Delta_{\text{str}} = 2 + \underbrace{2}_{n} = 4
$$
### Empty Structures
Empty containers reveal structural differences even at minimal sizes.
**Empty Object:**
$$
\Delta_{\text{EmptyObject}} = 2
$$
JSON requires `{}` (2 bytes), whereas a completely empty root object in TOON is represented as an empty document (0 bytes).
**Empty Array (field):**
$$
\Delta_{\text{EmptyArray}} = 3
$$
For a field named `key`, JSON uses `{"key":[]}` in compact form, while TOON uses:
```yaml
key: []
```
Under this model, that yields a constant 3-byte advantage for TOON. The legacy `key[0]:` form remains decodable for backward compatibility.
## Summary Table
The table below summarizes the formulas and which side wins under the modeling assumptions.
| Structure | Efficiency Formula | TOON Advantage? |
|-----------|-------------------|-----------------|
| Simple Objects | $f(n) = 2 + 3n$ | ✅ Yes |
| Nested Objects (1 level) | $f(n) = 5 + n$ | ✅ Yes (shrinks with depth) |
| Primitive Arrays | $f(n) = 2 + 2n - \lfloor \log_{10}(n) \rfloor$ | ✅ Yes |
| Root Arrays | $f(n) = -3 + 2n - \lfloor \log_{10}(n) \rfloor$ | ✅ Yes |
| Tabular Arrays | $f(n) = 1 + nm(3+\lvert k\rvert) - m(1+\lvert k\rvert) - \lfloor \log_{10}(n) \rfloor$ | ✅ **Best case** |
| Arrays of Arrays | $f(n) = 2 - 6n + 2nm - \text{overhead}$ | ❌ JSON wins here |
| String Literals | $f(n) = 2 + n$ | ✅ Yes (smaller gain) |
| Empty Structures | $\Delta = 2$ or $3$ | ✅ Yes |
In short:
- TOON's gains are **linear in the number of fields** for flat objects.
- For arrays, gains grow **linearly in the number of elements**, and for tabular arrays **linearly in both rows and fields**.
- Arrays of arrays are the main structural case where JSON is smaller.
- Deep nesting and heavy quoting can erode or reverse these advantages in real data.
## Conclusion
This simplified theoretical model supports TOON's design goal: structurally, it reduces overhead compared to compact JSON in many common patterns by:
- avoiding repeated keys in tabular arrays,
- omitting quotes on many keys and values,
- and replacing braces with indentation at shallow depths.
For the structure families examined here and under the stated assumptions, the structural overhead of TOON is lower than that of compact JSON except for arrays of arrays. Since UTF-8 byte length is a reasonable first-order proxy for tokens, these structural savings usually translate into lower token counts in those patterns.
At the same time, this is deliberately a simplified model. In real datasets, additional factors – deeper or irregular nesting, heavily quoted strings, exponent notation in JSON, and tokenizer idiosyncrasies – can reduce or even reverse these gains. Our [Benchmarks](/guide/benchmarks) and [When Not to Use TOON](/guide/getting-started#when-not-to-use-toon) show that compact JSON can be more efficient for deeply nested or low-tabularity data. Use this page as intuition for *why* TOON behaves the way it does, not as a universal guarantee.
## Related Resources
- [Benchmarks](/guide/benchmarks) – Empirical token count and accuracy comparisons across formats
- [Specification](/reference/spec) – Formal TOON specification
## References
This analysis is based on:
- **Original Research**: [TOON vs. JSON: A Mathematical Evaluation of Byte Efficiency in Structured Data](https://www.researchgate.net/publication/397903673_TOON_vs_JSON_A_Mathematical_Evaluation_of_Byte_Efficiency_in_Structured_Data)
- **TOON Specification**: [toon-format/spec](https://github.com/toon-format/spec)
- **JSON Specification**: [RFC 8259](https://datatracker.ietf.org/doc/html/rfc8259), [ECMA-404](https://www.ecma-international.org/publications-and-standards/standards/ecma-404/)
---
This page was contributed by Mateo Lafalce ([@mateolafalce](https://github.com/mateolafalce)).
*Have questions or found an error in the formalization? Open an issue on [GitHub](https://github.com/toon-format/spec) or contribute improvements to this analysis.*
docs/reference/spec.md
+175
View File
@@ -0,0 +1,175 @@
---
description: Guided tour of the TOON specification – sections, conformance checklists, media type, and versioning.
---
# Specification
The [TOON specification](https://github.com/toon-format/spec) is the authoritative reference for implementing encoders, decoders, and validators. It defines the concrete syntax, normative encoding/decoding behavior, and strict-mode validation rules.
You don't need this page to *use* TOON. It's mainly for implementers and contributors. If you're looking to learn how to use TOON, start with the [Getting Started](/guide/getting-started) guide instead.
> [!NOTE]
> The TOON specification is stable, but also an idea in progress. Nothing's set in stone – help shape where it goes by contributing to it or sharing feedback.
## Current Version
**Spec v{{ $spec.version }}** (2026-05-20) is the current published Working Draft. It is stable for implementation but not yet finalized; see "Status of This Document" in the spec for details.
## Media Type & File Extension
The spec defines a provisional media type and file extension in [§17](https://github.com/toon-format/spec/blob/main/SPEC.md#17-iana-considerations):
- **Media type:** `text/toon` (provisional, not yet IANA‑registered; UTF‑8 only)
- **File extension:** `.toon`
TOON documents are always UTF‑8 with LF (`\n`) line endings; the optional `charset` parameter, when present, is `utf-8`.
## Guided Tour of the Spec
### Core Concepts
[§1 Terminology and Conventions](https://github.com/toon-format/spec/blob/main/SPEC.md#1-terminology-and-conventions):
Defines key terms like "indentation level", "active delimiter", "strict mode", and RFC2119 keywords (MUST, SHOULD, MAY).
[§2 Data Model](https://github.com/toon-format/spec/blob/main/SPEC.md#2-data-model):
Specifies the JSON data model (objects, arrays, primitives), array/object ordering requirements, and canonical number formatting (canonical decimal for values in `[1e-6, 1e21)` or zero; exponent form permitted outside).
[§3 Encoding Normalization](https://github.com/toon-format/spec/blob/main/SPEC.md#3-encoding-normalization-reference-encoder):
Defines how non-JSON types (Date, BigInt, NaN, Infinity, undefined, etc.) are normalized before encoding. Required reading for encoder implementers.
[§4 Decoding Interpretation](https://github.com/toon-format/spec/blob/main/SPEC.md#4-decoding-interpretation-reference-decoder):
Specifies how decoders map text tokens to host values (quoted strings, unquoted primitives, numeric parsing with leading-zero handling). Decoders default to strict mode (`strict = true`) in the reference implementation; strict-mode errors are enumerated in §14.
### Syntax Rules
[§5 Concrete Syntax and Root Form](https://github.com/toon-format/spec/blob/main/SPEC.md#5-concrete-syntax-and-root-form):
Defines TOON's line-oriented, indentation-based notation and how to determine whether the root is an object, array, or primitive.
[§6 Header Syntax](https://github.com/toon-format/spec/blob/main/SPEC.md#6-header-syntax-normative):
Normative ABNF grammar for array headers: `key[N<delim?>]{fields}:`. Specifies bracket segments, delimiter symbols, and field lists.
[§7 Strings and Keys](https://github.com/toon-format/spec/blob/main/SPEC.md#7-strings-and-keys):
Complete quoting rules (when strings MUST be quoted), escape sequences (only `\\`, `\"`, `\n`, `\r`, `\t`, and `\uXXXX` for other U+0000–U+001F controls are valid), and key encoding requirements.
[§8 Objects](https://github.com/toon-format/spec/blob/main/SPEC.md#8-objects):
Object field encoding (key: value), nesting rules, key order preservation, and empty object handling.
[§9 Arrays](https://github.com/toon-format/spec/blob/main/SPEC.md#9-arrays):
Covers all array forms: primitive (inline), arrays of objects (tabular), mixed/non-uniform (list), and arrays of arrays. Includes tabular detection requirements.
[§10 Objects as List Items](https://github.com/toon-format/spec/blob/main/SPEC.md#10-objects-as-list-items):
Indentation rules for objects appearing in list items (first field on the hyphen line), including the canonical pattern when the first field is a tabular array (header on the hyphen line, rows at depth +2, sibling fields at depth +1).
[§11 Delimiters](https://github.com/toon-format/spec/blob/main/SPEC.md#11-delimiters):
Delimiter scoping (document vs active), delimiter-aware quoting, and parsing rules for comma/tab/pipe delimiters.
[§12 Indentation and Whitespace](https://github.com/toon-format/spec/blob/main/SPEC.md#12-indentation-and-whitespace):
Encoding requirements (consistent spaces, no tabs in indentation, no trailing spaces/newlines) and decoding rules (strict vs non-strict indentation handling).
### Conformance and Validation
[§13 Conformance and Options](https://github.com/toon-format/spec/blob/main/SPEC.md#13-conformance-and-options):
Defines conformance classes (encoder, decoder, validator), standardized options, and conformance checklists.
[§13.4 Key Folding and Path Expansion](https://github.com/toon-format/spec/blob/main/SPEC.md#134-key-folding-and-path-expansion):
Optional encoder feature (key folding) and decoder feature (path expansion) for collapsing/expanding dotted paths, with deep-merge semantics and strict/non-strict conflict resolution.
[§14 Strict Mode Errors and Diagnostics](https://github.com/toon-format/spec/blob/main/SPEC.md#14-strict-mode-errors-and-diagnostics-authoritative-checklist):
**Authoritative checklist** of all strict-mode errors: array count and width mismatches (§14.1), syntax and structural errors (§14.2), path expansion conflicts (§14.3), and duplicate sibling keys (§14.4).
### Implementation Guidance
[§15 Security Considerations](https://github.com/toon-format/spec/blob/main/SPEC.md#15-security-considerations):
Injection risks, quoting rules, and strict-mode checks relevant to security.
[§16 Internationalization](https://github.com/toon-format/spec/blob/main/SPEC.md#16-internationalization):
Unicode handling and locale-independent number formatting.
[§17 IANA Considerations](https://github.com/toon-format/spec/blob/main/SPEC.md#17-iana-considerations):
Media type registration plans and provisional status.
[§18 Versioning and Extensibility](https://github.com/toon-format/spec/blob/main/SPEC.md#18-versioning-and-extensibility):
How the spec evolves: major vs minor bumps and the extensibility policy.
[§19 Intellectual Property Considerations](https://github.com/toon-format/spec/blob/main/SPEC.md#19-intellectual-property-considerations):
Licensing and IP terms for the specification.
[Appendix F: Host Type Normalization Examples](https://github.com/toon-format/spec/blob/main/SPEC.md#appendix-f-host-type-normalization-examples-informative):
Non-normative guidance for Go, JavaScript, Python, Rust, and Java implementations on normalizing language-specific types.
[Appendix C: Test Suite and Compliance](https://github.com/toon-format/spec/blob/main/SPEC.md#appendix-c-test-suite-and-compliance-informative):
Reference test suite at [github.com/toon-format/spec/tree/main/tests](https://github.com/toon-format/spec/tree/main/tests) for validating implementations.
## Spec Sections at a Glance
| Section | Topic | When to Read |
|---------|-------|--------------|
| §1–4 | Data model, normalization, decoding | Implementing encoders/decoders |
| §5–6 | Syntax, headers, root form | Implementing parsers |
| §7 | Strings, keys, quoting, escaping | Implementing string handling |
| §8–10 | Objects, arrays, list items | Implementing structure encoding |
| §11–12 | Delimiters, indentation, whitespace | Implementing formatting and validation |
| §13 | Conformance, options, key folding/path expansion | Implementing options and features |
| §14 | Strict-mode errors | Implementing validators |
| §15–16 | Security, internationalization | Operational considerations |
| §17–19 | IANA, versioning, IP | Ecosystem and licensing |
## Conformance Checklists
The spec includes three conformance checklists:
### Encoder Checklist (§13.1) <sup>[↗ SPEC.md](https://github.com/toon-format/spec/blob/main/SPEC.md#131-encoder-conformance-checklist)</sup>
Key requirements:
- Produce UTF-8 with LF line endings
- Use consistent indentation (default 2 spaces, no tabs)
- Escape `\\`, `\"`, `\n`, `\r`, `\t` in quoted strings, and use `\uXXXX` for any other U+0000–U+001F control character; lone surrogates are rejected
- Quote strings with active delimiter, colon, or structural characters
- Emit array lengths `[N]` matching actual count
- Preserve object key order
- Emit numbers per §2 (canonical decimal in `[1e-6, 1e21)` or zero; exponent form permitted outside)
- Convert `-0` to `0`, `NaN`/±Infinity to `null`
- Emit booleans and null as lowercase literals (`true`, `false`, `null`)
- No trailing spaces or trailing newline
- When `keyFolding="safe"` is enabled, folding MUST follow §13.4:
- Only fold IdentifierSegment keys (letters/digits/underscores, no dots),
- Do not introduce collisions with existing sibling keys,
- Do not fold segments that would require quoting.
- When `flattenDepth` is set, folding MUST stop at the configured number of segments (§13.4).
### Decoder Checklist (§13.2) <sup>[↗ SPEC.md](https://github.com/toon-format/spec/blob/main/SPEC.md#132-decoder-conformance-checklist)</sup>
Key requirements:
- Parse array headers per §6 (length, delimiter, fields)
- Split inline arrays and tabular rows using active delimiter only
- Unescape quoted strings with only valid escapes
- Type unquoted primitives: true/false/null → booleans/null, numeric → number, else → string
- Enforce strict-mode rules when `strict=true`
- Preserve array order and object key order
- When `expandPaths="safe"` is enabled, expand dotted keys into nested objects per §13.4:
- Split on `.`, only expand when all segments are IdentifierSegments,
- Deep-merge overlapping paths (object + object),
- Do not perform element-wise array merges.
- With `expandPaths="safe"` and `strict=true` (default), MUST error on any expansion conflict (§14.3).
- With `expandPaths="safe"` and `strict=false`, MUST apply deterministic last-write-wins (LWW) conflict resolution (§13.4).
### Validator Checklist (§13.3) <sup>[↗ SPEC.md](https://github.com/toon-format/spec/blob/main/SPEC.md#133-validator-conformance-checklist)</sup>
Validators should verify:
- Structural conformance (headers, indentation, list markers)
- Whitespace invariants (no trailing spaces/newlines)
- Delimiter consistency between headers and rows
- Array length counts match declared `[N]`
- All strict-mode requirements (including path-expansion conflicts when enabled)
## Versioning
The spec uses semantic versioning (major.minor):
- **Major version** (e.g., v2 → v3): Breaking changes, incompatible with previous versions
- **Minor version** (e.g., v3.1 → v3.2): Clarifications, additional requirements, or backward-compatible additions
See [Appendix D: Document Changelog](https://github.com/toon-format/spec/blob/main/SPEC.md#appendix-d-document-changelog-informative) for detailed version history.
## Contributing to the Spec
The spec is community-maintained at [github.com/toon-format/spec](https://github.com/toon-format/spec). We welcome contributions of all kinds: reporting ambiguities or errors, proposing clarifications and examples, adding test cases to the reference suite, or discussing edge cases and normative behavior. Your feedback helps shape the format.
docs/reference/syntax-cheatsheet.md
+367
View File
@@ -0,0 +1,367 @@
---
description: JSON-to-TOON mappings at a glance for objects, arrays, quoting, key folding, and type conversions.
---
# Syntax Cheatsheet
Quick reference for mapping JSON to TOON format. For rigorous, normative syntax rules and edge cases, see the [Specification](/reference/spec).
## Objects
::: code-group
```json [JSON]
{
"id": 1,
"name": "Ada"
}
```
```yaml [TOON]
id: 1
name: Ada
```
:::
## Nested Objects
::: code-group
```json [JSON]
{
"user": {
"id": 1,
"name": "Ada"
}
}
```
```yaml [TOON]
user:
id: 1
name: Ada
```
:::
## Primitive Arrays
::: code-group
```json [JSON]
{
"tags": ["foo", "bar", "baz"]
}
```
```yaml [TOON]
tags[3]: foo,bar,baz
```
:::
## Tabular Arrays
::: code-group
```json [JSON]
{
"items": [
{ "id": 1, "qty": 5 },
{ "id": 2, "qty": 3 }
]
}
```
```yaml [TOON]
items[2]{id,qty}:
1,5
2,3
```
:::
## Mixed and Non-Uniform Arrays
::: code-group
```json [JSON]
{
"items": [1, { "a": 1 }, "x"]
}
```
```yaml [TOON]
items[3]:
- 1
- a: 1
- x
```
:::
> [!NOTE]
> When a list-item object has a tabular array as its first field, the tabular header appears on the hyphen line. Rows are indented two levels deeper than the hyphen, and other fields are indented one level deeper. This is the canonical encoding for this pattern.
::: code-group
```yaml [Multi-field object]
items[1]:
- users[2]{id,name}:
1,Ada
2,Bob
status: active
```
```yaml [Single-field object]
items[1]:
- users[2]{id,name}:
1,Ada
2,Bob
```
:::
## Arrays of Arrays
::: code-group
```json [JSON]
{
"pairs": [[1, 2], [3, 4]]
}
```
```yaml [TOON]
pairs[2]:
- [2]: 1,2
- [2]: 3,4
```
:::
## Root Arrays
::: code-group
```json [JSON]
["x", "y", "z"]
```
```yaml [TOON]
[3]: x,y,z
```
:::
## Empty Containers
::: code-group
```json [Empty Object]
{}
```
```yaml [Empty Object]
(empty output)
```
:::
::: code-group
```json [Empty Array]
{
"items": []
}
```
```yaml [Empty Array]
items: []
```
:::
## Quoting Special Cases
### Strings That Look Like Literals
::: code-group
```json [JSON]
{
"version": "123",
"enabled": "true"
}
```
```yaml [TOON]
version: "123"
enabled: "true"
```
:::
These strings must be quoted because they look like numbers/booleans.
### Strings Containing Delimiters
::: code-group
```json [JSON]
{
"note": "hello, world"
}
```
```yaml [TOON]
note: "hello, world"
```
:::
Strings must be quoted when they contain the active delimiter (inside an array scope) or the document delimiter (object field values, comma by default).
### Strings with Leading/Trailing Spaces
::: code-group
```json [JSON]
{
"message": " padded "
}
```
```yaml [TOON]
message: " padded "
```
:::
### Empty String
::: code-group
```json [JSON]
{
"name": ""
}
```
```yaml [TOON]
name: ""
```
:::
## Quoting Rules Summary
Strings **must** be quoted if they:
- Are empty (`""`)
- Have leading or trailing whitespace
- Equal `true`, `false`, or `null` (case-sensitive)
- Look like numbers (e.g., `"42"`, `"-3.14"`, `"1e-6"`, `"05"`)
- Contain special characters: `:`, `"`, `\`, `[`, `]`, `{`, `}`, or any control character (U+0000–U+001F, including newline/tab/CR)
- Contain the relevant delimiter – the active delimiter inside an array scope, or the document delimiter (comma by default) for object field values
- Equal `"-"` or start with `"-"` followed by any character
Otherwise, strings can be unquoted. Unicode and emoji are safe:
```yaml
message: Hello 世界 👋
note: This has inner spaces
```
## Escape Sequences
Six escape sequences are valid in quoted strings:
| Character | Escape |
|-----------|--------|
| Backslash (`\`) | `\\` |
| Double quote (`"`) | `\"` |
| Newline | `\n` |
| Carriage return | `\r` |
| Tab | `\t` |
| Any other U+0000–U+001F control character | `\uXXXX` |
Other escapes (e.g., `\x`, `\0`, `\b`) are invalid, and lone-surrogate `\uXXXX` values (U+D800–U+DFFF) are rejected.
## Array Headers
### Basic Header
```
key[N]:
```
- `N` = array length
- Default delimiter: comma
### Tabular Header
```
key[N]{field1,field2,field3}:
```
- `N` = array length
- `{fields}` = column names
- Default delimiter: comma
### Alternative Delimiters
::: code-group
```yaml [Tab Delimiter]
items[2 ]{id name}:
1 Alice
2 Bob
```
```yaml [Pipe Delimiter]
items[2|]{id|name}:
1|Alice
2|Bob
```
:::
The delimiter symbol appears inside the brackets and braces.
## Key Folding (Optional)
Standard nesting:
```yaml
data:
metadata:
items[2]: a,b
```
With key folding (`keyFolding: 'safe'`):
```yaml
data.metadata.items[2]: a,b
```
See [Format Overview – Key Folding](/guide/format-overview#key-folding-optional) for details.
## Type Conversions
| Input | Output |
|-------|--------|
| Finite number in `[1e-6, 1e21)` (or zero) | Canonical decimal |
| Finite number outside that range | Exponent form permitted |
| `NaN`, `Infinity`, `-Infinity` | `null` |
| `BigInt` (safe range) | Number |
| `BigInt` (out of range) | Quoted decimal string |
| `Date` | ISO string (quoted) |
| `Set` | Array of normalized values |
| `Map` | Object with `String(key)` keys |
| `undefined`, `function`, `symbol` | `null` |
::: info
TOON itself doesn't specify how `Date` should be encoded – the spec leaves this to implementations. This library emits an ISO 8601 string in quotes; other implementations may choose differently.
:::