wehub-resource-sync
a06f331eb8
CI / benchmark (push) Has been skipped
install-script / posix-syntax (push) Successful in 6m1s
CI / build-onnx (push) Failing after 6m43s
init-smoke / dry-run (push) Failing after 15m57s
security / govulncheck (push) Has been cancelled
security / trivy-fs (push) Has been cancelled
CI / test (1.26, ubuntu-latest) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
CI / test (1.26, macos-latest) (push) Has been cancelled
CI / build-windows (push) Has been cancelled
CI / lint (push) Has been cancelled
install-script / powershell-syntax (push) Has been cancelled
install-script / install (macos-14) (push) Has been cancelled
install-script / install (ubuntu-latest) (push) Has been cancelled
84 lines
3.1 KiB
Markdown
84 lines
3.1 KiB
Markdown
# GCX1 wire-format benchmark
|
||
|
||
Reproducible benchmark comparing the GCX1 compact wire format against
|
||
JSON (and, when `JCODEMUNCH=/path/to/jcm` is set, against jCodeMunch
|
||
MUNCH) on representative MCP tool responses.
|
||
|
||
## What it measures
|
||
|
||
For each fixture case the harness captures, for JSON and GCX:
|
||
|
||
- **Bytes** — raw UTF-8 byte length.
|
||
- **tiktoken (cl100k_base)** — LLM-relevant token count via
|
||
`github.com/pkoukk/tiktoken-go` (same loader Gortex uses at runtime).
|
||
Matches Claude 3 / Opus 4 / Sonnet 4 / Haiku 4.5 / GPT-4o budgets.
|
||
- **Claude Opus 4.7 input tokens** — second column populated either
|
||
by scaling cl100k_base (×1.35 empirical inflation factor, the
|
||
default; labeled `estimated`) or by calling Anthropic's
|
||
`messages/count_tokens` endpoint with `--use-api` (requires
|
||
`ANTHROPIC_API_KEY`; results cached to `opus47-counts.json` for
|
||
deterministic reruns; labeled `exact`).
|
||
- **gzip** — gzip-compressed byte length, fair comparison when the
|
||
transport would compress anyway.
|
||
- **Round-trip integrity** — encode → decode → re-marshal, compare to
|
||
the canonical JSON normalisation. Must be 100 % for the format to
|
||
be considered lossless.
|
||
|
||
Results land in `scorecard.md` with per-case rows and a summary of
|
||
medians / totals.
|
||
|
||
## Running
|
||
|
||
```sh
|
||
go run ./bench/wire-format -cases ./bench/wire-format/cases -out ./bench/wire-format/scorecard.md
|
||
```
|
||
|
||
Flags:
|
||
|
||
- `-cases DIR` — directory of fixture case files (default
|
||
`./bench/wire-format/cases`).
|
||
- `-out FILE` — output scorecard markdown path (default stdout).
|
||
- `-json FILE` — emit raw per-case metrics as JSON too.
|
||
- `-tokenizer cl100k|opus47|both` — which token-cost column(s) to
|
||
render (default `both`).
|
||
- `-use-api` — call Anthropic `count_tokens` for exact Opus 4.7
|
||
numbers (requires `ANTHROPIC_API_KEY`; degrades to the scalar
|
||
on network failure with a single warning).
|
||
- `-opus47-cache PATH` — exact-count sidecar (default
|
||
`bench/wire-format/opus47-counts.json`).
|
||
- `-opus47-model NAME` — model id passed to the API
|
||
(default `claude-opus-4-20250514`).
|
||
|
||
## Fixture format
|
||
|
||
Each case is a Go struct literal decoded from a YAML file with two
|
||
sections:
|
||
|
||
```yaml
|
||
tool: search_symbols
|
||
description: 20 search hits on a medium-size repo
|
||
input: |
|
||
[{...JSON rows as the tool would return...}]
|
||
```
|
||
|
||
The harness encodes `input` via the canonical Go encoder for the
|
||
specified tool and scores the two outputs.
|
||
|
||
## Target
|
||
|
||
GCX1 targets **≥20 % token savings vs JSON on the median case**
|
||
with **100 % round-trip integrity**. Current baseline (20 cases):
|
||
|
||
- **Median token savings (cl100k_base): −27.4 %**
|
||
- **Median token savings (Opus 4.7, scalar): −27.3 %**
|
||
- **Median byte savings: −26.8 %**
|
||
- **Round-trip integrity: 20/20**
|
||
|
||
Tabular list payloads (`search_symbols`, `analyze_hotspots`,
|
||
`find_usages_large`, `smart_context`) hit −30 to −38 %. Small
|
||
scalar-heavy records (`graph_stats`, `find_cycles`) can flip
|
||
positive — GCX1's header overhead exceeds the savings when there
|
||
are fewer than ~5 rows and no repeated field names. Payloads with
|
||
long inline bodies (`get_symbol_source`) are roughly neutral — the
|
||
source body dominates and neither encoding compresses it.
|