strukto-ai--mirage/conformance

Command Conformance Spec

One declarative spec for command behavior, run by both the Python and TypeScript implementations (issue #151). Each case pins the exact bytes of stdout, the exact bytes of stderr, and the exit code — so trailing newlines, empty output, and binary output are all part of the contract.

Runners

• Python: python/tests/conformance/test_conformance.py runs every case against ram and disk, plus redis when REDIS_URL is set. Runs as part of uv run pytest.
• TypeScript: typescript/packages/node/src/conformance.test.ts runs every case against ram. Runs as part of pnpm test.

Files

• seeds.json — files written into the workspace before every case, via the ops/fs write API (not shell commands). Values are {"text": ...} for UTF-8 content or {"base64": ...} for binary content.
• cases/<command>.json — cases for one command:

{
  "command": "wc",
  "cases": [
    {
      "id": "wc_default",
      "cmd": "wc /data/a.txt",
      "matrix": { "python": ["ram", "disk", "redis"], "typescript": ["ram"] },
      "expect": {
        "exit": 0,
        "stdout_text": " 5  5 24 /data/a.txt\n",
        "stderr_text": ""
      }
    }
  ]
}

• stdout / stderr use exactly one of *_text (UTF-8) or *_base64 (arbitrary bytes). Trailing newlines are significant.
• stdin_text / stdin_base64 optionally feed stdin to the command.
• matrix is explicit: a case runs only on the listed backends. Listing a backend is a claim of support — a missing command there is a failure, not a skip. A case whose matrix is empty is a load-time error in both runners.

Policy

• One expected value per case. If a backend or language legitimately diverges, that divergence is triaged first: either it is a bug (fix the implementation) or it is intended semantics (document it and add an explicit per-backend override mechanism — not yet needed).
• This spec is an acceptance/parity net, not a replacement for backend tests. API call counts, pushdown/fallback, cache invalidation, error injection, and concurrency stay in hand-written per-backend tests.
• spec/ (command definitions exported from the registries) is a different artifact: it describes the command surface; this folder describes command behavior.

Adding a backend

Add a backend when its command behavior should be compared against the same expectations as the existing implementations:

1. Add its name to SUPPORTED_MATRIX in the Python and/or TypeScript runner.

2. Add runner setup that constructs a fresh workspace and resets backend state between cases.

3. For API-backed resources, use mocked clients with payloads shaped like real provider responses. Do not call live APIs from this suite.

4. Add the backend to the applicable case matrices:

   "matrix": {
     "python": ["ram", "disk", "redis", "github"],
     "typescript": ["ram", "github"]
   }
   

5. Run the focused Python and TypeScript conformance tests.

Both runners are discovered by the existing Python and TypeScript CI test jobs; no workflow change is required.

Only add a backend to a case when both should produce the same exact stdout, stderr, and exit code. Keep provider-specific API calls, cache behavior, error injection, and concurrency in backend tests.